SHIFT_beepA geek blog by Steve Mould

Nice, I loved the solids of constant curvature, the Gomboc is another interesting idea in a similar vein: http://www.wired.com/gadgetlab/2008/02/the-gomboc-the/

perhaps you should consider seeing if ars is interested in you: http://arstechnica.com/staff/palatine/2009/11/want-to-freelance-for-ars-technica.ars

Nice, I loved the solids of constant curvature, the Gomboc is another interesting idea in a similar vein: http://www.wired.com/gadgetlab/2008/02/the-gomboc-the/

perhaps you should consider seeing if ars is interested in you: http://arstechnica.com/staff/palatine/2009/11/want-to-freelance-for-ars-technica.ars

Thanks Mat. I really want one of those Gombocs. Hmmm, €149, can I justify it? Very kind to suggest Ars. Might as well give it a go.

Thanks Mat. I really want one of those Gombocs. Hmmm, €149, can I justify it? Very kind to suggest Ars. Might as well give it a go.

I think the formula for P is actually log_10(1+1/d).

And it works in any other base, too. Say you’re looking at binary numbers, then the probability of a number starting with a 1 is P(1) = log_2(1+1/1) = 1 !

Hey FelixThanks for that. You’re absolutely right. I’ve fixed it now. And you’re right, you just have to change to what base the log is taken to match the base you’re counting it. Fascinating stuff.

FelixCQ is right, the formula is wrong and is supposed to be log_10(1+1/d). http://en.wikipedia.org/wiki/Benford%27s_law

[...] How to cheat with Frank Benford [SHIFT_beep via #tips] Tagged:evil weeknumbersprobability [...]

Thanks Andy. Absolutely. Quite a big typo that!

I’m not sure your example of the Financial Times is “correct” – can you really consider the numbers that appear on the front page to be random? I could chose to look at the flier from my local grocery, for example, and we’d find a very different distribution, for example. (lots of 9s, I’d wager!) I think the point here is that when someone fakes numbers, they assume a completely random system – and that’s rarely the case in reality.

The reason the Financial Times works well is because I’m pulling numbers from lots of DIFFERENT distributions (page numbers, dates, stack quotes etc.). This always shows a good match with Benford. Your grocery flayer will mainly have grocery prices all of a similar order of magnitude.

Benford’s law does work for single distributions so long as the distribution is spread over several orders of magnitude. And assuming that there are no funny quirks like a bias for things priced at 99 cents etc. And even then Benford’s law is a great way of discovering these biases.

I bet most of them don’t start with 9s…

We are talking about the probability of the number starting with a specific single digit, not of it containing the digit.

[...] How to cheat with Frank Benford | SHIFT_beep Pick a number at random from the universe. Not just from inside your head. Open a page of the financial times or look up the size of a planet; convert you height to cubits or measure the weight of your favourite book. Something like that. [...]

It seems that the key thing to note about this is that it is not pertaining to random numbers, but rather real-life measurements. This would then make sense, because most things in real-life we measure by counting up, and therefore, the higher the numbers get, the less likely they will be used. If we measured a subtractive value, like the distance left to reach something, we should see the opposite curve. This is why phone numbers (something truly random, or at least more random) won’t work.

I think you’ve got to the heart of it there. It’s about real world numbers and how the higher up you go the less often they are used. A random number generator in a computer on the other hand will give a nice even distribution of leading digits.

And phone numbers are like random numbers because they’re not really numbers! That is to say the actual value isn’t a measure of anything. It’s meaningless. Even more so in the UK where all numbers start with a zero!

The catch here is indeed that you want measurements that are ‘natural’ in the sense that they appear in ordinary life and are not pre-selected by humans. The Times does seem like an unlikely candidate to me, because the numbers are ‘selected’ for journalistic value.

I applied Benfords law to my own bank statements a while back and wrote on my blog about the experiment.

http://lukasvermeer.wordpress.com/2010/05/24/benfords-law/

I later found that my own behavior (at ATM machines I often withdraw 20 euros) was the root cause for an odd spike of twos in the data.

http://lukasvermeer.wordpress.com/2010/06/16/benfords-law-revised/

At the bottom of the first post there is a link to a test page I made where you can easily compare Benfords law to your own data. Check it out, and let me know what you find.

This is also why your suggestion to “convert you height to cubits” [sic] would not work. Human heights do not span orders of magnitude. You’re much more likely to get 3 cubits.

There is a non-zero chance that this would lead to 1 cubit: there are people less than 91 cm in stature according to http://en.wikipedia.org/wiki/List_of_shortest_people
but it is exceedingly small.

Planet size spans orders of magnitude. Book weights would too, but there’s probably a reasonably small std. deviation.

And adding multiple distributions that span less than an order of magnitude together would never lead you to Benford’s law.

I disagree. If you collected loads of individual measurements from different distributions at random (like your height in cubits and so on. But just one of each thing!) that collection of disparate things absolutely WOULD follow Benford’s law.

I do agree, that he distribution of human heights in cubits on it’s own would not follow Benford’s Law.

That’s awesome! I’ll have a go.

[...] …Keep Benford’s Law in mind. [...]

[...] How to cheat with Frank Benford | SHIFT_beep: [...]

Nice, Very good comments and even the post. Thanks

Neat! No problemo. Be sure to leave your findings in the comments.

http://pastebin.com/TRv8xqvm
little perl proof of concept

Looks like you’re using randomly generated numbers, not ‘real world’ numbers. What are you trying to prove?

The point here is that numbers from the real world are _not_ randomly distributed.

I insist that you need the subdistributions to straddle orders of magnitude. It is trivial to demonstrate this. If human heights in cubits have much less than 10% chance of starting with a 1 and you combine that with some other tiny distribution, such as imdb star rankings (0-10, but virtually no movies score 10, and relatively few score 1.something), there’s simply no way that you’ll get 30%. It is only when one of the subdistributions has a 30% chance or larger of starting with 1 that the full distribution might obey Benford’s law. That requires either choosing many subsets that are artificially weighted to 1 (human height in meters?) or that already follow Benford’s law because they span over an order of magnitude.

I insist that you need the subdistributions to straddle orders of magnitude. It is trivial to demonstrate this. If human heights in cubits have much less than 10% chance of starting with a 1 and you combine that with some other tiny distribution, such as imdb star rankings (0-10, but virtually no movies score 10, and relatively few score 1.something), there’s simply no way that you’ll get 30%. It is only when one of the subdistributions has a 30% chance or larger of starting with 1 that the full distribution might obey Benford’s law. That requires either choosing many subsets that are artificially weighted to 1 (human height in meters?) or that already follow Benford’s law because they span over an order of magnitude.

No, it’s not about the randomness, it’s about the range of choice. If you pick at random from 1 to 99 then then the distribution is the same, but if you pick from 1 to 199 then it’s a different picture. And the real world range always varies, so on average the numbers starting with 1 are 30% more.

I still disagree! For your demonstration you’ve cherry picked two distributions. The distributions need to be picked at random. See this para from wikipedia:

http://en.wikipedia.org/wiki/Benford%27s_law#Multiple_probability_distributions

The only “cherry picking” I’ve done is exactly what I said: to select distributions that do not span an order of magnitude (biased in scale/base).

Read Hill’s paper linked from the Wikipedia article. Your distribution needs to be unbiased in scale and base. Real life distributions and sampling does not necessarily satisfy this requirement. Combining multiple distributions that are less than one order of magnitude wide will certainly not!

You also cherry picked distributions that under represent 1s in the leading digit position. If you truly pick the distributions at random you would find that more of the distributions were biased towards having 1s as the leading digit and fewer biased towards having 9s as the leading digit in such a way as to lead to Benford’s Law with a large enough sample.

A “mixed bag” of items picked from many different distributions IS unbiased in scale even if the underlying distributions are not.

I don’t see how Benford’s law applies to raffles – surely a raffle is not on a log scale? If I have for example 1,000 raffle tickets in total, then the chance is not going to be 30% of any one randomly selected ticket starting with a 1.

You’re right, Benford’s Law does not apply to a raffle you know the size of. But what if you didn’t know the size? Assuming it could be any size you’d have to average the probability of the leading digit being 1 over all possible sizes. That gives you the 30% of Benford’s Law.

I still can’t see it. Surely the distribution of raffles is a linear thing, not like, say, a stock price, which could be exponential?

I mean, if I take every raffle size from 1 to infinity, there will be as many tickets that begin with 9 as begin with 1 or any other number, right?

” I mean, if I take every raffle size from 1 to infinity, there will
be as many tickets that begin with 9 as begin with 1 or any other
number, right?”

Nope That’s what the zig zag graph tries it illustrate. Pick an
example from those 1 to infinity raffles and look at the probability
of the winning ticket starting with a 1. How about a raffle that has
20 tickets. Half of them start with a 1! As the size of the raffle
increases that chance goes up and down (the zig zag graph). The
average of those probabilities is 30%

–
07946 424 688
http://www.stevemould.com
http://blog.stevemould.com
http://twitter.com/moulds

Day number and year number are meaningless numbers too. You should not include them in your example .

Not meaningless but limited. Therefore not appliable.

I think you’re saying that because dates don’t span several orders of magnitude they don’t follow Benford’s Law. And you’re right, they probably don’t. But the great thing is, if you’re choosing lots of different things from different distributions, those distributions you’re picking from don’t have to follow Benford’s Law. See this paragraph on wikipedia:

http://en.wikipedia.org/wiki/Benford%27s_law#Multiple_probability_distributions

And this discussion on age in these comments:

http://blog.stevemould.com/how-to-cheat-with-frank-benford/#comment-90473715

[...] as it sounds; it all makes sense when you read the very accessible and plain-English article at SHIFT_beep.  Check it out and it you’ll definitely feel smarter.   [via [...]

I had heard of Benford’s law several times (since the times of my maths degree), but never gave it more than one or two looks (I have always considered me quite “unfond” of probability!). But as you wrote it down, it is wonderful. I think far clearer than my post And e Appears from Nowhere, where the average of some random throws result in e (with a long, long number of runs).

Thanks for writing it so clearly,

Ruben

Nice, I loved the solids of constant curvature

This looks like a new kind of spam! You’ve just copied the first few words from the first comment on this blog. The link to your website is in your profile. I get a lot of spam from companies related to mould (mould removal, plastic injection moulding) because of my surname. They search the internet for things related to mould, find my blog and leave a comment or send me an email! It’s pretty funny really.

This is well known. The occurence of numbers differs by what they are being used for. So numbers in tax returns have a frequency distribution. In fact one statistician set up rules for the US Revenue dept. to audit tax returns. When you fake numerical entries it is difficult to replicate the frequencies of numbers that naturally occur in tax returns. You tend to think they are random and your numbers just don’t look like they should. So they audit you.

This is not new. . .

It certainly isn’t new, no. Frank Benford died in 1948! I hope my post doesn’t sound like it’s declaring something new!

That’s not true. If the raffle is of a finite, but unknown, size, then the data will be evenly distributed. It is only when the max size of the raffle is infinite, that your raffle metaphor works. Not for an unknown size.

Consider the results of this PHP snippet. For any maximum number, the distribution will be even.

$i = 0;
$spread = array();
while($i < 1000000)
{
$n = rand(1,10000000000000000)."";
$spread[$n[0]]++;
$i++;
}
ksort($spread);
var_dump($spread);

This is brilliant! Love the php. But what your php does is find the distribution of leading digits for a fixed raffle size of 10000000000000000. If you want to find the distribution when you don’t know the size UP TO 10000000000000000 you’ve got to modify the rand line to be like this:

$n = rand(1,rand(1,10000000000000000)).”";

The nested rand is giving you a random raffle size to choose from up to the maximum.

When the max size of the raffle is infinite, the distribution fits Beford exactly. But if it’s finite, so long as it spans several orders of magnitude it’ll fit very closely. And of course the more orders of magnitude you span the better the fit gets.

I just stumbled upon your site and wanted to say that I have really enjoyed reading your opinions.

All this discussion is fine and dandy, but you have yet to touch upon the most important aspect. How will this help me win the lottery?

Finally someone asking the important questions!

Just in case someone else want to take a look: http://testingbenfordslaw.com/
Great post dude!

That’s awesome. Thanks Ewerton

Thank you a lot for your interesting article. I have been looking for such message for a very long time.

Thanks for giving such kind of great information .

Interesting blog. It is always good if you can provide additional information about this. Thanks a lot!