The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day

by David J. Hand Narrated by Paul Hodgson

Overview

In The Improbability Principle, the renowned statistician David J. Hand argues that extraordinarily rare events are anything but. In fact, they're commonplace. Not only that, we should all expect to experience a miracle roughly once every month.
But Hand is no believer in superstitions, prophecies, or the paranormal. His definition of "miracle" is thoroughly rational. No mystical or supernatural explanation is necessary to understand why someone is lucky enough to win the lottery twice, or is destined to be hit by lightning three times and still survive. All we need, Hand argues, is a firm grounding in a powerful set of laws: the laws of inevitability, of truly large numbers, of selection, of the probability lever, and of near enough.
Together, these constitute Hand's groundbreaking Improbability Principle. And together, they explain why we should not be so surprised to bump into a friend in a foreign country, or to come across the same unfamiliar word four times in one day. Hand wrestles with seemingly less explicable questions as well: what the Bible and Shakespeare have in common, why financial crashes are par for the course, and why lightning does strike the same place (and the same person) twice. Along the way, he teaches us how to use the Improbability Principle in our own lives-including how to cash in at a casino and how to recognize when a medicine is truly effective.
An irresistible adventure into the laws behind "chance" moments and a trusty guide for understanding the world and universe we live in, The Improbability Principle will transform how you think about serendipity and luck, whether it's in the world of business and finance or you're merely sitting in your backyard, tossing a ball into the air and wondering where it will land.

Editorial Reviews

The New York Times Book Review - Jennifer Ouellette

Human beings are a superstitious lot; we see patterns everywhere. But as Hand makes clear in this enlightening book, it all comes down to the math.

Publishers Weekly

12/02/2013
Winning the lottery or being struck by lightning is rare but not impossible, so when it happens repeatedly to the same individual, it seems miraculous. Not so, writes Hand, emeritus professor of mathematics at Imperial College, London (Statistics: A Very Short Introduction), in this ingenious introduction to probability that mixes counterintuitive anecdotes with easily digestible doses of statistics. Thus, through the “law of truly large numbers,” he reveals that, among the billions of events we experience throughout our lives, outrageous ones are bound to occur. Meanwhile, the “law of selection” reveals how probabilities can be made to appear artificially high due to selecting criteria after an event. In other words, it generates miracles from otherwise routine events. And everyone has nightmares ending in disaster: historians have marveled at Lincoln’s pre-assassination dream. Similarly, according to pyramidologists, combining dimensions of the Great Pyramid of Giza produces amazing predictions or coded messages; the pyramid itself is huge and irregular, giving the examiner leeway in choosing the numbers. But it’s the “law of near-enough” that guarantees such spectacular connections. Hand offers much food for thought, and readers willing to handle some simple mathematics will find this a delightful addition to the “why people believe weird things” genre. (Feb.)

From the Publisher

Human beings are a superstitious lot; we see patterns everywhere. But as Hand makes clear in this enlightening book, it all comes down to the math.” —Jennifer Ouellette, The New York Times Book Review

“Very engaging . . . If you wish to read about how probability theory can help us understand the apparent hot hand in a basketball game, superstitions in gambling and sports, prophecies, parapsychology and the paranormal, holes in one, multiple lottery winners, and much more, this is a book you will enjoy. I will go further. The statistician Samuel S. Wilks (paraphrasing H.G. Wells) said that ‘statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.' With that laudable goal in mind, The Improbability Principle should be, in all probability, required reading for us all.” —John A. Adam, The Washington Post

“[A] lucid overview of the mathematics of chance and the psychological phenomena that can make probability seem counter-intuitive to so many . . . Hand has written a superlative introduction to critical thinking, accessible to everybody, regardless of mathematical ability.” —New Scientist

“[An] ingenious introduction to probability that mixes counterintuitive anecdotes with easily digestible doses of statistics . . . Hand offers much food for thought, and readers willing to handle some simple mathematics will find this a delightful addition to the 'why people believe weird things' genre.” —Publishers Weekly

“Lively and lucid . . . an intensely useful (as well as a remarkably entertaining) book . . . It can transform the way you read the newspaper, that's for sure.” —Salon

“[Hand] leads readers through this unfamiliar land of probability and statistics with wit and charm, all the while explaining in layman's terms the laws that govern it . . . We predict there's a very good chance you'll enjoy this book” —Success

“Enlightening and entertaining . . . an erudite but utterly unpretentious guide . . . ably and assuredly demystifies an ordinarily intimidating subject” —Kirkus

“In my experience, it is very rare to find a book that is both erudite and entertaining. Yet The Improbability Principle is such a book. Surely this cannot be due to chance alone!” —Hal R. Varian, chief economist at Google and professor emeritus at the University of California, Berkeley

“Considering that The Improbability Principle comes from the keyboard of David J. Hand, it was perhaps inevitable that it would be a certain winner!” —John Pullinger, president of the Royal Statistical Society

“Written by one of the world's preeminent statisticians, The Improbability Principle provides you with a sense of what chance and improbability really mean, and engenders an understanding that uncertainty rests at the core of nature. I highly recommend this book.” —Joseph M. Hilbe , president of the International Astrostatistics Association and ambassador for the NASA/Jet Propulsion Laboratory at the California Institute of Technology

“As someone who happened to meet his future wife on a plane, on an airline he rarely flew, I wholeheartedly endorse David J. Hand's fascinating guide to improbability, a subject that affects the lives of us all, yet until now has lacked a coherent exposition of its underlying principles.” —Gordon Woo, catastrophist at Risk Management Solutions and author of Calculating Catastrophe

“The Improbability Principle is an elegant, astoundingly clear, and enjoyable combination of subtle statistical thinking and real-world events. David J. Hand really does explain why ‘surprising' things will happen and why statistics matters.” —Andrew Dilnot, coauthor of The Numbers Game: The Commonsense Guide to Understanding Numbers in the News, in Politics, and in Life

catastrophist at Risk Management Solutions and aut Gordon Woo

As someone who happened to meet his future wife on a plane, on an airline he rarely flew, I wholeheartedly endorse David J. Hand's fascinating guide to improbability, a subject that affects the lives of us all, yet until now has lacked a coherent exposition of its underlying principles.

president of the International Astrostatistics Ass Joseph M. Hilbe

Written by one of the world's preeminent statisticians, The Improbability Principle provides you with a sense of what chance and improbability really mean, and engenders an understanding that uncertainty rests at the core of nature. I highly recommend this book.

Library Journal

03/01/2014
We frequently hear about extremely unlikely occurrences: someone wins a lottery twice, or gets hit by lightening thrice, or has a dream that uncannily predicts a future event. We think these instances are so unlikely that we want to attribute them to supernatural causes. But such things happen often enough that there should be other explanations. Hand (mathematics, Imperial Coll., London; Information Generation: How Data Rules Our World) provides explanations in this clear exposition. He uses actual examples to show why many strange events are really more likely than one would expect, while other cases are the result of deception (conscious or unconscious) based upon post-event selection (i.e., interpretation after the fact). Hand shows how the mind can equate matters that are in fact not the same or can misremember details. He also provides a short philosophical diversion into cosmology and evolution and a quick lesson in how statistics are used and misused in research. VERDICT This entertaining and illuminating book, explaining a lot with a minimum of mathematical alphabet soup, is recommended to all interested general readers.—Harold D. Shane, mathematics, emeritus, Baruch Coll., CUNY

Kirkus Reviews

2014-01-12
Enlightening and entertaining explanation of why extraordinary events are to be expected. Former Royal Statistical Society president Hand (Emeritus, Mathematics/Imperial Coll., London; Statistics: A Very Short Introduction, 2008, etc.) is an erudite but utterly unpretentious guide to the often confusing and counterintuitive subject of probability and its underappreciated complement, improbability. He explains why we should not be surprised, for example, when some people win lotteries or are hit by lightning multiple times, despite the odds against either event happening to any single person even once being vanishingly small. Chapter by chapter, Hand pieces together the threads of what he calls the "Improbability Principle," showing that if something can happen, given enough time and enough opportunities (according to the law of very large numbers), it will happen. But he also reveals that extraordinary events which, in theory, are highly improbable or even impossible, usually prove, upon closer inspection, to have "probability levers" that raise the odds they will happen: Roy Sullivan was struck seven times by lightning, which would seem fantastical if you didn't know he was a forest ranger. "The Improbability Principle tells us that events which we regard as highly improbable occur because we got things wrong," writes Hand. "If we can find out where we went wrong, then the improbable will become probable." Without taxing casual readers with strenuous math, the author coolly examines many fascinating examples of the unlikely, including odd coincidences—as when actor Anthony Hopkins found a copy of the book his next film project was to be based on in an empty seat on a subway train and learned weeks later that the copy belonged to a friend of the book's author who was preparing an American edition—hot streaks in sports, ESP research, C.G. Jung's accounts of synchronicity, and even the origins of life and the universe. Ably and assuredly demystifies an ordinarily intimidating subject.

Product Details

BN ID:	2940169426830
Publisher:	Macmillan Audio
Publication date:	02/11/2014
Edition description:	Unabridged

Read an Excerpt

Improbability Principle

Why Coincidences, Miracles, and Rare Events Happen Every Day

---

By David J. Hand

Farrar, Straus and Giroux

Copyright © 2014 David J. Hand
All rights reserved.
ISBN: 978-0-374-17534-4

---

1
 

THE MYSTERY
Fortune brings in some boats that are not steer’d.
 —William Shakespeare
Simply Unbelievable
In the summer of 1972, the actor Anthony Hopkins was signed to play a leading role in a film based on George Feifer’s novel The Girl from Petrovka, so he traveled to London to buy a copy of the book. Unfortunately, none of the main London bookstores had a copy. Then, on his way home, waiting for an underground train at Leicester Square tube station, he saw a discarded book lying on the seat next to him. It was a copy of The Girl from Petrovka.
As if that was not coincidence enough, more was to follow. Later, when he had a chance to meet the author, Hopkins told him about this strange occurrence. Feifer was interested. He said that in November 1971 he had lent a friend a copy of the book—a uniquely annotated copy in which he had made notes on turning the British English into American English (“labour” to “labor,” and so on) for the publication of an American version—but his friend had lost the copy in Bayswater, London. A quick check of the annotations in the copy Hopkins had found showed that it was the very same copy that Feifer’s friend had mislaid.1
You have to ask: What’s the chance of that happening? One in a million? One in a billion? Either way, it begins to stretch the bounds of credibility. It hints at an explanation in terms of forces and influences of which we are unaware, bringing the book back in a circle to Hopkins and then to Feifer.
Here’s another striking incident, this time from the book Synchronicity, by the psychoanalyst Carl Jung. He writes: “The writer Wilhelm von Scholz … tells the story of a mother who took a photograph of her small son in the Black Forest. She left the film to be developed in Strassburg. But, owing to the outbreak of war, she was unable to fetch it and gave it up for lost. In 1916 she bought a film in Frankfurt in order to take a photograph of her daughter, who had been born in the meantime. When the film was developed, it was found to be doubly exposed: the picture underneath was the photograph she had taken of her son in 1914! The old film had not been developed and had somehow got into circulation again among the new films.”2
Most of us will have experienced coincidences rather like these—if not quite so extraordinary. They might be more akin to thinking of someone just before she phones you. Strangely enough, while I was writing part of this book, I had precisely this sort of experience. A colleague at work asked me if I could recommend some publications on a specific aspect of statistical methodology (the so-called “multivariate t-distribution”). The next day, I did a little research and managed to identify a book on exactly that topic by two statisticians, Samuel Kotz and Saralees Nadarajah. I had started to type an e-mail to my colleague, giving him the details of this book, when I was interrupted by a phone call from Canada. During the conversation, the caller happened to mention that Samuel Kotz had just died.
And so it goes on. On September 28, 2005, The Telegraph described how a golfer, Joan Cresswell, scored a hole in one with a fifty-yard shot at the thirteenth hole at the Barrow Golf Club in Cumbria in the UK. Surprising, you may think, but not outlandishly so—after all, holes in one do happen. But what if I tell you that, immediately afterward, a fellow golfer, the novice Margaret Williams, also scored a hole in one?3
There’s no getting away from it: sometimes events occur which seem so improbable, so unexpected, and so unlikely, they hint that there’s something about the universe we don’t understand. They make us wonder if the familiar laws of nature and causality, through which we run our everyday lives, occasionally break down. They certainly make us doubt that they can be explained by the accidental confluence of events, by the random throwing together of people and things. They almost suggest that something is exerting an invisible influence.
Often such occurrences merely startle us, and give us stories to tell. On my first trip to New Zealand, I settled down in a café, and noticed that the notepaper being used by one of the two strangers at the neighboring table was from my own university back in the UK. But at other times, these uncanny events can significantly alter lives—for the better, as with a New Jersey woman who won the lottery twice, or for the worse, as with Major Summerford, who was struck by lightning several times.
Humans are curious animals, so we naturally seek the underlying cause of strange coincidences. What was it that led two strangers from the same university to travel to the far side of the world and end up sitting at neighboring tables in the same café at exactly the same time? What was it that led the woman to pick those two winning sets of lottery numbers? What was it that brought huge electrostatic forces to hit Major Summerford time and time again? And what steered Anthony Hopkins and The Girl from Petrovka through space, and through time, to the same seat in the same underground station at the same moment?
Beyond that, of course, how can we take advantage of the causes underlying such coincidences? How can we manipulate them to our benefit?
So far all my examples have been very small-scale—at the personal level. But there are countless more-profound examples. Some seem to imply that not only the human race, but the very galaxies themselves wouldn’t exist if those very unlikely events hadn’t occurred. Some relate to how sequences of tiny random changes in our genetic constitution could end up producing something as complicated as a human being. Others relate to the distance of the earth from the sun, the existence of Jupiter, and even the values of the fundamental constants of physics. Again the question arises as to whether blind chance is a realistic explanation for these apparently staggeringly unlikely events, or whether there are in fact other influences and forces directing the course of events behind the scenes.
The answers to all these questions hinge on what I call the Improbability Principle. This asserts that extremely improbable events are commonplace. It’s a consequence of a collection of more fundamental laws, which all tie together to lead inevitably and inexorably to the occurrence of such extraordinarily unlikely events. These laws, this principle, tell us that the universe is in fact constructed so that these coincidences are unavoidable: the extraordinarily unlikely must happen; events of vanishingly small probability will occur. The Improbability Principle resolves the apparent contradiction between the sheer unlikeliness of such events, and the fact that they nevertheless keep on happening.
We’ll begin by looking at prescientific explanations. These often go far back into the mists of time. Although many people still hold to them, they predate the Baconian revolution: that is the idea that the way to understand the natural world is to collect data, conduct experiments, take observations, and use these as test beds through which to evaluate proposed explanations for what’s going on. Prescientific notions predate the rigorous evaluation of the effectiveness of explanations through scientific methods. But explanations which have not been or cannot be tested can have no real force: they are simply anecdotes, or stories, with the same status as a child’s bedtime tale about Santa Claus or the tooth fairy. They serve the purpose of reassuring or placating those who are unwilling or unable to make the effort to dig deeper, but they don’t lead to understanding.
Understanding comes from deeper investigation. In this deeper investigation, thinkers—researchers, philosophers, scientists—have sought to devise “laws” that describe the way nature works. These laws are shorthand summaries encapsulating in simple form what observation shows about how the universe behaves. They are abstractions. For example, the progress of an object falling from a tall building is described by Newton’s Second Law of Motion, which says that the acceleration of a body is proportional to the force acting on it. Natural laws seek to get to the heart of phenomena, stripping away the superfluous, crystallizing the essence. The laws are developed by matching predictions with observations, that is, with data. If a law says that increasing the temperature of an enclosed volume of gas will increase its pressure, is this what actually happens, is this what the data show? If a law says that increasing the voltage will increase the current, is this what we actually see?
We’ve been extraordinarily successful in understanding nature by applying this process of matching data to explanation. The modern world, the cumulation of the awesome achievements of humanity’s science and technology, is a testament to the power of such descriptions.
Of course, some people seem to think that understanding a phenomenon takes away its mystery. This is true in the sense that understanding means removing obscurity, obfuscation, ambiguity, and confusion. But a grasp of the cause of the colors of the rainbow doesn’t detract from its wonder. What such a grasp brings is a more profound appreciation, and indeed awe, of the beauty underlying the phenomenon being studied. It shows us how all the pieces come together to give us the amazing world we live in.
Borel’s Law: Sufficiently Unlikely Events Are Impossible
Émile Borel was an eminent French mathematician, born in 1871. He was a pioneer of some of the more mathematical aspects of probability (of so-called measure theory), and several mathematical objects and concepts are named after him—such as Borel measure, Borel sets, the Borel-Cantelli lemma, and the Heine-Borel theorem. In 1943 he wrote a nonmathematical introduction to probability called Les probabilités et la vie, translated as Probabilities and Life. As well as illustrating some of the properties and applications of probability, in this book he introduced what he called the single law of chance, nowadays often simply called Borel’s law. This law says, “Events with a sufficiently small probability never occur.”4
Clearly, the Improbability Principle looks as if it is at odds with Borel’s law. The Improbability Principle says that events with a very small probability keep on happening, while Borel’s law says they never happen. What is going on?
Now, your first reaction on reading Borel’s law may well have been the same as mine when I first came across it: Surely it’s nonsense? After all, you might think (as I did) that events with very small probability certainly occur, just not very often. That’s the whole point about probability, and about small probabilities in particular. But when I read further into Borel’s book, I saw that he meant something rather more subtle.
He illustrated what he intended by referring to the classic example of the monkeys who, randomly hitting the keys of a typewriter, happen by chance to produce the complete works of Shakespeare.5 In Borel’s words: “Such is the sort of event which, though its impossibility may not be rationally demonstrable, is, however, so unlikely that no sensible person will hesitate to declare it actually impossible. If someone affirmed having observed such an event we would be sure that he is deceiving us or has himself been the victim of fraud.”6
So Borel is relating “very small probabilities” to human scales, and that’s what he means: in human terms the probability is so small that it would be irrational to expect ever to see it happen; it should be regarded as impossible. And, indeed, after stating his “single law of chance” (which, you’ll recall, was that events with a sufficiently small probability never occur) he added the comment, “or, at least, we must act, in all circumstances, as if they were impossible” [his italics].7
He gave a further illustration later in his book: “For every Parisian who circulates for one day, the probability of being killed in the course of the day in a traffic accident is about one-millionth. If, in order to avoid this slight risk, a man renounced all external activity and cloistered himself in his house, or imposed such confinement on his wife or his son, he would be considered mad.”8
Other thinkers had said similar things. For example, in the 1760s Jean d’Alembert had questioned whether it is possible to observe a very long run of occurrences of an event in a sequence in which occurrence and nonoccurrence are equally probable. A century before Borel, in 1843, in his book Exposition de la théorie des chances et des probabilités, Antoine-Augustin Cournot had discussed the actual, as opposed to theoretical, probability of a perfect cone balancing on its vertex.9 The phrase “practical certainty” has been associated with Cournot, being contrasted with “physical certainty.” Indeed, the idea that “It is a practical certainty that an event with very small probability will not happen” is sometimes called Cournot’s principle. Later, in the 1930s, the philosopher Karl Popper wrote, in his book The Logic of Scientific Discovery, “the rule that extreme improbabilities have to be neglected … agrees with the demand for scientific objectivity.”10
Given the other illustrious thinkers who have described similar concepts, we might ask why it’s Borel’s name that’s generally attached to the idea. The answer probably lies in Stigler’s law of eponymy. This law says that “no scientific law is named after its original discoverer” (and then has the corollary, “including this one”).
There’s an analogy between Borel’s law and the points, lines, and planes we learn about when we study geometry in school. We learn that these geometric objects are mathematical abstractions, and that they don’t exist in the real world. They are merely convenient simplifications—which we can then think about and mentally manipulate, and hence draw conclusions about the real-world objects we’re representing with them. Similarly, it’s a mathematical ideal that, although incredibly small probabilities are not actually zero, they may be treated as if they were zero because, in real practical human terms, events with sufficiently small probability never occur. That’s Borel’s law.
Here’s Borel again: “It must be well understood that the single law of chance carries with it a certainty of another nature than mathematical certainty, but that certainty is comparable to one which leads us to accept the existence of an historical character, or of a city situated at the antipodes, of Louis XIV or of Melbourne; it is comparable even to the certainty which we attribute to the existence of the external world.”11
Borel goes on to give a scale showing what might be meant by a probability being “sufficiently small” that an event would never occur. Here are (slightly paraphrased) versions of the definitions he gives of the points on his scale. In each case, I’ve tried to convey the sizes of the numbers involved by giving some examples.
Probabilities which are negligible on the human scale are smaller than about one in a million. The probability of being dealt a royal flush in poker is about 1 in 650,000, almost twice a one-in-a-million chance. There are just over thirty million seconds in a year, so, in terms of Borel’s scale, if you and I each randomly pick a second in which to do something, the chance that we will do it at the same time is negligible on the human scale.
Probabilities which are negligible on the terrestrial scale are smaller than about 1 in 1015. (If you’re unfamiliar with this notation, see my explanation in Appendix A.) The earth’s surface area is about 5.5 × 1015 square feet. So if you and I were each to randomly choose a square foot to stand on (ignoring niceties such as the fact that many of those square feet would be in the ocean), our chance of picking the same square feet would be pretty well negligible on the terrestrial scale. The probability of one player in a game of bridge getting a complete suit is roughly 1 in 4 × 1010, vastly more likely than an event which is negligible on the terrestrial scale.
Probabilities which are negligible on the cosmic scale are smaller than about 1 in 1050. Earth is composed of some 1050 atoms, so if you and I independently pick single atoms from the entire Earth, the chance we would pick the same one is negligible on the cosmic scale. To put that into perspective, in turn, there are “only” something like 1023 stars in the universe altogether.
Probabilities which are negligible on the supercosmic scale are smaller than about 1 in 101,000,000,000. Since the number of subatomic baryon particles in the universe is estimated to be around 1080, it’s difficult to devise any examples which put such small probabilities into context!
Borel’s scale of the “negligibly small” tells us when we should regard events as so unlikely that for practical purposes we can treat them as impossible. But the Improbability Principle tells us that, in contrast, highly unlikely events, even those as unlikely as the ones Borel characterizes, keep on happening. That is, not only are they not impossible, but we see such events again and again. Surely both of these cannot be right: either they’re so unlikely that we’ll never see them happen, or they’re so likely that we’ll see them again and again.
By peeling back the meaning of improbability, we’ll see that this apparent contradiction can be resolved. We can think of the different strands of the Improbability Principle as layers, like those of an onion, so that as each layer is peeled back the explanation becomes clearer. The different strands of the principle—the law of truly large numbers, the law of near enough, the law of selection, and other strands—each shed their own light on how both Borel’s Law and the Improbability Principle can be right simultaneously.
Some of the strands of the principle are very profound. The law of truly large numbers, for example, plays a critical role in determining whether apparent clusters of sickness are caused by a pollutant or are merely due to chance. But others are less so. You might like to see if you can come up with an explanation for the following event which, at face value, would seem so improbable that we wouldn’t expect to see it; so improbable that we should regard it as impossible. The observation is described in U.S. News & World Report of December 19, 2011.12 It refers to the late Kim Jong-Il, former leader of North Korea, and says, “In 1994, the very first time he played golf, Kim Jong-Il dominated the 7,700-yard Pyongyang Golf Course. He shot an unimaginable 38-under par, recording no worse than a birdie at the country’s lone golf course. His round included 11 holes-in-one, and the feat was verified by 17 bodyguards who were present.”
You might want to recall Borel’s hypothetical reaction to monkeys producing the complete works of Shakespeare when randomly hitting the keys of a typewriter. As I said, some strands of the Improbability Principle are straightforward. But others are very profound indeed, and this book explores them.


 
Copyright © 2014 by David J. Hand

(Continues...)

---

Excerpted from Improbability Principle by David J. Hand. Copyright © 2014 David J. Hand. Excerpted by permission of Farrar, Straus and Giroux.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---

Videos