Read an Excerpt
Summary and Analysis of The Signal and the Noise
Why So Many Predictions Fail â" but Some Don't
By Nate Silver OPEN ROAD INTEGRATED MEDIA
Copyright © 2015 Nate Silver
All rights reserved.
ISBN: 978-1-5040-4369-4
CHAPTER 1
Summary
Introduction
The invention of the printing press in 1440 led to the spread of information around Europe. Seventy-seven years later, the publication of Martin Luther's Ninety-five Theses plunged the continent into centuries of religious war. But this dissemination of ideas also led to the Industrial Revolution, which brought about extraordinary economic growth.
In the 2000s, big data has made huge progress in areas from baseball to betting, but still, entire sectors, from financial catastrophes to natural disasters, are not being predicted accurately. The amount of data available makes it almost impossible for us to sort through it all, but if we begin to understand our natural biases, we can use the data to help us make better predictions.
Need to Know: Just as the printing press brought a wave of information that destabilized Europe but ultimately led to the Industrial Revolution, the rise of big data will bring a lot of bad information — noise — which we must learn to sift for the truth — the signal.
1: A Catastrophic Failure of Prediction
The financial crisis of 2008 was a result of a failure of prediction by the agencies who gave excellent ratings to securities known as collateralized debt obligations (CDOs), which included bad mortgages and were incredibly unsafe. The rating agencies profited from the abundance of CDOs, so they incentivized banks to continue producing more.
The unprecedented American housing bubble in the 2000s was result of people being encouraged to buy or flip homes even if they couldn't afford to do so. This was supported by a financial market — lenders, brokers, and ratings agencies — which benefited from every sale — and from the incorrect belief that homeownership was always a profitable investment.
For each dollar that was being spent in housing sales, there were almost fifty dollars worth of trades in mortgage-backed securities. Institutions like Lehman Brothers were highly leveraged, and they were betting with money they didn't actually have or that they had borrowed, which put them in precarious positions if the value of their portfolios declined even a small amount. This should have made investors reluctant to purchase assets, but the positive ratings from the credit agencies convinced unknowing buyers that these were solid purchases.
When Barack Obama took office in 2009, his stimulus package was meant to keep unemployment in check; in reality, the recession was worse than people knew at the time, causing the unemployment rate to go higher than his administration predicted it would.
The recession was caused by a series of poor predictions, each caused by predictors overlooking key pieces of information. In the lead-up to the housing crash, ratings agencies were creating models that didn't include all data relevant to the current housing situation, making them useless. An important lesson was learned the hard way. We should always take all data into account, even data that disrupts our models and calls our accuracy into question. A false sense of confidence in accuracy can lead to avoidable disaster.
Need to Know: The housing crisis and subsequent financial collapse and economic recession were caused by a string of prediction errors, each a result of overlooking key information.
2: Are You Smarter Than a Television Pundit?
In the run-up to the 2008 presidential election, many television pundits were unable to predict the obvious: Barack Obama had a solid lead and was going to win. With this knowledge, Silver goes back to evaluate the predictions made on the public affairs show The McLaughlin Group and determines that, overall, they only got about half of their forecasts right. A similar trend could be seen in the 1980s, when political experts failed to predict the collapse of the Soviet Union, despite Gorbachev's sincere efforts to reform the country and the dire economic straits within the USSR. Historically, forecasts from experts in a variety of subject areas were barely more accurate than random chance.
While studying the personalities of these experts, Philip Tetlock divided them into two categories — hedgehogs and foxes. Hedgehogs believe in big ideas and maintain that governing principles affect all behavior. Foxes believe in a plethora of small ideas and understand that there is nuance and complexity in the world.
Tetlock then found that the latter group is better at predicting. For instance, foxes would have been able to see that the USSR was an increasingly unstable country for many reasons, while hedgehogs saw only an "evil empire" or a socialist stronghold. But hedgehogs — with their bold, unwavering beliefs — make better TV guests. For hedgehogs, the more information they have, the more likely they are to twist the data to fit with their pre-held beliefs, missing or ignoring any information that would disrupt their forecasts.
FiveThirtyEight was founded by Silver's desire to approach the 2008 Democratic primary with qualitative analysis rather than cable news fluff. It was founded on three "fox-like" principles:
Principle 1: Think Probabilistically
The forecasts on FiveThirtyEight are probabilistic, meaning they cover a range of likely outcomes, accounting for real world uncertainty. What this means practically is that an event with a 90% chance of happening will still not happen 10% of the time. This doesn't mean the prediction was incorrect.
Principle 2: Today's Forecast Is the First Forecast of the Rest of Your Life
Probabilities are moving targets and will change as new data is considered each day. A key to FiveThirtyEight predictions is the willingness to change as information becomes available.
Principle 3: Look for Consensus
Hedgehogs want to single-handedly predict a major event and bask in the glory of their skills, but foxes realize that the best way to forecast is to aggregate many predictions and look for consensus in the data.
Need to Know: Confident political pundits (hedgehogs) are likely to predict inaccurately because they are blinded by their own biases, while data-driven forecasters (foxes) can combine many perspectives to see the truth more accurately and make better predictions.
3: All I Care About Is W's and L's
In the baseball prediction system Silver created for Baseball Prospectus, PECOTA (Player Empirical Comparison and Optimization Test Algorithm), Silver had determined that Red Sox player Dustin Pedroia would be a success, despite scout reports that dismissed him. Silver's predictions proved to be true, and when he sought an interview with Pedroia about these numbers, he realized that the key to the athlete's success was his above-it-all attitude. Pedroia hadn't listened to the scouting reports, which could have brought him down.
Baseball projection systems must account for the context of the available statistic, decipher skill from luck, and understand the "aging curve," or how a player's performance changes as he ages. The first task is easy enough, but the second and third are much harder. With PECOTA, Silver used similarity scores to compare current players to similar past players as a way to predict future success.
Ever since the publication of Moneyball in 2003, the divide between stat-focused teams and scouting-focused teams has shrunk, as both have appreciated the strengths of the other. Silver pitched PECOTA as a way to project the performances of pitchers and hitters, and it proved to be successful. Eventually, he moved into predicting the performance of minor league players, a much harder task because there was no base to build on. Between 2006 and 2011, PECOTA performed slightly worse than predictions made by traditional scouts, proving that scouts' judgment sometimes does have benefits that can't be derived simply from statistics.
The scouts had the benefit of mixing some stats with information that the system couldn't include. There are unquantifiable characteristics that also indicate whether a minor league player is ready to move up. The key to a good forecast is including as much information as possible, even qualitative information that must be translated to a data set.
Dustin Pedroia didn't seem like a natural choice for scouts because he didn't fit into a standard template for a successful baseball player, but PECOTA could see that some of what scouts saw as flaws — like his short stature — could actually be benefits for his position at second base. The Red Sox stuck with him during a rough first couple of seasons because the stats were showing that he had potential and that things could turn around. He also has an immense amount of confidence. He never doubted his own skills, and that focused perseverance was an incalculable benefit.
Need to Know: Nate Silver's baseball predicting system built on past models to become a successful forecast in Major League Baseball, but the qualitative information that scouts could incorporate made them more successful at predicting success out of the minor leagues.
4: For Years You've Been Telling Us That Rain Is Green
When Hurricane Katrina hit New Orleans in 2005, the National Hurricane Center made excellent predictions about the severity and timing of the storm. Despite that, one-fifth of the city's population chose not to evacuate, mostly because they didn't think the storm would be as bad as it was. Sixteen hundred New Orleanians died.
Weather forecasting, broadly speaking, grew out of the Enlightenment belief that man could understand nature well enough to predict the future. Early twentieth-century weather predictions — both hand-calculated and those generated by early computers — were inaccurate, but weather forecasting is improving slowly, even as computing power grows at astronomical rates.
In 1972, Edward Lorenz coined the idea of the butterfly effect when trying to predict weather with an early computer. He and his team realized that even a tiny change, such as the number of decimal places used to represent a data point, could cause major differences in the forecasts. This led to his chaos theory, which occurs with nonlinear systems that are dynamic, meaning that the effect of one point affects its behavior at another point.
This is true of weather, making it highly vulnerable to inaccuracies. Inaccuracies can be caused by human error, such as our inability to record our surroundings with extreme precision.
As a result, weather forecasters run models with slightly tweaked information and evaluate all the different potential outcomes with these various data sets. For instance, if a weather forecaster says there's a 40% chance of rain, he may have run a variety of simulations and four out of ten times it rained.
At the National Weather Service, meteorologists improve on the statistical models with human intelligence, editing maps to include information that skilled forecasters can see but computers can't decipher. Predictions with human input have consistently shown to be more accurate.
All the data produced by the National Weather Service is free to reuse. Private companies like the Weather Channel use this data to produce more consumer-friendly forecasts. For-profit forecasters have been known to fudge the data slightly for commercial reasons. They have what is called a wet bias, which means they overestimate rain: When they say there's a 20% chance, it only rains about 5% of the time. This is a deliberate choice, as people prefer to be surprised when it doesn't rain rather than surprised when it does.
Local news has even more of a wet bias, consistently overpredicting precipitation. This allows for more dramatic television, and news stations seem less interested in producing good forecasts, since people don't trust forecasts much to begin with. This circular logic is mostly harmless, but it can become crucial during emergencies like Hurricane Katrina, when viewers don't take the predictions of major weather seriously.
Need to Know: Weather forecasting, especially hurricane prediction, has become increasingly accurate in recent decades, but commercial incentives have meant that consumers don't always receive the most-accurate forecasts.
5: Desperately Seeking Signal
L'Aquila, Italy, had suffered several small earthquakes in the spring of 2009 before a 6.3 magnitude quake killed three hundred people and caused more than $16 billion in damages. Despite sitting very near a fault line, the city had grown complacent about the likelihood of serious quakes and was not sufficiently prepared.
In seismology, the distinction between a prediction and a forecast is important. A prediction would determine a specific time and place a quake will strike; according to the US Geological Survey, earthquakes are impossible to predict. A forecast, on the other hand, is a probabilistic statement about the likelihood of an earthquake in a certain region over a certain period of time. Earthquakes can be forecasted.
There are more than a million very small earthquakes every year, but very few large ones. When graphed logarithmically, earthquakes are strikingly constant — a magnitude 6.0 earthquake occurs ten times more frequently than magnitude 7.0 and one hundred times more frequently than a magnitude 8.0. So while it's possible to predict how often an area should get a catastrophic earthquake, there's no way to say when it might, which makes it difficult to prepare for.
What seismologists are struggling with is overfitting, the act of mistaking noise for a signal. This happens when a limited sample set produces incredibly specific conclusions that cannot be applied to more broad data sets. Overfitting is tempting because it makes a model look more accurate, when it in fact performs less accurately. This was the case in the models in Japan before the earthquakes in 2011 — they had taken a slight over-occurrence of 7.5 magnitude quakes to indicate that this was as high as quakes were likely to get, and that a higher quake (like the 9.1 that they eventually got) was virtually impossible.
Given how rare major earthquakes are, it will be centuries before we can even conceive of using past events to predict future earthquakes. Some physicists believe earthquakes subscribe to the theory of complexity, which simply says that there are so many intertwined processes happening at once that we would never be able to predict them with any true certainty.
Need to Know: Earthquakes are possible to forecast probabilistically — how likely a certain type of earthquake is to hit an area in a span of time — but it is, at this point, impossible to predict the exact time or place with much accuracy.
6: How to Drown in Three Feet of Water
Political polls always include a reference to a margin of error, but economic forecasts rarely include such caveats. This gives the impression that they are extraordinarily accurate when they are not. When polled in November 2007, economists in the Survey of Professional Forecasters were confident that no recession was coming; they expected the economy to grow slightly in 2008.
In reality, GDP shrank by 3.3%, a likelihood that they said had a 1-in-500 chance of happening. On average, economists' GDP forecast have a margin of error of plus or minus 3.2%. But despite long-term inaccuracies, economists have not worked to improve their forecasts and reduce their bias towards overconfidence.
Part of the problem is that it's very difficult to determine cause and effect from economic data, which produces millions of data points. With so much data, it can be hard to understand the relationships between points. At the same time, the economy changes so quickly that indicators that are useful one year might be irrelevant only a few years later.
It's also often impossible to distinguish correlation and causation, with factors like unemployment and consumer confidence sometimes leading as indicators and sometimes following. At the same time, economic policy put into place by governments affects the economy, meaning that past data about economic booms and busts must take into account the policies of the time.
One possible reason that the Fed's economic forecasts in 2007 did not predict a recession may be that they were focusing on data from 1986 to 2006, a period of very little volatility. They were ignoring data from eras with recessions and focusing on their success in forecasting during this calmer period.
Another factor is that economic data is subject to revision by the government — for example, in the last quarter of 2008, the government estimated the GDP was declining by 3.8% when it was actually declining by almost 9%. The data that economists are working with to make forecasts is often inaccurate.
(Continues...)
Excerpted from Summary and Analysis of The Signal and the Noise by Nate Silver. Copyright © 2015 Nate Silver. Excerpted by permission of OPEN ROAD INTEGRATED MEDIA.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
