There is a particular kind of thinker, one who becomes more entrenched in their single big idea about how the world works even in the face of contrary facts, whose predictions become worse, not better, as they amass information for their mental representation of the world. They are on television and in the news every day, making worse and worse predictions while claiming victory, and they have been rigorously studied.
* * *
• • •
It started at the 1984 meeting of the National Research Council’s committee on American-Soviet relations. Newly tenured psychologist and political scientist Philip Tetlock was thirty years old, by far the most junior committee member. He listened intently as members discussed Soviet intentions and American policies. Renowned experts confidently delivered authoritative predictions, and Tetlock was struck by the fact that they were often perfectly contradictory to one another, and impervious to counterarguments.
Tetlock decided to put expert predictions to the test. With the Cold War in full swing, he began a study to collect short- and long-term forecasts from 284 highly educated experts (most had doctorates) who averaged more than twelve years of experience in their specialties. The questions covered international politics and economics, and in order to make sure the predictions were concrete, the experts had to give specific probabilities of future events. Tetlock had to collect enough predictions over enough time that he could separate lucky and unlucky streaks from true skill. The project lasted twenty years, and comprised 82,361 probability estimates about the future. The results limned a very wicked world.
The average expert was a horrific forecaster. Their areas of specialty, years of experience, academic degrees, and even (for some) access to classified information made no difference. They were bad at short-term forecasting, bad at long-term forecasting, and bad at forecasting in every domain. When experts declared that some future event was impossible or nearly impossible, it nonetheless occurred 15 percent of the time. When they declared a sure thing, it failed to transpire more than one-quarter of the time. The Danish proverb that warns “It is difficult to make predictions, especially about the future,” was right. Dilettantes who were pitted against the experts were no more clairvoyant, but at least they were less likely to call future events either impossible or sure things, leaving them with fewer laugh-out-loud errors to atone for—if, that was, the experts had believed in atonement.
Many experts never admitted systematic flaws in their judgment, even in the face of their results. When they succeeded, it was completely on their own merits—their expertise clearly enabled them to figure out the world. When they missed wildly, it was always a near miss; they had certainly understood the situation, they insisted, and if just one little thing had gone differently, they would have nailed it. Or, like Ehrlich, their understanding was correct; the timeline was just a bit off. Victories were total victories, and defeats were always just a touch of bad luck away from having been victories too. Experts remained undefeated while losing constantly. “There is often a curiously inverse relationship,” Tetlock concluded, “between how well forecasters thought they were doing and how well they did.”
There was also a “perverse inverse relationship” between fame and accuracy. The more likely an expert was to have his or her predictions featured on op-ed pages and television, the more likely they were always wrong. Or, not always wrong. Rather, as Tetlock and his coauthor succinctly put it in their book Superforecasting, “roughly as accurate as a dart-throwing chimpanzee.”
Early predictions in Tetlock’s research pertained to the future of the Soviet Union. There were experts (usually liberal) who saw Mikhail Gorbachev as an earnest reformer who would be able to change the Soviet Union and keep it intact for a while, and experts (usually conservative) who felt that the Soviet Union was immune to reform, ruinous by its very nature, and losing legitimacy. Both sides were partly right and partly wrong. Gorbachev did bring real reform, opening the Soviet Union to the world and empowering citizens. But those reforms uncorked bottled-up forces in the republics outside of Russia, where the system had lost legitimacy. Starting with Estonia declaring its sovereignty, the forces blew the Soviet Union apart. Both camps of experts were completely taken by surprise at the swift end of the USSR, and their predictions about the course of events were terrible. There was, however, one subgroup within the experts that managed to see more of what was coming.
Unlike Ehrlich and Simon, they were not vested in a single approach. They were able to take from each argument and integrate apparently contradictory worldviews. They agreed that Gorbachev was a real reformer, and that the Soviet Union had lost legitimacy outside of Russia. Some of those integrators actually foresaw that the end of the Soviet Union was close at hand, and that real reforms would be the catalyst.
The integrators outperformed their colleagues on pretty much everything, but they especially trounced them on long-term predictions. Eventually, Tetlock conferred nicknames (borrowed from philosopher Isaiah Berlin) that became famous throughout the psychology and intelligence-gathering communities: the narrow-view hedgehogs, who “know one big thing,” and the integrator foxes, who “know many little things.”
Hedgehog experts were deep but narrow. Some had spent their careers studying a single problem. Like Ehrlich and Simon, they fashioned tidy theories of how the world works through the single lens of their specialty, and then bent every event to fit them. The hedgehogs, according to Tetlock, “toil devotedly” within one tradition of their specialty, “and reach for formulaic solutions to ill-defined problems.” Outcomes did not matter; they were proven right by both successes and failures, and burrowed further into their ideas. It made them outstanding at predicting the past, but dart-throwing chimps at predicting the future. The foxes, meanwhile, “draw from an eclectic array of traditions, and accept ambiguity and contradiction,” Tetlock wrote. Where hedgehogs represented narrowness, foxes ranged outside a single discipline or theory and embodied breadth.
Incredibly, the hedgehogs performed especially poorly on long-term predictions within their domain of expertise. They actually got worse as they accumulated credentials and experience in their field. The more information they had to work with, the more they could fit any story to their worldview. This did give hedgehogs one conspicuous advantage. Viewing every world event through their preferred keyhole made it easy to fashion compelling stories about anything that occurred, and to tell the stories with adamant authority. In other words, they make great TV.
* * *
• • •
Tetlock is clearly a fox. He is a professor at Penn, and when I visited his home in Philadelphia I was enveloped in a casual conversation about politics he was having with colleagues, including his wife and collaborator, Barbara Mellers, also a psychologist and eminent scholar of decision making. Tetlock would start in one direction, then interrogate himself and make an about-face. He drew on economics, political science, and history to make one quick point about a current debate in psychology, and then stopped on a dime and noted, “But if your assumptions about human nature and how a good society needs to be structured are different, you would see this completely differently.” When a new idea entered the conversation, he was quick with “Let’s say for the sake of argument,” which led to him playing out viewpoints from different disciplines or political or emotional perspectives. He tried on ideas like Instagram filters until it was hard to tell which he actually believed.