For example, in 2015, forecasters were asked if Greece would exit the eurozone that year. No country had ever left, so the question seemed totally unique. But there were plenty of examples of international negotiation failures, exits from international agreements, and forced currency conversions that allowed the best forecasters to ground themselves in what usually happens without focusing narrowly on all the unique details of the present situation. Starting with the details—the inside view—is dangerous. Hedgehog experts have more than enough knowledge about the minutiae of an issue in their specialty to do just what Dan Kahan suggested: cherry-pick details that fit their all-encompassing theories. Their deep knowledge works against them. Skillful forecasters depart from the problem at hand to consider completely unrelated events with structural commonalities rather than relying on intuition based on personal experience or a single area of expertise.
Another aspect of the forecaster training involved ferociously dissecting prediction results in search of lessons, especially for predictions that turned out bad. They made a wicked learning environment, one with no automatic feedback, a little more kind by creating rigorous feedback at every opportunity. In Tetlock’s twenty-year study, both foxes and hedgehogs were quick to update their beliefs after successful predictions, by reinforcing them even more strongly. When an outcome took them by surprise, however, foxes were much more likely to adjust their ideas. Hedgehogs barely budged. Some hedgehogs made authoritative predictions that turned out wildly wrong, and then updated their theories in the wrong direction. They became even more convinced of the original beliefs that led them astray. “Good judges are good belief updaters,” according to Tetlock. If they make a bet and lose, they embrace the logic of a loss just as they would the reinforcement of a win.
That is called, in a word: learning. Sometimes, it involves putting experience aside entirely.
CHAPTER 11
Learning to Drop Your Familiar Tools
JAKE, THE ATHLETIC-LOOKING sandy blond, speaks first. He wants to race the car. “What if everybody just agrees?” he asks. “I say, race this thang.”
It was early afternoon in fall, and Jake and six of his second-year Harvard Business School classmates found a shady spot where they could eat their lunches and talk.* Their professor had given them three pages containing one of the most famous business school case studies ever created, known as Carter Racing. The crux is whether the fictional Carter Racing team’s car should compete in the biggest race of the season, which begins in one hour.
The argument in favor of racing: thanks to a custom turbocharger, Carter Racing has placed in the money (top five) in twelve of twenty-four races. That success secured an oil company sponsorship, and a trial sponsorship from prestigious (and also fictional) Goodstone Tire. Carter Racing won the last race, its fourth win of the season. Today’s race will be on national TV, and if Carter Racing finishes in the top five, it will likely draw a $2 million sponsorship from Goodstone. If Carter Racing chooses not to race and withdraws, it would lose part of its entry fee and have to pay back some sponsor money. The team would end a stellar season $80,000 in the hole, and may never get another shot this big. Racing seems like a no-brainer.
The argument against racing: in seven of twenty-four races, the engine failed, each time damaging the car. In the last two races, the mechanics used a new engine-prep procedure and had no trouble, but they aren’t sure what caused the problem before. If the engine fails on national TV, the team will lose the oil sponsorship, kiss Goodstone goodbye, and go back to square one, or perhaps out of business. So: race, or don’t race?
The group begins with a vote. Three students vote to race, four to sit it out. Now the debate begins.
Even with the engine failures, Jake says, the team has a 50 percent chance of its biggest triumph. The upside of the Goodstone sponsorship is much more money than the team stands to lose if the engine fails and the existing sponsors walk. If Carter Racing withdraws, an excellent season ends with debt, “which, as we all know, is not a sustainable business model.”
“I just don’t think they can afford not to race,” Justin says.
Alexander agrees, and addresses the dissenters: “What’s going to change going forward to convince you that now you’re ready?” he asks.
Mei, wearing a Harvard hoodie and sitting across the circle, has a calculation to share. “To me, the risk of not racing is about one-third of the downside of [another engine failure],” she says. She adds that she’s focusing on loss mitigation, and does not want to race.
The case study says that at the last minute, the team owner, BJ Carter, called his mechanics. Pat, the engine mechanic, dropped out of high school and has no sophisticated engineering training, but he has a decade of race experience. Temperature could be the issue, he suggested. When the turbocharger warms up on a cool day, engine components might expand at different rates and set up failure of the head gasket, a metal seal in the engine. Pat admitted that each engine failure looked different, but all seven had breaks in the head gasket. (Two of the engine failures had multiple breaks in the gasket.) He didn’t know what was going on, but couldn’t think of anything else on short notice. He was still hyped to race, and jubilant about the new Goodstone uniforms. At 40 degrees, it is the coldest race day of the season. Robin, the chief mechanic, endorsed Pat’s idea to look at the temperature data. He plotted it on a graph, but saw no correlation:
Dmitry, his black hair flopped to one side, is firmly against racing. He agrees that there is no apparent linear relationship between gasket failure and temperature; three gasket breaks occurred on the coolest race day (53 degrees), and two on one of the hottest days (75 degrees). But what if there is an optimal range for the engine, not too cold and not too hot? “If the failures are random, the probability that you both finish and get in the top five is 50 percent,” Dmitry says. “But if it’s not random, the probability is lower. This day is a very, very low temperature that they haven’t experienced before. We don’t know if there’s a correlation with temperature, but if there is, it’s like a sure thing that it fails.”
Julia thinks mechanic Pat’s temperature idea is “nonsense,” but like Dmitry views the engine problem as a black box that does not give the team any information to calculate probability for today’s race. She acknowledges that she’s being risk averse, and would personally never get involved in car racing at all.
Except for Dmitry, the group agrees that there is “zero correlation at all,” as Alexander puts it, between temperature and engine failure. “Am I the only one?” Dmitry asks, to a few giggles.
Jake is particularly unimpressed with engine mechanic Pat’s reasoning. “I think Pat’s a really good mechanic,” he says. “I don’t think he’s a really good root cause analysis engineer, and those are two very different things.” Jake thinks Pat is falling prey to a well-known cognitive bias, overemphasizing the importance of a single, dramatic memory—the three gasket breaks on a cool day. “We don’t even have the information to understand this graph,” Jake says. “There’s twenty-four races, right? How many of those were around 53 degrees and didn’t break? I don’t mean to attack your point,” he says to Dmitry, smiling and giving him a friendly tap on the hand.