Sociologist Diane Vaughan’s book The Challenger Launch Decision came to be regarded by NASA as the definitive causal account of the tragedy. “More stunning is the observation that they did have the pertinent data,” it reads. “There were charts [that several Thiokol engineers who wanted to postpone launch] did not imagine and did not construct that, if created, would have provided the quantitative correlational data required to sustain their position.”
Business professors around the world have been teaching Carter Racing for thirty years because it provides a stark lesson in the danger of reaching conclusions from incomplete data, and the folly of relying only on what is in front of you.
And now for one last surprise. They all got it wrong. The Challenger decision was not a failure of quantitative analysis. NASA’s real mistake was to rely on quantitative analysis too much.
* * *
• • •
Before ignition, Challenger’s O-rings sat squashed in the joints that connected vertical sections of the booster. At ignition, burning gas came shooting down the booster. The metal walls that connected to form a joint pulled apart for a split second, at which point the rubber O-rings immediately expanded to fill the space and keep the joint sealed. When the O-rings got cold, the rubber hardened and could not expand as quickly. The colder the O-ring, the longer the fraction of a second when the joint was not sealed and burning gas could shoot right through the booster wall. Even so, temperature usually did not matter; the O-rings were protected by a special insulating putty meant to block burning gas from reaching them in the first place. On the seventeen flights with no O-ring problems—akin to the seventeen Carter Racing races with no engine problems—the putty worked perfectly. Those flights provided no information whatsoever about how O-rings might fail, no matter the temperature, because the burning gas could not even get to the O-rings to cause a problem. Sometimes, however, small holes formed in the putty when the joints were assembled. On the seven flights that had O-ring issues, burning gas pushed through the holes in the protective putty and reached the O-rings. Only those seven data points were relevant to how the O-rings could be damaged or fail.
And on those seven shuttle flights—unlike gasket breaks in Carter Racing, which was the same problem every time—the O-ring issues came in two different varieties. The first: erosion. On five flights, burning gas that came shooting down the booster at ignition hit the O-rings and eroded the rubber surface. This was not a life-or-death condition. There was more than enough rubber for the O-ring to do its job. And erosion had nothing at all to do with temperature.
The second variety: blow-by. If the rubber ring did not expand instantly to fully seal the joint at ignition, burning gas “blew by” and could potentially shoot right through the booster wall. Blow-by was a life-or-death condition and, engineers would later learn, dramatically worsened when cool temperatures hardened the O-ring rubber. Two pre-Challenger flights had blow-by, but still returned home safely.
Thiokol engineers who opposed the launch on the emergency prelaunch conference call did not really have twenty-four relevant data points on O-ring failure to work with, as the Carter Racing study indicates. They did not even have seven, like the Harvard students. They had two.
Now what does the chart tell you?
Ironically, Allan McDonald, then director of the rocket-booster project at Morton Thiokol, told me, “Looking only at the relevant data points supported NASA’s [prelaunch] position that it was inconclusive.” There was no 99.4 percent certainty that was missed. The engineers were not poorly educated.
There was other important information the Thiokol engineers presented that could have helped NASA avert disaster. But it was not quantitative, so NASA managers did not accept it. The Carter Racing study teaches that the answer was available, if only engineers looked at the right numbers. In reality, the right numbers did not contain an answer at all. The Challenger decision was truly ambiguous. It was a wicked problem, rife with uncertainty, and outside of previous experience, where demanding more data actually became the problem itself.
* * *
• • •
The infamous emergency conference call convened thirty-four engineers —every manager was also an engineer—in three locations. Thiokol engineer Roger Boisjoly had personally inspected the joints after both flights with blow-by, and presented photographs from each. Following the 75-degree flight, he found a very thin streak of light gray soot beyond an O-ring in the joint, from a tiny amount of gas that had blown by before the O-ring sealed. It was nowhere close to a catastrophic problem. After the 53-degree flight, he found jet-black soot fanned out across a large swath of the joint. A lot of burning gas had blown past that time. In Boisjoly’s opinion, the reason the 53-degree launch looked so much worse was that cool conditions had hardened the O-rings and made them slow to expand and seal at ignition. He was right, but he did not have the data to prove it. “I was asked to quantify my concerns, and I said I couldn’t,” Boisjoly later testified. “I had no data to quantify it, but I did say I knew that it was away from goodness.”
Thanks to an extraordinarily strong technical culture, NASA had developed quantitatively rigorous “flight readiness reviews.” They were productively adversarial, like superforecasting team discussions. Managers grilled engineers and forced them to produce data to back up their assertions. The process had worked remarkably. The space shuttle was the most complex machine ever built, and all twenty-four flights had returned safely. But on the emergency conference call, that same quantitative culture led them astray.
On their engineers’ advice, McDonald and two Thiokol VPs on the call initially supported a no-launch decision. The Challenger had already been cleared, so this was an eleventh-hour reversal. When NASA officials asked Thiokol engineers exactly what temperature range was safe for flight, they recommended setting a limit at 53 degrees, the lower bound of previous experience.
NASA manager Larry Mulloy was flabbergasted. He thought the shuttle was supposed to be cleared to launch from 31 to 99 degrees. A last-minute 53-degree limit was setting an entirely new technical criteria for launches. It had never been discussed, was not backed by quantitative data, and meant that suddenly winter was off-limits for space exploration. Mulloy found it frustrating; he later called it “dumb.”
How had the engineers arrived at that number? “They said because they had flown at 53 degrees before,” a NASA manager reflected, “which is no reason to me. That’s tradition rather than technology.” Boisjoly was asked again for data to support his claim, “and I said I have none other than what is being presented.”
With the conference call at an impasse, a Thiokol VP asked for a five-minute “offline caucus,” during which Thiokol concluded that they had no more data to provide. They returned to the call a half hour later with a new decision: proceed with launch. Their official document read, “temperature data not conclusive on predicting primary O-ring blow-by.”