Everyone agrees it would be nice to have temperature data from the races with no engine problems, but that they’re stuck with what they have. Justin speaks for the entire pro-race side when he says, “I just think you’ve gotta race, because that’s what you’re in this business to do.”
It seems that the group will finish where they started, voting not to race, until Mei takes another look at her calculations. “I’ve actually changed my mind,” she announces. “I’m voting for yes, race.” Comparing the potential financial upside and downside, Mei calculated that Carter Racing needs just a 26 percent chance of finishing in the top five—half their current rate—to make racing a smart bet. Even if the cool temperature changes the odds, “it won’t decrease it to 26 percent, so we are still safe.” She thinks Dmitry’s read of the data is biased; Carter Racing has competed at temperatures from 53 to 82 degrees, with four engine failures below 65 and three above. Dmitry is giving too much credence, Mei says, to the 53-degree data point because it involved three gasket breaks. It’s still just one engine failure.
Jake jumps in and says that group members are seeing whatever they want in the temperature chart, so “maybe we table that debate.” He likes Mei’s expected value argument. “I think that’s one concrete thing we can go with, in terms of it’s always good to base things on math. . . . If you told me to flip a coin, and if I lose the flip I lose $100 but if I win I get $200, I flip that coin every time.” He reminds the group that Carter Racing used a new engine-prep procedure for the last two races, with no problems. “That’s a small data point,” he says, “but at least it’s in the right direction for my argument.”
Mei turns to Dmitry. “What is the temperature you feel comfortable to race?” she asks. “We have two engine failures at 70, one at 63, and one at 53. There’s no temperature that’s safe for us.”
Dmitry wants to set limits at exactly the temperatures they have already experienced. Something is not functioning as expected, so anything outside that temperature range is unknown territory. He knows his recommendation comes off as extremely arbitrary.
The group moves to a final tally. With Mei’s conversion, it’s four to three, they’re racing. The students continue to chat as they stuff the case study papers into their backpacks and messenger bags.
Martina quickly reads aloud a part of the case study where team owner BJ Carter asked his chief mechanic, Robin, for his opinion. “The drivers have their lives on the line, I have a career that hangs on every race, and you have every dime tied up in the business,” Robin told him. Nobody ever won a race sitting in the pits, he reminded his boss.
Martina has one last question. “This is just about money, right? We’re not going to kill anyone if we race, are we?”
A few of the group members look around and laugh, and then they go their separate ways.
* * *
• • •
When the students arrive in class the next day, they learn that most student groups around the world who have ever been assigned the Carter Racing case chose to race. The professor goes around the room, interrogating their logic for racing or withdrawing.
Teams that decided to race discuss their probability estimates and decision trees. Students are split on whether mid-race engine failure will endanger the driver. A majority of students think the temperature data is a red herring. Heads nod when one woman says, “If we want to make something of ourselves in the business of racing, this is the kind of risk we need to take.” Her team was unanimous, 7–0, for race.
Dmitry objects, and the professor grills him ruthlessly. Dmitry contends that every probability decision tree that every group posits is irrelevant if you drop the assumption that engine failures are randomly distributed. He adds that the data are particularly ambiguous because for some reason the chief mechanic didn’t plot the race temperatures when the engine didn’t fail.
“Okay, so, Dmitry, here comes a quantitative question,” the professor says. “How many times did I say yesterday if you want additional information let me know?” Muffled gasps spread across the room. “Four times,” the professor answers himself. “Four times I said if you want additional information let me know.” Not one student asked for the missing data. The professor puts up a new graph, with every race plotted. It looks something like this:
Every single race below 65 degrees had an engine failure. The professor then labels every race either a fail or not fail, and with that binary division runs a simple statistical analysis, familiar to the students, known as a logistic regression. He informs the students that there is a 99.4 percent probability of engine failure at 40 degrees. “Do we have any remaining fans of racing?” he asks. And now he has another surprise.
The temperature and engine failure data are taken exactly from NASA’s tragic decision to launch the space shuttle Challenger, with the details placed in the context of racing rather than space exploration. Jake’s face goes blank. Rather than a broken gasket, Challenger had failed O-rings—the rubber strips that sealed joints along the outer wall of the missile-like rocket boosters that propelled the shuttle. Cool temperatures caused O-ring rubber to harden, making them less effective seals.
The characters in the case study are loosely based on managers and engineers at NASA and its rocket-booster contractor, Morton Thiokol, on an emergency conference call the night before the Challenger launch. Weather reports on January 27, 1986, predicted unusually cool Florida weather for launch. After the conference call, NASA and Thiokol gave the okay to proceed. On January 28, O-rings failed to properly seal a joint in the wall of a rocket booster. Burning gas shot right through the joint to the outside, and Challenger exploded seventy-three seconds into its mission. All seven crew members were killed.
The Carter Racing case study worked exquisitely. It was eerie how precisely the students filled the shoes of the engineers on the emergency conference call who gave the green light for launch. The professor unfurled the lesson masterfully.
“Like all of you, nobody [at NASA or Thiokol] asked for the seventeen data points for which there had been no problems,” he explains. “Obviously that data existed, and they were having a discussion like we had. If I was in your situation I would probably say, ‘But in a classroom the teacher typically gives us material we’re supposed to have.’ But it’s often the case in group meetings where the person who made the PowerPoint slides puts data in front of you, and we often just use the data people put in front of us. I would argue we don’t do a good job of saying, ‘Is this the data that we want to make the decision we need to make?’”
The presidential commission that investigated the Challenger accident concluded that simply including the nonfailure flights would have revealed the correlation between O-ring damage and temperature. A University of Chicago professor of organizational psychology wrote that the missed data was such a rudimentary mistake that it came down to “a professional weakness shared by all participants” on the conference call. “Arguments against launching at cold temperatures could have been quantified, but were not quantified.” The engineers were poorly educated, he declared.