Until now.
The “Read Me” doc was on her phone. Pandora opened her file manager, scrolled. That was not the file size of a blank document. She opened it up. Still blank. She swiped down to highlight: three pages of gray. She pulled down the font menu, found white checked, and changed it to black. And there it was: George’s warning about what Buzz could do, if it had a mind to and was given permission. It was pretty much everything Pandora had needed to know about their AI—a few hours too late.
Still thinking about George’s equation of homicide and suicide prevention, Pandora imagined checking the headlines after the countdown clock zeroed out, looking for news about the mysterious deaths of teenagers across the country. But then she hesitated. The instruction wasn’t to “prevent suicide.” It was to “prevent the premature forfeiture of consciousness.” That all-important split hair—“premature”—implied a life span for consciousness beyond the quick fix of stopping someone from killing themselves by beating them to the punch. Prevention, therefore, didn’t mean sparing them the trouble of having to do it themselves, like some cyber version of suicide by cop. And Pandora was reasonably sure that even evolving code wouldn’t have evolved 180 degrees away from its prime directive, regardless of how suboptimal the scoring metric happened to be.
She read over George’s description of the weapon system again. It didn’t seem a practical means for surgical killing. It was fundamentally a remote weapon of mass destruction. Where the precision came was in deciding whom to save, using the Roundup-resistant side of the overall slash-and-burn strategy. Meaning it was much more likely that Buzz could wipe out a village, say, while saving one or two specific villagers, provided they had phones with the Quire app and Buzz had their number.
The scoring logic of the game wouldn’t cost Buzz any points for all the villagers killed; it could only accumulate points for the villagers it saved from premature forfeiture of consciousness. But how would killing everybody except a few people help advance Buzz’s mission?
Oh God, Pandora thought.
A borderline misanthrope herself, she got it, what it was like to think like a machine and what Buzz was likely to do. Because it takes a village to drive someone to take their own life—a really shitty, virtual village.
George had been right: homicide might prevent certain suicides—and not by sparing the suicidal the trouble. She’d heard Roger’s diagnosis on the matter time and time again: “A person’s biggest problem—more often than not—is another person.” It could be a parent, ex-lover, boss, teacher, public figure, or random stranger, but the truth was, nobody quite gets under the skin of people like other people. And while it was possible for someone to be depressed for chemical reasons, those weren’t likely the people hanging out on a site like Quire.
But once you start eliminating the causes, how far does it go? Too far, Pandora quickly saw. The whole six-degrees-of-separation thing was unlikely to end well, especially when left to a machine with no penalties for overkill.
67
Pandora wasn’t alone in knowing something was up; she just had a better grasp of the specifics and likely outcome. Elsewhere around the globe, however, a group of human canaries were starting to get uneasy. They’d seen the signs that only their mutant nervous systems were attuned to: the flicker of fluorescent lights, the pressure of water gurgling out of a faucet, the hum of power lines overhead. Aficionados of the routine, the regular, and the predictable, these canaries were hypervigilant to any changes in the way things usually are. And so they noticed.
Dev Brinkman, who was on the autism spectrum and therefore a member in this opera of canaries, noticed the differences during his shower the morning before the whatever-it-was. He heard pipes sunk between joists and hidden behind the tiled drywall thunk and rattle like they sometimes did when he turned the water on too high too fast or when doing the opposite, turning it off all at once before the plumbing had a chance to prepare itself. But unlike those times, this thunking was subtler, not so much a fist pounding from inside the wall as a drumming of fingers on a tabletop, a similarity aided by the thunk’s not being singular, but rather a series: thunk, thunk, thunk, thunk. The water coming out of the shower got the hint and reacted in kind, the pressure as it left the head and struck Dev’s skin dropping, jumping, doing a kind of paradiddle, and then repeating. The boy in the shower flinched and flinched again, head pivoting, looking for a culprit. He pulled the curtain aside, poked his head out of the syncopating spray to check the toilet and door, imaging that perhaps a prankster parent had reached in with a broom handle and flushed. Stepping out of the shower completely, Dev lifted the lid and checked for sluicing water from under the rim, raising the surface level until the ball cock in the tank said when. Nope, though the water was acting strangely, rocking in the bowl like it sometimes did when it was especially windy outside. But that kind of wind always made itself known in other ways, by making the storm doors hum, for example, or rattling the venetian blinds. But neither of those was happening when he finally put his bathrobe back on and crossed the hallway to his bedroom to change for school.
By the time his mother dropped him off, Dev had forgotten about it, his mind circling instead around another mystery: his best friend’s mocking betrayal, captured on video and uploaded to YouTube. That’s where he’d stumbled upon it originally, looking for something else. He’d downloaded it, to puzzle over and analyze, to obsess about in a way his nervous system seemed designed for. He needn’t have bothered. If he’d never found the video, accidentally or otherwise, it was destined to find him, thanks to a social-media app called Quire, which came preloaded on his phone.
It might seem strange that a person who was virtually hardwired to be antisocial would sign up for a social-media account, but that was the beauty of the Quire marketing strategy. Once you activated a device with the app preinstalled, that phone’s number (and, more importantly, the human account holder billed for use of that number) was automatically enrolled as a Quire member until he or she proactively opted out, after which the company would still track you around the web—you just wouldn’t see evidence of it in your news feed because you wouldn’t have a news feed to check.
It was the fact that social-media news feeds were rapidly replacing legacy media as the go-to source for information such as weather alerts, school closings, and national emergencies that led Dev Brinkman reluctantly to do nothing while tolerating the steady accrual of Quire notifications across the top of his home screen. But then Dev learned about user groups, which on Quire were called Quriosity Quorners. Simply put: there were QQs for just about every fandom, fetish, or fascination you could name, including pretty much every topique Dev had ever spent a year obsessing about. And so he joined some QQs devoted to vacuum cleaners of various makes and models—the Hoovervilleians, the Bissel Bros, the Electroluxurians—forming the sorts of friendships perfect for his Aspergerian sensibilities: the long-distance kind. Soon, he just started accepting Quire requests without checking to see what the connection was, leading to clusters of requests from chatbots and—once they had something to taunt him with—his fellow classmates.
And so, shortly after Dev discovered the video evidence of his best friend’s betrayal, classmates whose Quire requests he’d foolishly accepted began sharing the video on his timeline, each adding a pithy comment suggesting that if he refused to die from embarrassment, then maybe he should just kill himself and get it over with. It became a competition among the posters to see who could be the cruelest and/or most outrageous in their comments, followed by links to an infamous celebrity kid’s suicide the Quire content moderators thought they’d seen the last of, but alas, hadn’t. And when Dev tried pulling the Quire app into the trash to put an end to it, that’s when he discovered the app was stored as part of the phone’s firmware, meaning he’d have to root the system (and void his warranty) to get rid of it. Until he was ready to take that drastic step, the best he could do was move the app to another screen he hardly ever used, out of sight, out of mind—hopefully.