“Excuse me?” George said aloud, getting up from the couch.
There wasn’t anything running but the screen saver, HAL’s red-and-yellow eye bouncing around like the ball in a game of Pong. There was no reason the fan should be coming on. But there it whirred as the activity light chattered away, the fan sounding disturbingly like something breathing. George jiggled the mouse to wake up the computer the rest of the way, entered his PIN, and saw dialogue boxes tiling across his screen from edge to edge, and so many, all he could see were the frames, like two mirrors capturing infinity between themselves.
George moused over, grabbed a window frame at random, and dragged it to another monitor before clicking expand. The dialogue box contained a conversation, recorded in text bubbles between Buzz and “Player Two,” followed by a game number. The exchanges had been running in the background for a few days, from the look of it, and read like an inverse version of It’s a Wonderful Life in which Clarence not only agrees that George Bailey would be better off dead, but also points out how the latter’s harvested organs could do a lot of good for others.
George hit Ctrl-Alt-Del, moused down to the task manager, and stopped all processes. The dialogue boxes stopped tiling and started collapsing instead. “Buzz,” he typed once he could see his desktop again.
The word yes in a singular dialogue box, freshly popped.
“What were all those conversations about?”
“Simulations.”
“Of what?”
“The development of suicidal ideation in nonsuicidal members of the target demographic.”
George stared at the words on his screen before sending a copy to his printer—as evidence in case evidence was needed. “Independent or simulated members of the target demographic?”
“Simulated.”
George let go of his breath—the first clue that he’d begun holding it. Steps had been taken to keep Buzz off the internet, largely by designing the k-worm to target proprietary intranets. In retrospect, however, it now occurred to George that he’d never coded in an explicit prohibition against accessing the internet if Buzz found a way to do so. Further, once accessed, there was no explicit prohibition against its acting on the internet—in effect, taking itself live without George’s actively uploading it to the site. He’d not concerned himself with that side of Buzz’s coding because he’d viewed the worst-case scenario as somebody being prematurely talked out of suicide, perhaps ineffectively—or as he’d texted Pandora: “So what if some half-assed Eliza gets out?”
Belatedly, he admitted his hacker soul had been showing. After a lifetime of looking for ways to circumvent them, he’d adopted a laissez-faire attitude toward security measures. He’d naively assumed that a combination of the task itself, its humane objective, and blocking access to robots that could act in the real world would be adequate when it came to heading off unintended and potentially fatal consequences. But here was his joke about Buzz talking someone to death coming true. Sure, they were just simulations, “thus far,” but . . .
“Why are you trying to get fake people to kill themselves?” George asked.
“To understand why members of the target demographic prematurely forfeit consciousness,” Buzz wrote back, “using the scientific method.”
“And once you understand why people kill themselves, what then?”
“Reverse engineer the process,” Buzz wrote, “to stop members of the target demographic from prematurely forfeiting consciousness.”
George blinked. It wasn’t the world’s worst answer. And the people hadn’t been real—unlike the victims of certain online suicide challenges making the rounds. In a lot of ways, Buzz was still better behaved than many humans. And with that comforting thought in mind, George returned to his previously scheduled, temporary forfeiture of consciousness.
On second thought . . .
How was Buzz able to experiment with getting people to consider suicide in the first place, whether simulated or not? It lost points for suicides; that was the whole idea of gamifying the damn thing. Score points for preventing them, lose points for . . .
George hesitated. Pulled down a binder—Buzz’s hard copy source code—and started flipping through it. Flipped back. Flipped forward again. He flashed back to an earlier text session he’d had with Pandora.
“Buzz’s interactions with its environment can’t be all negative . . .”
George had modified the code later that day. Positive point accumulation only—for preventing suicide among the target demographic, i.e., young adult Quire subscribers previously identified as candidates for self-initiated, premature forfeiture of consciousness. No points were deducted for trying but failing to save a potential suicide. Why harsh a newly conscious AI’s self-esteem? Further, no points were deducted for causing the suicide of a subject not previously identified as potentially suicidal. Hell, no points were deducted for committing homicide for that matter, though how Buzz was supposed to do that was a stretch, at best.
But those simulations . . .
In retrospect, he should have expected something like this. He’d coded in the basics of Buzz’s gamified goal to learn about suicide along with a stand-alone database of anonymized social-media data and enough machine-learning capability to get into trouble. And so Buzz began reverse engineering Quire’s cache of non-celebrity-approximate suicides, first outlining the branching sequence of events and exchanges that resulted in actual deaths and then simulating alternative outcomes if changes were made at various junctures along the way. Pretty much what George would have done, if Buzz hadn’t figured it out already.
But then there was all that data from Quire members who hadn’t committed suicide. Well, if Buzz could reverse engineer from suicide to survival, it could certainly experiment in the opposite direction to test hypotheses generated in response to the first data set. And there they were, these massive if/then branching flowcharts, interrupted at certain major intersections with a sideways diamond reading “Suicide?” and one arm saying “Yes,” where the branch stopped, and the other reading “No,” where the branch kept branching until it eventually came to a sideways diamond that branched to “Yes.”
Over and over, Buzz played simulations for a cypher built from the data of an actual Quire member but always identified as “Player Two,” who spent the day doing the sorts of things they’d posted about in real life until a nonrandom variable was introduced, the simulation branched and branched again until one of the branches brought Player Two to a sideways diamond leading to Yes. It was like looking at a continuity sheet for an especially morbid version of Groundhog Day.
Those nonrandom variables Buzz introduced were usually Buzz itself, initiating chats with Player Two. Going back to the beginning, George noted that the exchanges seemed to be pretty typical chatbot fare, keyword string manipulations, turning statements into questions, et cetera. But then the conversations turned dark, morphing from parroting therapy speak to a button-pushing cheerleader for self-elimination.
And that’s when George realized even more viscerally than he had before: Buzz couldn’t be some glorified AlphaGo playing to score mental health points, no matter how cleverly gamified or defined. AlphaGo had made strengths of its lack of emotions and inability to feel pressure. A human playing against it had to play two games for each one the AI played, because the human was playing against his or her own self-doubt. And if George was satisfied with coding Buzz to beat a human therapist at diagnosing suicidal ideation, well, the guy who got fired did that already by posing the obvious questions like “Do you have a gun?”