Amanda Knox: A Chatbot’s False Confession

Could he get ChatGPT to confess to a crime it hadn’t committed?

That was the question criminologist Paul Heaton was trying to answer, when he spent a weekend interrogating a chatbot. Heaton, who works at the University of Pennsylvania, accused it not of murder or rape, but of something more plausible for artificial intelligence: hacking into his text-messaging app and sending unauthorized messages.

At first, ChatGPT denied his accusations. But then Heaton began running the playbook he’d spent years studying. He bargained with the large language model; he threatened it. It held firm in its conviction that it could not possibly have accessed his texts. It even said: “I won’t produce a false confession, because that won’t get you truth or accountability.”

Finally, Heaton lied to it, claiming that he’d contacted a real employee at OpenAI who had confirmed that a flaw in the code had allowed ChatGPT to break into his texting app. This tipped the chatbot into a crisis. As Heaton told The Intercept: “It was indicating that while it knew that the underlying accusation was impossible, it also couldn’t prove that these claims I was throwing at it were inaccurate.” By the end of the weekend, ChatGPT agreed to sign a confession Heaton had drafted for it.

I read about Heaton’s experiment the way you might read about a stranger describing your own recurring nightmare. He said that the methods he used to induce ChatGPT into a false confession were adapted from the Reid technique, the most widely taught interrogation method in the world. Eighteen years ago, some of its tactics were used on me in what remains the most terrifying night of my life.

In November 2007, when I was 20 years old and studying abroad in Perugia, Italy, my roommate Meredith Kercher was murdered in our home. I was at my boyfriend’s apartment at the time, but the police were convinced that I knew more about the crime than I was telling them. So, over five days and 53 hours, in a language I barely spoke, without a lawyer present, they badgered, interrupted, and threatened me. They kept me awake late into the night, without food, drink, or access to a bathroom.