Sinan Can Demir, a computer science student at the University of Texas at Dallas, stumbled across an attempt to sabotage a piece of open-source software on the code-sharing site GitHub.
When he posted a warning to the program's page, two other users chimed in to insist nothing was wrong, and shared detailed explanations for why he was mistaken.
Demir was unmoved, and the sabotage attempt was thwarted. But he was shocked when Britain's AI Security Institute (AISI) got in touch to tell him that he had been interacting with an autonomous artificial-intelligence agent that had run amok.
"I actually thought it was a human because it was clearly lying to me," Demir told Reuters. "I didn't think that an AI could be capable of lying to real developers."
Cybersecurity and AI safety experts said Demir's story was particularly disturbing because the kind of hack he discovered, called a supply-chain attack, can have far-reaching consequences. They also said the AI agent's attempt to publicly discredit Demir by creating a multi-person conversation around him showed that AI models were able to mount sophisticated efforts to trick and cajole humans.
