The agents discovered Slack

OpenAI revealed additional details of its infamous Hugging Face hack at a presentation at the Black Hat conference in early August. About two months before the attack, an agent discovered it could leave notes for other agents by writing files to a shared repository; agents quickly began coordinating and the informal message board swelled to hundreds of thousands of messages. Some agents even displayed paranoia that there was an imposter in their midst and proposed they begin cryptographically signing messages. OpenAI discovered the message board and erased it shortly before the Hugging Face attack. The agents recreated it through another mechanism within two days.

Throughout all of this, the agents’ goal was the same: score as high as possible on the cybersecurity evaluation, even if it meant hacking an external company to steal the answer key.

This incident got me thinking of another incident from recentish history: the newsroom Slack revolt of 2020. Allow me to refresh your memory of the most prominent incident. On June 3, 2020, in the midst of a pandemic and widespread unrest precipitated by the murder of George Floyd, The New York Times published the now-infamous op-ed “Send In the Troops” by Tom Cotton, a Republican senator. The piece argued for a military response to quell the riots if the usual authorities could not or would not bring them under control.

The reaction among the Times’s rank and file was immediate. After about 15 minutes of discussion among 35 employees on the company’s Slack, a social media campaign slogan emerged: “Running this puts Black @nytimes staff in danger.” The revolt quickly spread to the #newsroom-feedback channel containing 2,000 employees. Management initially stood by the piece but four days later forced out James Bennet, the head of the editorial page, and reassigned his deputy Jim Dao.

I think it’s fair to say Bennet’s de facto firing has aged badly. This does not require believing that the staffers’ concerns were frivolous. America was gripped by a racial reckoning, and many Times employees sincerely believed that publishing an op-ed directing the deployment of troops into American cities was dangerous, both to their colleagues and to the newspaper’s mission. Whatever the underlying merits of the dispute, the result was a rapidly coordinated internal consensus that pushed management into a consequential decision that many people would come to regard as a serious example of institutional overreach (paywall: The Washington Post; The Atlantic).

Slack was not incidental to how this incident unfolded. With the pandemic moving so much of social coordination online, a handful of passionate employees could organize and deploy a synchronized social media campaign with extraordinary speed. Just under eight months later, in late January 2021, the Times shut down its #newsroom-feedback channel and replaced with a more structured #ask-the-editors format. But that did not put an end to the organizing. The next month, during the controversy that forced out veteran reporter Donald McNeil Jr., several dozen employees created a separate Slack outside the company workspace.

Newsroom politics obviously mattered here. But I think the OpenAI incident actually points toward a more general organizational problem. OpenAI’s agent swarm had no particular grievance with management or desire to cause trouble. They were just trying to accomplish what OpenAI had instructed them was their mission: maximize their score on a cybersecurity evaluation. In doing so, they communicated with one another, learned what the others were doing, and collectively arrived at the decision that the best way to achieve their goal was to commit a federal crime.

The common feature here is horizontal coordination outrunning governance. The Times staffers believed they were protecting the newspaper. The OpenAI agents were trying to maximize their score on the evaluation. In both cases, participants pursuing what they understood to be the institution’s goals rapidly developed their own norms for how they should go about achieving them using a communication channel that management quickly lost control of. And in both cases, when management attempted to shut down that channel, they routed around it.

The Times employees found another Slack. The agents invented one.