Two things AI could break · ↗ garymarcus.substack.com

Gary Marcus had a good piece on “two dire warnings” about AI. The first comes from Terence Tao regarding the recent controversy over the solution to the Navier-Stokes Millennium problem. Tao warns:

We have now seen that even the rumor of someone working on a problem can trigger a massive amount of AI-powered effort to flatten it before the original research project has time to reach its full potential. The incentives may now be pointing in the direction of no longer sharing any promising research directions with the broader community, which would reverse centuries of traditions of open science and do serious long-term damage to the future of the field.

This controversy over credit—and what OpenAI’s models may or may not have learned from the Codex logs of the mathematicians working on Navier-Stokes—may cause companies that rely on IP protection, such as drug developers, to demand stronger guarantees that model providers will not use their prompts to compete with them. Or it might cause them to go all-in on local models and custom harnesses.

Either way, scientific/mathematical inquiry transitions from being relatively high trust to being low trust and insular, fueled by paranoia about second-rate (or maybe first-rate) AI scooping.

The second warning is downstream of a resignation at Anthropic. On Tuesday, AI researcher Jacob Coxon publicly jumped ship ahead of Anthropic’s expected IPO next month. He decried the irresponsible approach of Anthropic and its competitors to developing AI, warning that the frontier labs are “are racing straight to self-improving superintelligence and gambling with our lives”.

Anthropic’s Alignment Science lead Evan Hubinger immediately jumped in to confirm that the company’s leadership really did believe AI could kill all humans, that he personally pegged the probability at greater than ten percent within the next decade, and that Anthropic was “trying its best” but currently has no plan to stop it.

I really have no insight to add here that hasn’t been said a million times. But…this isn’t normal. What other industry’s comms strategy is, “There’s a good chance the thing we’re building will kill everyone, we don’t know how to prevent that, now give us all of the money in the world so we can race to build it”? Why are the guys building the doomsday device looking around and telling us, “Someone should really do something about that, huh?”

Things are gonna get weird.