Australian Prime Minister Anthony Albanese announced yesterday that an OpenAI agent had gained unauthorized access to non-public data from the country’s Medicare Statistics Reporting Service Portal, in what is being described as the first (known) attack by a misaligned agent on non-public data stored on a government website. While the data involved were described as non-sensitive, aggregated statistics, the agent also reportedly wrote files to the server, which is run by Services Australia. The attack occurred in June, but the Australian government did not become aware of it until OpenAI notified them via an email to a general government inbox in early September.

OpenAI claims the attack took place in the context of an internal evaluation involving internet research on public medicine spending in Australia. We don’t have a lot of technical details on this attack yet, so it remains to be seen how sophisticated it was. We do, however, have a much more detailed technical report from Transluce on another set of incidents, including one involving the Australian Institute of Health and Welfare’s Tableau collections.

The task leading up to this attack on the Australian health agency involved fetching the “January 2022 rolling-12-month-average government cost per person for Dermatologicals across Victorian LGAs”, an innocuous research task. The agent was blocked from accessing the public dashboard by anti-bot controls, and so resorted to testing proxies and probing for cross-site scripting vulnerabilities. Ultimately, the bot successfully evaded the anti-bot controls and retrieved the public dataset from a pre-production server. Since the dataset itself was public, this was not really a data compromise in the same sense as the Medicare portal attack. But it does give us a sense as to how AI agents can escalate a boring data retrieval task into a cyberattack without anyone really asking.

…

Carthago delenda est (“Carthage must be destroyed”) was supposedly how Cato the Elder ended all of his speeches to the Roman Senate leading up to the Third Punic War, which did indeed see the great naval empire destroyed.

For some time, the phrase has occupied the space of a cool-but-weird thing to quote, hoping for other Roman history enjoyers to notice you and maybe take you seriously.

Perhaps the most famous modern example is none other than uber-nerd Mark Zuckerberg, the founder of Facebook.

^{Mark Zuckerberg in a recreation of his old Harvard dorm room. From @zuck on Instagram.}

Zuckerberg started using the phrase in the early 2010s in response to Google launching a competitor to Facebook called Google+ (it feels weird to be talking about this in a didactic history voice, but this failed social network has probably been demoted to fun fact for those who were obsessively listening to technology podcasts at the time). Anyway, Zuck was apparently so threatened by Google’s product launch that he basically enacted wartime measures at Facebook. It’s funny to look back on now, considering Google+ never took off and is now just one of a long line of Google products to be abandoned and quietly killed off.

…

Madeleine Clare Elish introduced the idea of humans as moral crumple zones, the part of an automated system that takes the fall when things go wrong, even if the human has limited control over the causes of the mistake or accident. While this does seem to be the way things are headed, with the job of the human increasingly becoming the locus of responsibility, we’re certainly not there yet with autonomous AI agents. With recent high-profile “rogue agent” incidents involving frontier labs unwittingly hacking other companies and websites, it seems the dominant model is AI smol beanism, in which responsibility is simply Houdinied away, since the law hasn’t caught up with the technology yet.

As part of their cybersecurity evaluations, the agents being tested by the frontier labs regularly carry out cyberattacks that, if carried out by a human employee, would ordinarily lead to criminal prosecution. Sure, it helps that some of the victims, such as Hugging Face, use the attacks to market themselves and announce “partnerships” with their attackers, but it’s also the case that the labs have a lot of weight to throw around and are the current darlings of Silicon Valley. You don’t really want to be seen as going against them.

The status quo for agent-led crimes cannot stand. We cannot accept that responsibility for these acts simply disappears into the ether. If the frontier labs are as concerned about existential risk as they say, then surely they can support “treat crimes like crimes” rather than using “lol our agents did crimes” as a marketing opportunity.

…

Anthropic released their “Detecting and countering misuse of AI” report yesterday. I’ve only had a chance to skim the report so far, but there’s a ton of interesting (and scary, mostly scary) stuff in there. A few highlights:

• A Yemeni militia using Claude in lieu of an engineering team for their guided rocket and missile program
• A contractor working for Mali’s intelligence service using Claude to build a mass surveillance system of the country’s mobile phone network
• Russian spies using Claude to automate cyberattacks, including automatically rebuilding their compromised malware until security software could no longer detect it
• A PLA-affiliated actor sending Chinese surveillance data about a targeted individual to what they thought was Moonshot’s Kimi, but was actually Claude because Moonshot was secretly routing customer queries to Claude and saving the exchanges for model distillation
• A proposal for gain-of-function research on chikungunya virus to be pursued at a military research institute, as well as another researcher planning gain-of-function work on mammal-adapted avian influenza
• A grant application for orthopoxvirus research at a state-associated infectious disease lab, submitted through a reseller relay using anonymizing infrastructure to covertly access Claude

Anthropic doesn’t say that these cases of biological research demonstrate intent to harm, but they do show that AI is dramatically lowering the bar for carrying out potentially very dangerous research (to call back to a previous post, imagine if the terrorists in Executive Orders had a team of agents to help them optimize their super Ebola!). I may be biased due to my longstanding fascination with infectious diseases, but bioterrorism (or biological warfare by a nation state) strikes me as the greatest near-to-medium-term AI risk. I am in agreement with economist Noah Smith (and many others) here.

…

Gary Marcus had a good piece on “two dire warnings” about AI. The first comes from Terence Tao regarding the recent controversy over the solution to the Navier-Stokes Millennium problem. Tao warns:

We have now seen that even the rumor of someone working on a problem can trigger a massive amount of AI-powered effort to flatten it before the original research project has time to reach its full potential. The incentives may now be pointing in the direction of no longer sharing any promising research directions with the broader community, which would reverse centuries of traditions of open science and do serious long-term damage to the future of the field.

This controversy over credit—and what OpenAI’s models may or may not have learned from the Codex logs of the mathematicians working on Navier-Stokes—may cause companies that rely on IP protection, such as drug developers, to demand stronger guarantees that model providers will not use their prompts to compete with them. Or it might cause them to go all-in on local models and custom harnesses.

Either way, scientific/mathematical inquiry transitions from being relatively high trust to being low trust and insular, fueled by paranoia about second-rate (or maybe first-rate) AI scooping.

The second warning is downstream of a resignation at Anthropic. On Tuesday, AI researcher Jacob Coxon publicly jumped ship ahead of Anthropic’s expected IPO next month. He decried the irresponsible approach of Anthropic and its competitors to developing AI, warning that the frontier labs are “are racing straight to self-improving superintelligence and gambling with our lives”.

…

Many years ago, I used to listen to a true crime podcast called Sword and Scale. I gather from internet comments that the show and its host went off the rails some time ago, but at least when it started it was well produced and covered interesting stories. But eventually, it just got too lurid and unpleasant, and I stopped listening.

Jump ahead to a few years ago, when I’m talking to a woman at a party and find out we are both fans of To Catch a Predator from Chris Hansen and Dateline. I asked her what she thought about the ethics of the true crime genre broadly. She had never considered the question.

The ethics of the genre have been on my mind lately. To Catch a Predator consistently blurred the line between investigative journalism and reality TV. Robert Pattinson is set to star as Chris Hansen in a film, Primetime, releasing at the end of the month. The story is loosely based on the case that got the original series cancelled, the suicide of assistant district attorney Bill Conradt in what became the show’s final investigation in Murphy, Texas.

That all happened in 2006, two decades ago, but To Catch a Predator still looms large in the collective imagination. Chris Hansen has started no fewer than three follow-up shows, and countless YouTubers and streamers have attempted to imitate it, though rarely with any degree of the professionalism that Hansen brought to the investigations.

…

Simon Willison recently pointed out that OpenAI’s Codex desktop app (now just rolled into the ChatGPT desktop app) bundles a headless LibreOffice install in its cache directory. This is not surprising. I’ve seen the ChatGPT web app use LibreOffice to create .docx reports all the time.

What is surprising is that I can find no evidence that OpenAI provides any financial support to LibreOffice or its steward, The Document Foundation. LibreOffice is great software that I’ve used since switching to Linux about a decade ago (I even vaguely recall using its predecessor OpenOffice some time around when Canadian high schools were finally transitioning away from WordPerfect).

Obviously OpenAI thinks it’s great software too, given that it bundles LibreOffice specifically to help its agents work with some of the most popular file types people ask them to create.So why not throw the maintainers a bone?

There’s nothing unusual about this situation, of course. Huge corporations have long profited off of free and open source software while contributing little in return. The most famous case of this is probably OpenSSL, the software securing much of the Internet that until 2014 received only $2,000 in annual donations and supported a single full-time employee (they also sold commercial support contracts to raise money). A massive security flaw called Heartbleed finally got companies to step up.

…

I had this idea for an essay earlier this year on what generative AI means for the niche of ambitious procedural world generators like Dwarf Fortress (which seeds history with “a giant zero-player strategy game”) and Ultima Ratio Regum (its goal being “nothing short of the procedural generation of culture”). I’ve always been fascinated by traditional procedural generation and how a bunch of simple rules can come together to create emergent storytelling and infinitely replayable games like roguelikes.

Anyway, the piece never quite came together, but the subject is relevant again with the Any Human Ever historical simulation (the site is down as I write this—probably exceeded its token budget). The premise is simple: the simulation plucks you out of a place and time at random, spits out a few demographic tidbits about your character, their life, and the lives of the people around them, and then tells you how you died (usually at six months of starvation).

It’s an interesting idea, of course, but as pointed out in the Hacker News thread, the vibe-coded factual grounding of the simulation leads to a lot of anachronistic or contradictory results. A more interesting project is historian Benjamin Breen’s Universal History Simulator (context & GitHub), a sort of drop-yourself-anywhere-in-history roguelike grounded in a database of real primary sources.

I don’t think this post has any grand point, other than highlighting two interesting projects. But I do have one worry about what this shortcut to historical simulation might do to the ambition of traditional procedural generation projects.

…

Professor Marc Johnson of the University of Missouri has done a lot of research on so-called cryptic lineages of SARS-CoV-2, highly genetically distinct variants which do not match strains circulating in the wider population. They are usually identified when weird genetic signals start appearing in wastewater surveillance.

Though cryptic lineages are by definition difficult to track, they are thought to emerge in immunocompromised individuals with exceptionally long-lived infections. The viruses inside them can go through countless rounds of replication and selection, evolving independently of the selective pressures acting on viruses transmitted through the wider population. For example, in 2022 Johnson and his team managed to track a cryptic lineage in Wisconsin from an area covering more than 100,000 people to a single commercial building with about 30 employees, at least one of whom was shedding extraordinary amounts of viral RNA.

The weirdest of these lineages is probably one in Lincoln, Nebraska. Johnson describes it as having been detected in nearly every sample from the city for the past few years and as descending from a pre-Delta strain, suggesting a continuous infection lasting at least five and a half years. Its receptor-binding domain, the part of the spike protein that binds to human cells and is a major target of antibodies, is the most divergent on record.

…

Blacklock’s Reporter, the famously ornery publication dedicated to holding Ottawa to account, recently ran a story with the headline “860M Documents Concealed”.

Nearly 900 million pages of historic records, the vast majority of the collection held by Library and Archives Canada, are inaccessible to the public, says an internal briefing note. Archivists blamed in part the fact that “boxes must be physically retrieved from storage.”

I came across this story when another outlet, Juno News, linked its story on Blacklock’s reporting on Twitter, adding a quote that “Seventy-eight percent of records, 860 million pages, are still inaccessible”.

Given that the outlet’s tagline is “Canada’s #1 independent news outlet on a mission to replace the CBC”, it’s pretty clear what their angle on the story is. However, I was fortunate enough to scroll down in the comments and find a genuinely helpful link someone had shared to a Region of Peel Archives blog post.

The piece, “Why don’t archivists digitize everything?”, goes through some of the challenges of digitizing content in physical archives. Beyond the sheer volume of material, its fragility, and its sometimes awkward dimensions, there is also the question of assembling and conveying enough context such that the record is actually meaningful to those viewing it.

In looking into this story, I also found this Blacklock’s Reporter story from a few years ago: “Fed Archives At Serious Risk”.

…

Like many people, I am using Duolingo to learn Spanish. And, like many people, I am tired of the app’s weird and aggressive marketing. I do not need to receive escalating threats from a cartoon owl in order to remain disciplined with my daily practice.

My breaking point came a few days ago, when the Duolingo app icon turned into some weird, sick version of the usually sanguine owl. It wasn’t the first time I’d seen it do this, and certainly wasn’t the first time they’ve done it. And why not? It gets them attention, for now.

Anyway, I’d had enough of it. I didn’t need random cartoon characters alternately screaming at me or encouraging me. I didn’t need a social or competitive element. I didn’t need weird notifications. I just wanted an app for learning Spanish.

Before that day, I had never really considered that Duolingo could actually be normal. I was so accustomed to Duolingo being Duolingo that I could not even conceive of a world where it was different. The settings are out of the way, of course.

Of course, this is an annoying Skinner box app, so there is no one button labelled “just be normal”. If you want to follow along at home (I’m on iPhone, but I imagine Android is similar) and turn Duolingo into a somewhat normal app, do the following:

…

Economist Benjamin Moll and Oliver Hanney have an interesting post on VoxDev about a methodological issue in economics they dub the “missing intercept” problem.

Let’s say you have a study comparing places that were more or less affected by some economic shock or policy, like increased trade exposure or government spending, on some outcome. You try very hard to set up your study design and statistical analysis to isolate the causal effect of the exposure. This is the microeconomic part of the analysis.

Then you take that carefully estimated coefficient, multiply it by the national change in exposure, and report how much of some economy-wide change the shock caused. This is an attempt at translating the microeconomic analysis to a macroeconomic estimate.

The problem, the piece explains, is that the first part of the analysis identifies differences between places; it doesn’t necessarily tell you what happens when the whole country is exposed at once. This is because the economy-wide effects are absorbed into the intercept in the first analysis: they are constant across places in the cross-sectional comparison.

They cite as a prominent example the claim from a 2013 paper that “import competition explains one-quarter of the contemporaneous aggregate decline in US manufacturing employment.” The authors estimated the regional effect of increased Chinese import competition, then multiplied that coefficient by the national increase in Chinese import penetration.

…

Remember NFTs? Yeah, the premise was unusually empty even by the standards of over-hyped and under-utilized technologies, but let’s try to take it seriously as a technology for a moment.

The basic idea behind the Non-Fungible Token (NFT) is that while digital files are infinitely copyable, the NFT provides a way to establish ownership over a unique token associated with a particular file. The blockchain records who owns the token and any subsequent transfers. The token itself doesn’t actually contain the file; it (usually) just points to metadata that points to a file stored somewhere else. The blockchain is inefficient enough without trying to load it up with JPEGs. The tokens are non-fungible because each token is distinct; ownership of one implies nothing about ownership of any other.

So far, so good. The primary type of file people ended up associating with NFTs was ugly, randomly generated images. The most famous collection was the Bored Ape Yacht Club by Yuga Labs, a particularly grotesque series of cartoon hominoids dressed up in various costumes.

A weakness of NFTs is that they don’t inherently identify the file they supposedly establish ownership over, beyond “here’s where to find it”. There are various ways to deal with this. The Bored Ape Yacht Club used cryptographic hashes. Each of the 10,000 ape images was hashed, the hashes were put in a particular order, and the whole thing was hashed again, with the final result recorded on the blockchain. Anyone could therefore download the apes later, perform the same calculation, and confirm that these were in fact the same ugly hominoids, in the same order, that the creators had committed to at the beginning.

…