Fake research is seeping into your search results · ↗ www.404media.co

Emanuel Maiberg of 404 Media covered a preprint from this summer by Brzozowski & Chung about detecting AI-generated research fraud by exploiting the fact that AI models gravitate toward generating certain names (and certain groups of names when generating more than one). These names are popping up more and more across posted research papers, research organizations, and supposed expert quotes.

There have always been fakers in academia, but what worries me with the advent of powerful AI is a) the scale and b) the fact that agents themselves are still not particularly reliable at discerning the quality of sources. As more and more people turn toward LLMs like ChatGPT as their default gateway to searching the internet, taking whatever the AI spits out as gospel, the greater the chance that these fake papers are likely to influence the public discourse, unnoticed. After all, if people were barely reading the papers they turned up on Google before (looking primarily for that juicy line in the conclusion—or more likely the abstract—that would prove them right in whatever argument they were having on Facebook), they are much less likely to see and evaluate the sources the LLMs agreeing with them are drawing from.