This is a neat paper from Owen Queen and James Zou of Standford University: “The shelf life of LLM findings”. The authors revisited 50 widely cited claims about LLMs and found that most either no longer held or had significantly weakened across 20 models. These claims operate as bits of folk wisdom people carry about with them and shape public understanding of LLM capabilities, especially among those that are not frequent users of AI. It is instructive to see quantified just how quickly many of these claims became out of date.
For example, until mid-2026 models would frequently fabricate scholarly citations, but this behaviour has much reduced among most recent models (though even if only 1% of citations are fabricated, models still do this far, far more often than humans, who are prone to misrepresenting the content of citations, rather than inventing references wholesale).
Another slightly dubious “expired” claim is that models no longer refuse obviously harmless requests. This may be true according to the methods of the original 2023 paper, but certainly we still have high-profile refusals like Fable being unwilling to discuss the powerhouse of the cell and other basic biological concepts. So this bit of wisdom is not entirely out-of-date.
Interestingly, three of their 50 claims had actually strengthened relative to the original research: models praise what their users like, LLM judges favor their own writing, and random-number requests favor a few answers.
I guess you really do have to read Hacker News every day to keep on top of the LLM zeitgeist. The rest of us are doomed to walking around discussing LLMs like it’s still 2023.