Trakkr just released an interesting report: they asked five AI assistants for a wide variety of statistics, and then traced 1,285 of the responses back to their original sources to determine whether the numbers could ultimately be supported by a study. Only 40% of claims could be. (Of course, just being able to find a study to support a statistic does not mean it is actually true or applicable, but it’s a start.)
This is not to say the AIs were generally hallucinating sources. In 8% of cases, a source was identified, but with a different value or context than reported. In 40% of cases, a source was named but either did not contain the statistic or was inaccessible or missing. In 11% of cases, the statistic was simply repeated everywhere with no attempt at a credible attribution, common on company blogspam sites that AIs still like to cite. The final 1% was when the rabbit hole ended at pages that would not load.
I have complained before about the poor performance of AIs in distinguishing the credibility of sources (sometimes to amusing/horrifying results). It’s neat to see an attempt to actually quantify this problem.