Knowing when not to guess

About a month ago, I wrote about a specific AI failure where ChatGPT confidently misidentified an image of a scene from Once Upon a Time in the West as coming from Jaws. Recognizing that the genre of pointing and laughing at AI fails got tired years ago, I reflected on my renewed capacity to be disappointed by AI failure—not because the capabilities of the technology did not match the hype, but because AI has actually gotten so good that failure at such a mundane task is genuinely surprising.

Out of curiosity, I tried the task again in a temporary, unpersonalized chat. Once again, ChatGPT (5.6 Sol High) misidentified the source of the image, this time as Sleuth (1972), citing elements not actually present in the scene:

The model even searched 13 websites as part of generating its response. None of those, apparently, was a reverse image search, which would have immediately identified the source of the image. Google, of course, identifies it immediately:

As detailed in the original post, I lifted this image directly from a publicly indexed and accessible website; I didn’t screenshot it from a YouTube clip or something.

Modern multimodal models have an incredible, superhuman breadth of knowledge to draw from for simple tasks like identifying a movie scene from a still. But they still haven’t nailed the judgment component, like trying a simple reverse image search when identification is uncertain. Obviously, the tiniest bit of steering on my part in the prompt would have fixed this problem. But it was a test, and a test I am surprised it is still possible to fail.

Maybe we need a “know when to just Google it” benchmark. (Or Bing, or whatever ChatGPT is actually using for search on the backend these days.)