The provider layer · ↗ mmoustafa.com

Mo Moustafa has a revealing write-up of his experience using OpenRouter for 18 million messages to power his iMessage-based AI agent. He discovered that providers were very much not interchangeable in how they served the same model. For example:

• Same model, very different benchmarks (e.g., deepseek-v4-flash-0731 scored 81% on TAU-Bench first-party vs. 58% on DigitalOcean)
• Some providers ignored certain model features (e.g., some providers would silently drop/ignore vision inputs or reasoning effort)
• Tool calls would sometimes leak into responses
• Empty completions (e.g., some model providers would hand back empty completions with an all-clear 200 status; a single provider might be responsible for most of these failures in a period)

Even pinning to a few of the “best” providers by his own benchmarks did not solve the problem: he kept getting rate limited (429 errors), and one provider stopped serving the model. Availability suffered.