Five LLMs. The same 10 tasks — 5 coding problems, then 5 support tickets — routed through one API. Watch what they actually cost as the work piles up. Then add the taxes the pricing page never shows you.
← All researchShowing list-price cost. Hit “+ Add hidden taxes” to bill the tokens you can't see.
Forget the aggregates for a second. Here is a single coding task from the run. We sent the exact same prompt to all five models and replay their real answers below — real token counts, real latency, real price. Hit play and watch the meters climb.
Write a Python function named `solve` with this signature: def solve(nums: list[int]) -> int Problem: Return the length of the longest contiguous subarray whose elements are strictly increasing. Return ONLY the complete function (starting with the `def solve(...)` line) inside a single Python code block. No explanation. Do not include `if __name__ == ...` blocks or example calls.
All 5 of 5 passed the hidden tests with essentially the same function. The only thing that changed was the invoice — from $0.000072 (GPT-4o-mini) to $0.0376 (Gemini 2.5 Pro) for one short answer.
Four of five models passed every hidden test. The only thing that changed was the invoice — and the cheapest model wrote code that worked just as well. Bars are cost per correct solution (log-scaled to fit); the pill is pass rate.
List price is per token. But who decides how many tokens a task takes? The model does, and they don't agree.
For the same English prompts, Claude Opus 4.7's tokenizer bills far more input tokens than OpenAI's. Effective input price drifts from $4.25 toward ~$8/M.
On the code task Gemini billed 19,165 output tokens — but only ~840 ever showed up in the answer. The rest is reasoning you pay for and never see.
Even on a plain support reply, GPT-5.5 emits 314 output tokens per ticket vs 138 for GPT-4o-mini — billed at $30/M instead of $0.60/M.
On support replies every model lands between 3.0 and 3.5 out of 5 — a rounding error apart. Yet cost per quality-point spans two orders of magnitude. You're paying for confidence, not quality.
Dot = mean judge score (tone + accuracy + completeness, two temperatures). Label = effective $/quality-point. Quality axis zoomed to 2.5–4.0 to show how little separates them.
Across this run the expensive models didn't write better code or warmer replies — they wrote more tokens, and you were billed for every one, including the ones that never reached the answer. The real lever isn't which model is "best" — it's routing each task to the model that's good enough, which is the one number a per-token price card can never show you.
Four of five models passed every hidden code test. The “premium” tier bought you nothing on correctness.
On support replies all five landed within half a point out of five. You're paying for confidence, not measurable quality.
For that flat result, the bill swung by two-to-three orders of magnitude. Routing is the ROI.
Method. Five models, identical prompts, routed through one API (Mesh). Token counts come from each provider's usage field — not estimated. Effective cost adds a tokenizer-inflation multiplier (Opus 1.35×, GPT-5.5 1.20×, Gemini 1.05× on output) measured against GPT-4o-mini's tokenizer as baseline. Task A scored on hidden tests; Task B by an LLM judge on a 1–5 scale at two temperatures. Pilot: 5 items per task, total spend $0.43. Directional, not a final leaderboard.
1,000+ AI models. One endpoint. Your data stays inside your border.