Google tests cheap classifier routing across Gemini models
A 50-question coding-agent benchmark argues cost per successful task beats price per token, and that simple routers need real classifiers behind them.
What Google measured
On Nov. 3, 2026, the Google Developers Blog published a benchmark on cost per task routing: whether a cheap classifier can send each coding task to the least expensive Gemini model and thinking level that still solves it. Sergio Villani, writing for Google Cloud AI, argues that falling per-token prices do not control what autonomous agents cost. The metric he puts first is total spend across every loop iteration, tool call, retry and internal thinking pass, divided by the tasks that pass verification. The published write-up tests a 50-question, auto-graded coding-agent benchmark against live Gemini and Jev APIs.
The router sits in front of Google Antigravity's model calls and uses Jev, a structured-output classifier from TypeSafe that answers in a single forward pass instead of generating text: a Choice from a list, a numeric Score, or a Noul, a 0 to 1 probability for a binary statement. Google states it has no partnership with TypeSafe and that the post is not an endorsement.
Where cheap heuristics break
Google's first attempt was one broad Choice question, simple or complex. On 16 tasks it routed nearly everything to simple, including work that needed an LRU cache trace or modular exponentiation. One atomic Noul question did better: 13 of 16 in agreement with hand labels on the first run, 14 of 16 on the second.
A zero-cost alternative, a Python heuristic matching words like sequence and modulo and escalating any prompt over 35 words, scored 15 of 16 (93.8%) on the 16 development questions it was written against. Frozen and run on 34 unseen tasks across seven buckets, it dropped 23.2 percentage points to 70.6%. Long but easy prompts, such as pulling a port from a Kubernetes YAML manifest, triggered false escalations. Jev classified all 34 unseen tasks with 100% agreement, reaching 48 of 50 (96.0%) overall.
Output caps are not thinking budgets
Comparing gemini-3.8-flash at maxOutputTokens=300 with the same model at 4096, thinkingConfig left at the API default, Google measured almost identical thinking tokens on the 27 tasks Jev labelled SIMPLE. The capped run cost slightly more on easy work and broke 7 of 25 hard tasks: six never answered, one answered incorrectly. A router that sent easy tasks to the capped model and hard ones to normal Flash got all 50 right, but cost slightly more and ran slower than Flash alone.
What beat no router at all
Gemini 3.8 Flash on low thinking, with no router, solved all 50. The Jev router matched it by sending everything to Flash-Lite first and escalating what it flagged as complex, and Google reports it can reduce both cost and response time versus running low thinking everywhere. Flash-Lite alone answered simple tasks in half a second but missed 6 of 25 hard ones, all of which Jev escalated with high confidence. The keyword heuristic came out cheaper only because it was wrong: 5 hard tasks reached Flash-Lite and 2 failed.
Limits and open questions
The post names its limits: one run per setup, 50 prompts with hand-assigned difficulty, simple automatic grading, and no dollar figures, only relative comparisons. Routing also pulls against context caching, which cuts the cost of resending a stored block by 75% to 90% but belongs to one model, so switching between Flash-Lite and Flash per turn means two caches or no discount. Google suggests routing short, self-contained work such as log parsing and schema checks, and keeping context-heavy turns on one model. It also notes the Pro configuration declined two short code-security review prompts that Flash answered, and that DiffusionGemma, an open-weights model, can run the same Choice, Score and Noul questions in your own Cloud project, billed by GPU-hour.
What this suggests for app builders
This suggests the test for any model choice is cost per finished task on your own prompts, not a token cap or a keyword rule. Most people using an AI app builder are not running a classifier in front of several models. Draftship's paid plans do include a choice of AI model, covered in the docs on choosing a model in the AI chat. Draftship builds and hosts apps with a built-in database and sign-in, and is not affiliated with Google or TypeSafe.
Primary sources
- Cost per task: Using cheap classifiers to route across Gemini models and thinking levels · Google Developers Blog
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.