Google Walks Through the Agents CLI Evaluation Loop
A new Google Developers Blog post shows how to run, grade and compare agent evals to catch prompt regressions and test cheaper models.
What Google published
On Oct. 28, 2026, the Google Developers Blog published a walkthrough of the Agents CLI evaluation loop by product manager Pier Paolo Ippolito and solutions specialist Elia Secchi. The premise is simple: an agent that works in a demo is a weekend project, while an agent that keeps working through model upgrades, tool changes and new question shapes needs a loop. Write test cases, run them, grade the traces, read what broke, change the agent, run again. The post calls that cycle hill-climbing, and the Agents CLI evaluation loop is the tooling built around it.
The five commands, and the setup
Agents CLI exposes a small, framework-agnostic set of eval commands. eval generate runs the agent against a dataset of cases. eval grade scores the resulting traces. eval compare diffs two run results side by side. eval analyze clusters failures into themes. eval optimize runs an automated hill-climbing pass on the agent's prompt.
The demo scaffolds a LangChain template that ships with Agents CLI v1.8.0, targeting cloud_run. The template includes a dataset at tests/eval/datasets/basic-dataset.json, an eval_config.yaml and a custom LLM-as-judge metric in response_quality.py. That metric grades each response from 1 to 5 on accuracy, relevance and clarity, using gemini-3.8-flash as the judge. The template's default dataset holds one trivial case, so the authors expand it to four: a greeting, a San Francisco weather question the mock tool answers with a hardcoded foggy response, a Rome weather question the tool falls back on, and an off-topic question about the 2024 World Series.
Baseline, then a deliberate regression
The first graded run reports a mean of 4.0 out of 5 across four cases, with a standard deviation of 1.41. The authors note that spread is a healthier starting point than a flat 5.0, which would suggest the dataset is too easy to distinguish anything.
To show the loop working, they paste the wrong system prompt into the agent: a chef persona meant for a different codebase. The agent still compiles, lints, type-checks and runs, so nothing before the eval catches it. Re-running generate and grade drops the mean to 2.75 with a standard deviation of 0.96. The per-case breakdown is where the detail lives: greeting fell from 5.0 to 4.0 after pitching a recipe unprompted, sf_weather fell from 4.0 to 2.0 because the chef persona pulled attention off the registered weather tool and the model invented conditions, rome_weather stayed at 2.0, and off_topic fell from 5.0 to 3.0 after answering the question and appending a recipe anyway.
That sf_weather case is the argument for end-to-end evals in one example. The tool is still registered and the code still runs. Only grading the response in context shows the model stopped calling it.
Reading a model swap correctly
With the helpful prompt restored, the authors swap gemini-3.8-flash for gemini-2.5-flash and re-run. The mean drops from 4.00 to 2.75, the same size as the chef-prompt regression. The per-case diff tells a different story: greeting held at 5.0, sf_weather slipped to 3.0 on slightly less polished phrasing, rome_weather held at 2.0, and the entire visible drop came from off_topic, which fell to 1.0 because the cheaper model hedged that the event had not happened yet, which the post attributes to a training cutoff.
The takeaway is that the mean alone would rank the cheaper model as bad as the broken prompt, while the breakdown suggests it is reasonable for most of the workload if off-topic questions about recent events are rare. The authors add two caveats: prompts tuned for a stronger model can lose capability when ported to a weaker one, so the fair comparison pairs each model with its own tuned prompt, and the judge model is held constant across comparisons because changing it is a separate experiment.
Limits and open questions
Agents CLI does not ship a scheduler. The post describes dropping the eval commands into cron or Cloud Scheduler and writing results to a dated folder under artifacts/scheduled_runs/. The introduction also promises threshold-based alerting for tracking quality over time, but the excerpt ends partway through the scheduling script.
Open questions remain: what those thresholds and alerts actually hook into, what the judge calls cost at volume, how large a dataset needs to be before the signal is trustworthy, and what eval optimize changes to a prompt in practice beyond the one-line description.
What it means for building apps with AI
The practical shift is treating a system prompt as a scored artifact rather than a string. A one-line edit can move a tool call, a tone and a scope decision in three different directions, and only per-case scores show which. If you are iterating on agent instructions in your own project, Draftship's guide to writing effective prompts covers the input side; this post covers how to measure whether the change helped.
Primary sources
- Hill-Climbing Agent Quality with Agents CLI Evaluation · Google Developers Blog
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.