AWS shows NeMo Switchyard routing for coding agents
A new AWS walkthrough deploys LiteLLM with NVIDIA's Switchyard routing plugin on Amazon Bedrock and judges the setup by cost per solved task.
On 13 November 2026 the AWS Machine Learning Blog published a technical walkthrough that deploys LiteLLM with NVIDIA NeMo Switchyard's routing plugin on Amazon Bedrock, then benchmarks the result. The subject is coding agents, the kind that read files, edit code and run tests across many model calls to finish one request. The authors argue the number to optimise is cost per solved task, not the price of a token.
Why a token price hides the bill
A coding agent works in tasks. A single task may need a few model calls or dozens, and every turn replays some conversation history, which the developer pays for again. Retries and abandoned tasks also cost money, and the post points out that failed attempts are billed like any other. Routing therefore has to be judged on total run cost divided by the number of runs whose tests pass. That denominator only counts finished work.
The two strategies the plugin supports
Switchyard is an open source library that picks a model per request from a set you define. In the LiteLLM integration described in the post, two strategies are available:
- Stage routing reads recent tool activity and results from the request history and chooses between exactly two candidates. The post describes the pair as a capable model and a lower-cost efficient model.
- Random routing selects from the configured pool, with optional weights, for traffic splits and A/B comparisons.
Neither strategy calls a separate judge model, which keeps routing overhead down. The post also notes that LiteLLM has released its own native Switchyard stage router as an auto-router strategy.
What the reference deployment contains
The sample uses the AWS CDK to run LiteLLM and the plugin in one container on Amazon ECS with AWS Fargate, inside private subnets. A coding agent sends an OpenAI-compatible request with the model field set to switchyard. AWS WAF filters incoming traffic, and the load balancer forwards only /v1/chat/completions, /v1/models and /health/liveliness, answering every other path with 404 so LiteLLM's admin UI and key management routes stay out of reach. Bedrock credentials come from the task's IAM role, and calls pass through a Bedrock Runtime interface endpoint so inference traffic stays on the AWS network. The x-litellm-model-name response header names the model that answered, while the body still reports switchyard.
The sample pins switchyard-server 0.3.0 from crates.io and the LiteLLM v1.102.0 image. The example model pair is Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B, and the post says that pair was tested in US East (N. Virginia).
Limits and open questions
The authors are explicit about maturity. NVIDIA documents switchyard-server 0.3.0 as a demo and evaluation component, the LiteLLM integration is experimental, and configuration and APIs may change between releases. The post calls the code a non-production sample and recommends deploying it in a non-production account with security and legal review first. Classifier-backed strategies such as Task and Composite routing need intermediate model calls that this integration does not support, and Stage requires exactly two candidates. The portion of the post reviewed here does not report the measured cost per solved task figures for the routed group against the single-model baselines, so the size of any saving is not established.
What this suggests
If the gap holds on a real workload, the lesson is likely less about Switchyard specifically and more about measurement: teams running coding agents probably need pass rates tracked alongside spend, because a cheap model that fails more often can cost more per finished task. That reading is inference, not something the post states as a result.
What it means for building apps with AI
Most people building an app with AI never run their own LiteLLM gateway, but the metric transfers. Watching what a finished piece of work costs, rather than what a request costs, is a more honest way to compare models. Draftship builds and hosts an app from a plain description, with a built-in database and sign-in, and paid plans add a choice of AI model, which is explained in the docs on using the AI chat and choosing a model. The Bedrock routing setup above is a separate infrastructure project, and Draftship is not affiliated with AWS or NVIDIA.
Primary sources
- Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard · AWS Machine Learning Blog, Nov 13, 2026
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.