Skip to content

AWS ships a KV caching agent skill for self-hosted LLMs

A new Agent Toolkit skill lets coding agents check, size and configure a shared ElastiCache KV cache for self-hosted inference.

Draftship Editorial4 min read

The short version

On 20 November 2026, AWS published a post describing a distributed KV caching skill for its Agent Toolkit for AWS. The skill lets a coding agent work out whether a self-hosted LLM workload would benefit from a shared key-value cache, estimate the savings, and set the cache up on Amazon ElastiCache. AWS says distributed KV caching can cut self-hosted inference cost by up to 40% and time to first token by up to 80%.

What the source confirms

The problem is repeated GPU work. Every turn, a self-hosted model recomputes the key-value attention states for the whole context, a step AWS calls prefill, because each server only holds a small local cache. In the walkthrough, the agent is handed a fleet serving a legal assistant with 64K-token contexts on p5 instances, where a returning conversation takes 13 seconds to produce its first token. Distributed KV caching adds a shared tier beside the GPU fleet so any server can reuse that state across sessions, routing shifts and server failures. The announcement post covers the benchmarks and technical detail behind those figures.

How the skill works

The skill runs inside a coding agent that supports Agent Skills. AWS names Kiro and Claude Code, and says it also works with OpenAI Codex. The agent moves through four stages: a fit check before any spend, an estimate from the user's own traffic, setup on the existing serving stack, and a correctness check.

For the fit check, the agent makes a read-only describe-instances call against the GPU fleet and counts Elastic Fabric Adapter interfaces. It then asks for what it cannot see: engine, model, tensor-parallel size, KV data type, server price, and how often prompts repeat. If the workload does not fit, it stops and explains why.

Sizing uses KVTrace, which AWS says estimates benefits from your own model and traffic before any infrastructure is deployed, at no additional charge. The agent also weighs a cutover point. With short prompts, loading cached KV state can take longer than recomputing it, so the cache only pays off above a certain prompt length and repeat rate.

For setup, the agent adds the LMCache connector to an existing vLLM or SGLang engine, including vLLM workers running under llm-d or Ray Serve. It sets every server in the fleet to one KV ID mechanism, a hash and seed, and to an explicit KV data type, because mismatched settings produce keys other servers cannot find.

Before production traffic, the agent empties the local tiers and checks that two cache-loaded runs return the same output. AWS notes that a layout mismatch can leave healthy hit counters and wrong answers. The skill also asks for explicit approval before creating, changing or deleting resources.

What it changes for app builders

The multi-tenant detail is the part worth reading twice. AWS says a shared cache can let one tenant infer from response times whether another tenant sent the same text, so cases that need isolation get separate caches, and tenants that share a cache add their own ID to the KV hash.

The agent also checks cheaper fixes first. If prefix caching was switched off, turning it back on can speed up back-to-back turns, and if the GPU prefix cache hit rate is already above 90% with session affinity, a shared cache may not be needed at all.

Limits and open questions

The 40% and 80% figures are both described as up to, and AWS says benefits depend on model, traffic patterns and routing choices. The prerequisites are narrow: GPU instances that support EFA RDMA write such as p5, LMCache 0.5.8rc1 or later with vLLM 0.11.1 or later or SGLang 0.5.13 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. In the sizing example, the agent recommended two cache.i8ge.48xlarge nodes for eight p5 instances serving 64,000-token prompts to a multi-turn retrieval-augmented generation assistant. The text reviewed gives no node pricing and no list of supported regions, so the real cost of the shared tier is still an open question.

What this means for building apps with AI

If your app calls a hosted model API, this changes nothing for you. It matters when you run the weights yourself and your users come back to long threads, which is exactly the pattern that makes self-hosting expensive. For readers who build on Draftship, the useful takeaway is the sizing logic: caching only wins past a cutover point, and a high existing hit rate can make a shared tier redundant. If you are adding a model call to an app, the guide to adding AI to your app covers the app-side wiring. Draftship is an AI app builder that writes, builds and hosts apps from a plain-language description, with a built-in database, user sign-in and hosting, and it is not affiliated with AWS.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.