AWS ships a KV caching agent skill for self-hosted LLMs
AWS's Agent Toolkit for AWS now includes a distributed KV caching skill that lets coding agents size, set up and validate an ElastiCache cache for self-hosted LLM inference.
Compute, hosting and deployment news that affects AI-built apps.
AWS's Agent Toolkit for AWS now includes a distributed KV caching skill that lets coding agents size, set up and validate an ElastiCache cache for self-hosted LLM inference.
AWS announced the SageMaker HyperPod Global Inference Gateway, a single managed endpoint that routes inference requests across clusters and Regions with automatic failover.
Exa optimized embedding inference on AWS Inferentia2 and Trainium2 with NKI kernels, reporting roughly 1.5x throughput and a 47% gain in performance and cost per million documents.
SGLang-Neuron is now publicly available in an AWS Deep Learning Container, bringing the SGLang serving workflow and PD disaggregation to AWS Trainium on native PyTorch.
Google Cloud has made an Anthropic-managed Claude deployment option generally available, running alongside the Google-managed option at the same models and pricing.
API Gateway streaming support is now in Public Preview, carrying SSE, WebSockets, gRPC and chunked responses behind the same policies as your other APIs.