SageMaker HyperPod adds a global inference gateway
AWS now offers one managed endpoint that routes inference across HyperPod clusters in one Region or several, with automatic failover.
What AWS announced
On 20 November 2026, AWS announced the Amazon SageMaker HyperPod Global Inference Gateway, a SageMaker-managed endpoint that routes inference requests across the HyperPod clusters a team registers, whether those clusters sit in one AWS Region or across several. The announcement post describes it as the second tier of the routing architecture AWS introduced earlier with the per-cluster Inference Gateway.
That earlier tier solved routing inside a single cluster. It reads signals such as key-value cache utilization and queue depth to place each request on the best-suited pod. What it did not do was route between clusters, so a request could wait in a queue on a saturated cluster while GPUs elsewhere sat idle.
The problem the gateway addresses
AWS illustrates the gap with a team serving four models across five HyperPod clusters in two Regions. Without any gateway, clients track 20 endpoints, one per model per cluster. The per-cluster Inference Gateway cuts that to five, but each client still has to choose a cluster itself. That choice brings uneven load during peaks, manual failover when a cluster breaks, and custom load balancers, DNS and cross-Region networking to maintain.
The Global Inference Gateway collapses the client's view to a single endpoint. AWS lists its benefits as putting idle GPUs to work during peaks, failing over automatically, using one endpoint for everything, skipping the custom networking, and keeping traffic private by default.
How it is set up
Setup involves two decisions. First, you publish each cluster by turning on the spec.globalRouting block in the cluster's existing InferenceGatewayConfig. The HyperPod Inference controller then creates a Network Load Balancer in front of that cluster's Inference Gateway, publishes a VPC endpoint service on it, and uploads the listener certificate to your TLS bucket. Those resources live in your account. Second, you call CreateGlobalInferenceGateway with the clusters you want routed across, and SageMaker returns one endpoint. If you list a cluster before publishing it, the call fails with a message telling you what to fix.
At request time, the client signs with SigV4 and sends to the gateway's regional endpoint. The gateway authenticates the caller and picks a cluster, AWS PrivateLink carries the request there, and the cluster's own Inference Gateway routes it to a model pod. AWS operates the routing fleet that terminates TLS, authenticates callers and scores clusters, plus the private connectivity, certificate validation through an execution role you create, health checks and failover, and rate limiting. A service-managed metrics publisher with its own IAM role reports each cluster's load back to the gateway.
Bagelcode's Jinhyung Park, VP of AI Platform, said in the announcement that beta tests routed requests across two AWS Regions through one endpoint and kept serving when a cluster could no longer serve the model, giving a small platform team cross-Region failover without building a routing layer.
Limits and open questions
The source does not give pricing, availability status, supported Regions, or how many clusters a single gateway can route across. It also mentions that teams decide how to weight routing but does not explain the weighting options. The Global Inference Gateway does not replace the per-cluster tier; pod-level placement still happens inside each cluster.
What it means for building apps with AI
This is infrastructure for teams that run their own model servers on HyperPod and already pay for GPU capacity spread across clusters. For most people building a web app, the model call is an API request to a hosted provider, and the routing question is handled upstream. If you are adding AI features to an app, the guide to adding AI in your app covers the app-side work. Running clusters across Regions is a different tier of problem, and the announcement does not say Draftship supports any of it.
Primary sources
- Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters · AWS Machine Learning Blog, Nov 20, 2026
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.