SGLang-Neuron brings SGLang serving to AWS Trainium
AWS says the plugin is publicly available in a Deep Learning Container, letting teams serve models and generate RL rollouts on Trainium through the standard SGLang interface.
What AWS announced
On 12 November 2026, AWS announced that SGLang-Neuron is publicly available through an AWS Deep Learning Container. The release comes from RadixArk working with Annapurna Labs. In practice, SGLang-Neuron on AWS Trainium lets teams serve models and generate reinforcement learning rollouts with SGLang, using the interface they already use, running on native PyTorch. Trainium is AWS's family of AI chips, which the company positions on economics for training and inference at scale. The news is a supported route for an existing serving stack onto that silicon, not a new chip.
The mechanics are the interesting part. AWS says launching a model follows the SGLang workflow teams already know: start the server, then call the endpoint with the same generation APIs.
How the plugin fits together
SGLang-Neuron is an out-of-tree plugin. Installing it registers the neuron platform through SGLang's plugin entry points, and setting SGLANG_PLATFORM=neuron selects it when the server starts. SGLang core keeps doing what it did before: the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific pieces, namely the models, attention, mixture of experts and the graph runner.
Underneath, TorchNeuron registers Trainium as a native PyTorch device. The model execution path uses standard PyTorch operations, with kernels written against the Neuron Kernel Interface where performance matters. Those operations and kernels are compiled into executable programs with torch.compile, and the graph runner prepares and executes them during serving. Because the plugin sits outside SGLang core, Trainium support and SGLang itself can be developed and iterated on separately.
The post also covers PD disaggregation. In that setup a prefill server computes the prompt KV cache and hands it to a decode server over Elastic Fabric Adapter using NIXL; the decode server produces the remaining tokens. Both servers still run SGLang's scheduler and request lifecycle, while the plugin handles Trainium execution and the cache transfer path.
What the walkthrough shows
The example uses a Trainium3 Amazon EC2 instance with a pytorch-inference-sglang-neuronx container pulled from a public ECR image tagged v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04. Inside the container, the server is launched with SGLANG_PLATFORM=neuron and a set of flags, including tensor parallelism of 8, expert parallelism of 8, an fp8 KV cache, a context length of 10240, a chunked prefill size of 8192, up to 64 concurrent requests, a token budget of 655360 and a static memory fraction of 0.8. The sample model in the post is openai/gpt-oss-120b, and the final step is a curl call to /v1/chat/completions with a temperature of 0 and a 128 token cap.
For anyone who already writes against an SGLang server, the shape of that code does not change. This suggests existing clients and evaluation scripts can be pointed at a Trainium deployment without a rewrite, provided the model is supported.
Limits and open questions
The post does not give pricing, throughput or latency numbers, so there is no published comparison against other accelerators. It does not list which models are supported or validated; it points readers to the repository documentation for that. Region availability for the container and for Trainium3 instances is also not stated, and the walkthrough uses a fixed image tag rather than a general availability statement for future versions. Whether the example configuration is a recommended default or simply one working setup is not said either.
The 8-way tensor and expert parallelism in the sample is my reading of a setup aimed at a multi-chip instance rather than a single device, but the post does not spell out the instance size.
What this means for people building apps
Most people building web apps with AI will not run their own inference cluster, and nothing here changes that. What it does change is the option set for teams whose costs or data rules push them toward self-hosted models on AWS hardware. If you are prototyping an app that calls an AI model, the choice of provider is a configuration decision you can revisit later; Draftship's notes on adding AI to your app cover how a model call fits into an app you build, and the full details of this release are in the AWS announcement post.
Primary sources
- Deploy frontier AI models on AWS Trainium with SGLang-Neuron · AWS Machine Learning Blog, Nov 12, 2026
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.