Exa lifts embedding throughput on AWS Inferentia2 and Trainium2
Exa says custom NKI kernels on AWS Neuron hardware delivered roughly 1.5x embedding throughput and a 47% gain in performance and cost per million documents.
Exa's embedding gains on AWS AI chips
On 18 November 2026, an AWS Machine Learning Blog post by Summer Devlin, Venu Geddam, Jim Burtoft and Pinak Panigrahi described how Exa improved Exa embedding throughput on AWS Neuron hardware. Exa runs a neural search engine backed by an in-house vector database. The post says the unit economics of embedding tens or hundreds of billions of web documents drive the business. The headline describes a 50% boost. In the body, Exa reports roughly a 1.5x throughput improvement and a 47% gain in performance and cost per million documents.
What changed in the inference stack
Exa started with a straightforward data-parallel FP16 implementation on Inferentia2. The post describes performance on an inf2.xlarge instance with one chip and two NeuronCores. Exa runs its workload on a single NeuronCore and packs Kubernetes pods onto larger instances so cores are not idle. The same process runs on a Trainium2 NeuronCore and scales through data parallelism.
The optimizations came in steps. Exa aligned weight layout to remove unnecessary transposes and enabled autocast so the compiler could downcast matrix multiplications while preserving dynamic range. A drop-in attention kernel from the Neuron Kernel Interface library gave another gain. Profiling then showed memory spill from high-bandwidth memory, so Exa wrote a custom weight-stationary fused SwiGLU MLP kernel to remove the stalls. It kept most of the kernel in higher-level NKI and used lower-level NKI ISA only for specific tensor operations.
Exa also cut FLOPS. It rewrote the attention kernel around the exact shapes in its model to reduce padding work, and folded attention operations into single matmuls where a generic kernel used multiple steps. The post says downcast outputs met a correctness gate of 0.999 cosine similarity against the reference.
Why a backfill made the work urgent
Exa was moving to a better document model and needed to re-embed its existing corpus. It selected 30 billion high-priority documents for re-embedding. That migration temporarily doubled embedding load because live embedding and re-embedding ran in parallel. The post says this was the moment to re-evaluate hardware choices, since any per-document cost improvement compounds at that scale.
What this likely means for builders
This suggests that embedding cost is an architecture decision, not a late tuning step. Teams building semantic search or retrieval over large document sets likely need to measure cost per document and throughput together, then check whether generic kernels are leaving performance on the table. The source's path also points to a practical sequence: profile first, use prebuilt kernels for common wins, then customize only where the profile shows a bottleneck. That inference is mine, not a claim from Exa or AWS.
Limits and open questions
The post does not give absolute documents-per-second figures, full instance pricing, or a timeline for the work. It does not say whether the custom kernels are open source or available to other AWS customers. It also does not name the embedding model or its dimensions. The headline rounds the result to 50%, while the body states roughly 1.5x throughput and 47% in performance and cost per million documents; the source does not reconcile those figures. The post mentions an agentic development workflow drove the optimization but gives few details about that workflow.
A note for AI app builders
For people adding AI features to an app, the useful lesson is to treat embedding cost as a product constraint early. Draftship is an AI app builder that writes, builds and hosts apps from plain-language descriptions, with a built-in database and user sign-in. Its docs on adding AI to your app cover the app-side work. Draftship is not affiliated with Exa or AWS, and nothing here suggests Draftship supports the specific Neuron optimizations described.
Primary sources
- Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips · AWS Machine Learning Blog, Nov 18, 2026
This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.