TrnRT scales 1M-token LLMs on Trainium 2
RunAnywhere's open-source TrnRT inference engine serves high-throughput large language models on AWS Trainium 2 instances using custom Neuron Kernel Interface megakernels and an OpenAI-compatible API. The architecture pairs two tensor-parallel engines per trn2.48xlarge instance with prefix caching and 1M-token context support for models like GLM-5.3-Flash.
While Nvidia hardware dominates production LLM serving, custom runtimes like TrnRT demonstrate that AWS Trainium 2 can achieve competitive throughput when software bypasses generic compilers in favor of fused, hardware-aware megakernels.
- –NKI megakernels fuse attention, linear layers, and MoE routing into single tile-level dispatches, shaving decode latency from 32.2 ms down to 24.2 ms per step.
- –Benchmarks on the AgentX suite report 3.57 million tokens per minute at $0.040 per million tokens on spot pricing, providing a compelling economic alternative to GPU clusters.
- –A hybrid radix-tree prefix cache bridges device HBM and host memory to sustain low time-to-first-token across 1M-token agent contexts.
- –Upfront compile friction remains the primary hurdle, requiring hours of host-side graph compilation and limiting initial turnkey support to GLM-5.3-Flash.
DISCOVERED
6h ago
2026-11-19
PUBLISHED
7h ago
2026-11-19
RELEVANCE
AUTHOR
Github Awesome