RunAnywhere releases TrnRT, an open‑source LLM inference engine for AWS Trainium
TrnRT lets developers serve 1‑million‑token context LLMs on a single AWS Trainium2 instance with OpenAI‑compatible API, promising lower latency and cost.
Source: Hacker News — front pageTrnRT, an open‑source inference engine for large language models, was announced on Hacker News on 17 Nov 2026. It enables developers to run models with up to a 1 million‑token context on a single AWS Trainium2 instance while exposing an OpenAI‑compatible API, a combination that could reshape cost and latency expectations for LLM serving.
What is TrnRT and why does it matter?
TrnRT is a Python‑based engine that compiles model graphs into NKI (Neuron Kernel Interface) megakernels and serves them from an AWS trn2.48xlarge box. According to the project README, it supports "NKI megakernels, 1M‑token context, OpenAI‑compatible API" and currently ships with GLM‑5.3‑Flash, a 320‑billion‑parameter mixture‑of‑experts model that runs with FP8 precision. The open‑source nature means anyone can inspect, modify, or extend the code, unlike many proprietary serving stacks.
How does TrnRT achieve high‑performance inference?
The engine fuses roughly one kernel per half‑layer, handling KDA attention, a fused DSA layer, multi‑head cache (mHC) and MoE feed‑forward networks in a single megakernel. This reduces kernel launch overhead and keeps the working set in the SBUF (scratch buffer) of each NeuronCore. A prefix cache is implemented as a radix tree over FP8 key‑value pages, with a host‑memory tier for evicted pages and state checkpoints, allowing session‑affine routing and rapid context reuse.
What hardware and model support does TrnRT provide?
TrnRT targets the AWS Trainium2 family, specifically the trn2.48xlarge instance that contains 16 Trainium chips, each exposing two engines with tensor‑parallelism over 32 logical NeuronCores. The current supported model is GLM‑5.3‑Flash (320B MoE, 18B active, FP8, 1M‑token context). The repository notes that additional open models are "coming soon" and that the architecture is designed to accommodate new families with minimal changes (see docs/ARCHITECTURE.md).
Benchmarks and cost efficiency
The authors measured real‑world coding‑agent traffic (AgentX) on a single trn2.48xlarge box. In the default batch‑12 configuration the engine processed 28.9 requests per minute, handling 3.57 M tokens per minute, with a median decode speed of 22.3 tokens / s per user and a first‑token latency of 6.53 seconds. The cost per million tokens on spot instances ($8.60 / h) was $0.040, while a capacity block ($35.76 / h) raised the cost to $0.167 per million tokens. These figures suggest that TrnRT can deliver sub‑$0.05 / M‑token pricing at scale, a notable improvement over many GPU‑based serving solutions.
Getting started: quickstart steps
- Clone and build – Install Docker, then run
docker build -f docker/Dockerfile … -t trnrt .on a machine with Neuron SDK 2.32 and Python 3.13. - Compile the cache – Using any x86_64 host, execute the
trnrt compilecommands to generate shape, decision, capture and strip artifacts. This step can take several hours but does not require Trainium hardware. - Deploy to Trainium – Copy the cache and NKI intermediate files to the trn2.48xlarge instance, download the GLM‑5.3‑Flash model via
hf download, then launch the container withdocker run --privileged … trnrt serve --model /model --port 8000. - Call the service – Any OpenAI client can now target
http://localhost:8000/v1. The README shows a Python snippet that sends a chat completion request and demonstrates thesession_idparameter for prefix‑cache reuse.
The repository also provides scripts for batch sizes 4, 8 and 12, detailed latency tables, and a full test suite that runs on CPU‑only hardware.
Using TrnRT with Grafy
Developers can experiment with TrnRT directly from Grafy Code (/code). The integrated terminal lets you clone the repository, run Docker builds, and launch the compile cache without leaving the workspace. Once the service is running, you can open a Grafy Chat (/chat) session, configure the OpenAI client endpoint to the local TrnRT URL, and interact with the model just like any hosted ChatGPT‑style API. For deeper investigation of the engine’s documentation, the Grafy Reader (/reader) app can ingest the Markdown files in the docs/ folder, enabling natural‑language search across the architecture and benchmark sections.
Frequently asked questions
How does TrnRT differ from other LLM serving stacks?
TrnRT targets AWS Trainium hardware exclusively, using NKI megakernels to fuse layer operations and a radix‑tree prefix cache. This design reduces kernel launch overhead and enables 1 M‑token contexts on a single box, whereas many GPU‑based stacks require multiple servers to reach similar context lengths.
Can I serve models other than GLM‑5.3‑Flash?
The current release only lists GLM‑5.3‑Flash as fully supported, but the architecture is built to accommodate additional open models. The documentation (docs/ARCHITECTURE.md) outlines the steps needed to add a new model, and the team promises more models "coming soon".
What are the cost implications of using TrnRT on Trainium?
On AWS spot pricing ($8.60 / h) the engine costs about $0.040 per million tokens at batch‑12. Using a capacity block ($35.76 / h) raises the cost to $0.167 per million tokens. These rates are lower than many GPU‑based inference services, especially for long‑context workloads.
Source: Hacker News — front page
Source: Hacker News — front page — Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium


