Fuse-All MoE kernel speeds TPU LLM inference by up to 34% with load balancing
Researchers unveiled Fuse-All MoE, a TPU kernel that fuses expert computation and output routing.
Source: Grafy Research LabWhat is Fuse-All MoE and why does it matter?
Mixture‑of‑Experts (MoE) models increase the parameter count of large language models (LLMs) while keeping per‑token compute low by activating only a few experts per token. The approach promises higher accuracy without proportional cost, but on Tensor Processing Units (TPUs) two practical issues have limited its payoff: heavy intermediate data movement and uneven distribution of work across the many expert‑parallel devices. According to the arXiv paper (arXiv:2611.17857v1), these bottlenecks can cause large latency spikes and under‑utilized hardware during both prefill and decode phases.
Fuse‑All MoE is a new TPU kernel that tackles both problems in a single design. By fusing token collection, expert projections, gated activation, and output writes into one launch, the kernel keeps intermediate tensors on‑chip and eliminates separate materialization steps. Fixed token‑slot destinations preserve the routing semantics, allowing the scheduler to move work around without breaking the output contract. The result is a tighter compute‑communication loop that better matches the TPU’s high‑throughput architecture.
How does the fused kernel improve TPU inference?
The kernel is built on Pallas, a tiled programming model that compiles into static TPU programs. It issues local and remote DMA transfers ahead of time, overlaps them with matrix‑multiply units (MXU) and vector processing units (VPU), and uses explicit semaphores to guard buffer lifetimes. Figure 1 shows the fusion boundary: token collection and expert computation stay in VMEM, while final results are written directly to the token‑slot buffers, bypassing a separate receive‑side redistribution stage.
By retaining weight and activation intermediates in VMEM, the design reduces the number of HBM reads and writes. The paper reports that for a skewed EP16 prefill workload the kernel runtime drops from the reference time of a balanced workload to , a dramatic improvement in efficiency. The fused pipeline also respects the TPU’s roofline limits: with packed FP4 weights and BF16 arithmetic, it achieves higher arithmetic intensity while staying within the 7.38 TB/s HBM bandwidth.
How does the design handle load imbalance?
Even with a fused compute path, MoE inference can still suffer when the router concentrates many tokens on a few experts. Fuse‑All MoE adds several regime‑specific balancing mechanisms:
- Load‑aware tile scheduling – rows are bucketed to avoid padding, and hot‑expert tiles are interleaved to smooth traffic bursts.
- Intra‑chip expert redistribution – during execution, active experts can be moved between chip‑mates, keeping each TensorCore busy.
- Cross‑chip preloaded copies – hot experts are duplicated on multiple devices so that routing decisions do not create a single point of congestion.
- Symmetric EP8TP2 decode – for the decode phase, active experts are shared between the two TensorCore devices on each chip, aligning tile computation and reducing latency spikes.
Figure 5 illustrates the pipeline dependencies that enable these strategies: start‑to‑wait intervals expose overlap windows, and send‑buffer lifetimes ensure that late DMA transfers do not stall subsequent tiles.
What performance gains were reported?
The authors evaluated the kernel on a TPU7x (Ironwood) platform with 16 expert devices (EP16) and a symmetric EP8TP2 configuration for decode. Key results include:
- Prefill – kernel runtime for a highly skewed token distribution fell from to the baseline balanced time.
- Decode – the symmetric EP8TP2 variant cut kernel latency by up to compared with a fused EP16 baseline.
- End‑to‑end serving – when integrated into a full serving stack with Megablox routing, overall throughput increased by as much as over the previous EP16 system.
These numbers demonstrate that co‑optimizing computation, communication, and load placement can unlock a sizable fraction of the theoretical TPU performance ceiling for MoE models.
How can Grafy users explore or apply these ideas?
While Grafy does not yet expose a native MoE kernel for TPUs, creators and researchers can still benefit from the concepts presented in the paper. The Grafy Reader app lets you upload the PDF, run semantic search, and ask follow‑up questions about the methodology, making it easy to dissect the algorithmic details without leaving the workspace. For developers interested in prototyping similar fused kernels, the Grafy Code studio provides an AI‑assisted environment to write and test Pallas‑style code snippets that could later be ported to TPU hardware.
If you are evaluating MoE models on cloud TPU instances, the paper’s load‑balancing heuristics (row bucketing, hot‑expert interleaving, static placement) can be implemented in a custom routing layer. Grafy’s Chat app can help you draft the necessary JAX or TensorFlow code, while the Files tool can convert model checkpoints into the packed FP4 format described in the work.
What are the remaining challenges?
The authors note several open questions. Adaptive placement during the prefill phase still requires a separate activation‑exchange step, and the current design assumes static token‑slot destinations. Extending the fused kernel to support dynamic routing decisions or mixed‑precision arithmetic beyond FP4/FP8 will need deeper compiler support. Moreover, the evaluation focuses on a single TPU generation; performance on newer or alternative accelerator families remains to be validated.
For a deeper dive into the paper’s methodology, you can load the PDF into Grafy Reader and ask specific questions. If you want to experiment with kernel‑like code, try drafting it in Grafy Code.
Frequently asked questions
Does Fuse-All MoE eliminate all data movement on the TPU?
No. The design keeps intermediate tensors on‑chip and fuses many stages, but it still requires initial token collection and final output writes that cross device boundaries. The goal is to minimize, not completely erase, data movement.
Is the performance gain limited to the prefill phase?
The paper reports gains for both prefill and decode. Prefill sees a runtime reduction from to under skewed routing, while decode latency drops up to with the symmetric EP8TP2 variant.
Can the load‑balancing techniques be used on GPUs?
The principles—such as row bucketing and expert duplication—are hardware‑agnostic, but the specific implementation relies on TPU‑specific features like VMEM residency and the ICI interconnect. Adapting them to GPUs would require analogous memory‑hierarchy and communication primitives.
Source: Grafy Research Lab — Fuse-All MoE Inference Kernel with Load Balancing on TPUs


