Grafy News
Try Grafy free

FlashAttention beats kernelized attention; LARA++ closes quality gap

A NeurIPS‑2026 paper benchmarks kernelized attention against FlashAttention, showing lower memory use and faster training.

Illustration for the news story: FlashAttention beats kernelized attention; LARA++ closes quality gapSource: Grafy Research Lab

What did the study evaluate?

The paper The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale (Fırat Öncel et al., NeurIPS 2026) set out to measure both efficiency and accuracy of recent kernelized‑attention methods. The authors compared several state‑of‑the‑art (SOTA) linear‑complexity approaches against the hardware‑aware exact implementation FlashAttention, across two major transformer use‑cases: causal language models (CLMs) and vision transformers (ViTs). Their methodology included a unified benchmarking suite that records actual GPU memory consumption, wall‑clock throughput, and downstream task performance under a consistent distillation protocol.

How does FlashAttention compare on efficiency?

According to the authors, FlashAttention consistently required less memory than the optimized kernelized variants during training, and it matched or exceeded them in raw throughput for most sequence lengths. The only exception appeared for very long sequences (≥ 32 K tokens) where some linear‑attention methods narrowed the gap. In vision transformers, the study found no speed advantage for kernelized attention at any practical length. For CLMs, the authors noted that linear attention provided clear memory savings during the prefill phase and during auto‑regressive generation, confirming that the asymptotic O(N)O(N) advantage can translate into practical gains only in inference‑heavy scenarios. The paper also highlighted that many previously reported efficiency improvements could not be reproduced when tested against up‑to‑date baselines, suggesting that earlier results were overstated.

Does kernelized attention match softmax quality?

The second axis of the benchmark examined how closely kernelized models could approximate the output of a standard softmax transformer after being distilled from a pretrained teacher. The authors report performance gaps ranging from 2 to 10 percentage points on image‑classification tasks (DeiT and DINOv2) when using methods such as Hedgehog and DiJiang under the same distillation protocol. These gaps persisted across model scales, indicating that increasing depth or width alone does not close the accuracy deficit. The study therefore questions the assumption that kernelized attention can serve as a drop‑in replacement for softmax without sacrificing quality.

What is LARA++ and how does it improve accuracy?

To test whether the quality gap is inherent to the kernelization idea, the authors introduced LARA++, an extension of the importance‑sampling formulation originally proposed in LARA. LARA++ employs deterministic feature maps and a clustering‑based proposal distribution to approximate the softmax kernel more faithfully. In their experiments, LARA++ recovered up to 99 % of the softmax teacher’s accuracy on vision‑transformer distillation tasks. The improvement demonstrates that the gap is not intrinsic to kernelization. However, LARA++ inherits the computational overhead of its sampling strategy, leaving the efficiency wall largely untouched. The authors conclude that while accuracy can be restored, the added cost means pure kernelized attention still lacks a clear practical advantage over FlashAttention.

How can Grafy users explore these findings?

Grafy Readers are built for deep paper analysis. By loading the arXiv PDF into the Grafy Reader (/reader), creators and researchers can query the benchmark tables, reproduce the memory‑usage graphs, or ask the model to summarize specific sections. For developers interested in experimenting with the code, the authors have released a GitHub repository (https://github.com/firatoncel/KernelRealityCheckNeurIPS2026). Users can clone the repo, run the provided scripts, and then use Grafy Code (/code) to modify or extend the evaluation pipeline. Finally, to get quick answers or compare model outputs, the Grafy Chat interface (/chat) can be prompted with “How does FlashAttention’s memory usage scale with sequence length compared to linear attention?” to obtain on‑the‑fly explanations.


Source: https://arxiv.org/abs/2611.22947

Frequently asked questions

Does the paper claim that kernelized attention is faster than FlashAttention for all tasks?

No. The authors found FlashAttention to be faster and more memory‑efficient for training and for most inference scenarios, with the exception of very long sequences (≥ 32 K) and the prefill stage of causal language models where linear attention shows some benefit.

What is the main reason LARA++ still struggles with efficiency?

LARA++ relies on importance sampling with adaptive proposal distributions, which introduces extra computation per token. While this approach narrows the accuracy gap to softmax, the additional sampling steps outweigh the theoretical O(N)O(N) memory advantage, keeping the overall runtime comparable to or slower than FlashAttention.

Can Grafy’s tools replace the need for the authors’ code repository?

Grafy provides complementary utilities: the Reader app lets users interrogate the paper, the Code studio enables running and tweaking the released scripts, and Chat can answer specific questions. However, reproducing the full benchmark still requires the original codebase and hardware setup described by the authors.


Source: Grafy Research Lab — The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale