Grafy News
Try Grafy free

Study Finds Mixed‑Precision Quantization Can Meet Blackwell GPU Limits but INT2 Grids Fail on Generation

A new arXiv paper evaluates low‑bit quantization of Qwen3‑8B on NVIDIA Blackwell B200.

Illustration for the news story: Study Finds Mixed‑Precision Quantization Can Meet Blackwell GPU Limits but INT2 Grids Fail on GenerationSource: Grafy Research Lab

Low‑bit quantization of large language models (LLMs) promises cheaper memory and faster inference, but only if the reduced precision does not collapse accuracy, can run on existing GPU kernels, and stays within measured latency budgets. A new arXiv pre‑print titled Beyond Bitwidth: Deployment‑Constrained Mixed‑Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell investigates exactly these trade‑offs for the 8‑billion‑parameter Qwen3‑8B model on NVIDIA’s B200 "Blackwell" GPU using the vLLM serving stack. The authors, led by Hyojung Han, report a detailed evaluation across several quantization schemes, language families, and task types, and their findings carry practical implications for developers and researchers planning to run multilingual LLMs at scale.

What is mixed‑precision quantization and why does it matter for LLM inference?

Mixed‑precision quantization refers to assigning different numeric bit‑widths to different parts of a neural network rather than using a single uniform precision. In the context of LLMs, this can mean keeping critical layers in higher precision (e.g., INT8 or FP16) while compressing less sensitive linear modules to ultra‑low bits such as INT2 or INT3. The goal is to shrink the model’s memory footprint and improve throughput without sacrificing the quality of generated text. As the paper notes, low‑bit quantization is only valuable when the model preserves service‑relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits.

How did the researchers evaluate quantization on the NVIDIA Blackwell B200?

The study focused on the Qwen3‑8B model, a multilingual LLM, and ran inference through vLLM – an open‑source high‑throughput serving library. The authors tested several quantization techniques:

  • INT2 and INT3 packed kernels – native GPU kernels that store weights in 2‑ or 3‑bit formats.
  • GPTQ – a post‑training quantization method that optimizes per‑layer scales.
  • NVFP4 – NVIDIA’s FP4 format, a 4‑bit floating‑point representation.
  • Mixed‑precision configurations – combinations of the above applied selectively across the model’s 144 fused linear modules.

Each configuration was benchmarked for memory usage, latency, and generation quality on four test suites: Korean, English, numerical reasoning, and logical reasoning. The authors used “preregistered gates” – predefined accuracy thresholds – to decide whether a configuration passed the service‑relevant quality bar.

Which precision configurations succeeded and which failed?

The paper reports that packed INT2 and INT3 kernels do execute on the Blackwell GPU, confirming kernel support for ultra‑low‑bit arithmetic. However, all seven INT2 grid configurations the authors tried degenerate on generation: the model either stalls or produces nonsensical output, failing the accuracy gates. In contrast, certain mixed‑precision setups that retained higher precision in a subset of critical layers managed to stay within the latency budget while meeting the multilingual quality thresholds.

The authors also observed that isolated module‑restoration effects are strongly non‑additive. Restoring a single layer from a lower‑precision version to a higher‑precision one does not linearly improve overall performance; interactions between layers can amplify or dampen the impact, making systematic search for the optimal precision map more complex.

What do the results imply for multilingual LLM deployment?

The findings suggest that developers cannot rely on a blanket INT2 quantization to cut memory and speed costs for multilingual LLMs. While the hardware can technically run INT2 kernels, the quality degradation makes them unsuitable for real‑world generation tasks. Mixed‑precision strategies that judiciously allocate higher bits to sensitive modules provide a more viable path, especially when serving diverse languages and reasoning tasks.

For teams targeting NVIDIA’s Blackwell platform, the study offers a concrete checklist:

  1. Verify kernel support – ensure that the chosen quantization format has native packed kernels on the target GPU.
  2. Benchmark end‑to‑end latency – measure actual serving latency with vLLM or an equivalent stack, not just theoretical FLOP reductions.
  3. Run multilingual quality gates – test on language‑specific benchmarks (e.g., Korean and English) and on numerical/reasoning suites to catch hidden failures.
  4. Iterate mixed‑precision maps – treat precision allocation as a combinatorial optimization problem; simple per‑layer restoration may not yield additive gains.

Overall, the paper underscores the importance of deployment‑constrained evaluation: research on quantization must consider the full stack—from model weights to GPU kernels to application‑level latency and accuracy.

How can Grafy users explore or apply these findings?

Grafy does not yet provide a dedicated low‑bit quantization engine, but creators, developers, and researchers can still engage with the study using existing tools:

  • Reader – Upload the PDF of the arXiv paper to Grafy /reader and ask follow‑up questions about specific sections, methodology, or results. The AI can surface tables, extract the mixed‑precision configurations tested, and compare them against your own models.
  • Chat – Use Grafy /chat to experiment with different LLMs (including Qwen3‑8B if available) at standard precision, then discuss with the assistant how to integrate GPTQ or NVFP4 quantization pipelines.
  • Code – In Grafy /code you can prototype a quantization script, import open‑source tools like GPTQ, and run inference on a local or cloud GPU that mimics the Blackwell architecture. The environment lets you profile memory and latency directly.
  • Graph – Document the exploration process in Grafy /graph, linking each precision configuration to its performance metrics, creating a visual map of the non‑additive effects observed in the paper.

By leveraging these apps, Grafy users can replicate the paper’s evaluation on their own hardware, adapt the mixed‑precision strategies to different models, and share reproducible results in the Community gallery.


Source: Han et al., “Beyond Bitwidth: Deployment‑Constrained Mixed‑Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell,” arXiv pre‑print, 2026‑11‑18.


Source: Grafy Research Lab — Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell