Grafy News
Try Grafy free

TempQuant lets quantized LLMs regain accuracy by learning adaptive temperatures

Researchers introduce TempQuant, a method that learns temperature and top‑p settings for quantized language models, boosting performance without extra labels.

Illustration for the news story: TempQuant lets quantized LLMs regain accuracy by learning adaptive temperaturesSource: Grafy Research Lab

Large language models are often run at reduced precision to save memory, but this can hurt generation quality. A new study shows that learning model‑specific temperature and top‑p settings—called TempQuant—can restore much of the lost performance, a finding that matters for anyone deploying quantized LLMs.

Why does quantization change the behaviour of LLMs?

Quantization compresses model weights to fewer bits, introducing numerical error that perturbs the logits used for next‑token prediction. According to the authors of the arXiv pre‑print, these perturbations shift the probability distribution and, on average, make the model warmer—i.e., its output entropy increases. Because temperature (tau\\tau) also controls entropy, the interaction between quantization and the sampling temperature becomes critical: a quantized model reacts more strongly to changes in tau\\tau than its full‑precision counterpart.

What is TempQuant and how does it work?

TempQuant is a lightweight post‑training technique that does not modify the model weights. Instead, it learns a pair of decoding hyper‑parameters—temperature and top‑p (pp)—that minimize the Kullback–Leibler (KL) divergence between the quantized model’s token distribution and that of the original full‑precision model. The authors train two variants:

  • Global TempQuant learns a single temperature and top‑p for each user‑requested temperature. This global setting can be applied at inference time without any labelled data.
  • Per‑step TempQuant predicts a temperature and top‑p for every decoding step, allowing the sampler to warm some tokens while cooling others based on whether quantization flipped their ranking.

Both variants use a small predictor network (similar to the AutoDeco architecture) that takes features such as the raw logits, entropy, and token‑position information to output the optimal tau\\tau and pp.

How much does TempQuant improve performance?

The paper reports several experiments on 3‑bit quantized versions of Qwen3‑4B. Using the temperature and top‑p values recommended for the full‑precision model ( tau ⁣= ⁣0.6\\tau\!=\!0.6, p ⁣= ⁣0.95p\!=\!0.95) yields a 34 % accuracy on the MATH benchmark. After applying Global TempQuant, accuracy climbs to 51 %, a gain comparable to exhaustive tuning on labelled answers but achieved without any supervision. The authors also show that per‑step TempQuant can recover additional gains on tasks where token‑ranking flips are frequent, though the global version is already practical for today’s deployments.

Why does temperature matter more for quantized models?

Measuring entropy during generation, the researchers observed that quantization consistently raises the entropy of the output distribution. A higher entropy means the model’s probabilities are flatter, so small changes in tau\\tau produce larger shifts in token ranks. Consequently, the same temperature that works well for a full‑precision model can either over‑randomise or under‑exploit a quantized model, leading to degraded accuracy. TempQuant directly addresses this mismatch by re‑calibrating tau\\tau (and pp) to bring the quantized distribution back in line with the full‑precision reference.

How can Grafy users experiment with TempQuant ideas?

While Grafy does not yet ship an automatic TempQuant module, creators and developers can explore the principle using two existing tools:

  1. Grafy Chat – Load a quantized model (if available via OpenRouter or Vertex AI) and manually adjust the temperature and top‑p sliders while observing changes in generation quality. This hands‑on approach mirrors the global TempQuant workflow and lets you see the “warming” effect first‑hand.
  2. Grafy Reader – Upload the arXiv PDF to the Reader app and ask the AI to summarise the method, extract the predictor architecture, or even generate a small script that implements the global TempQuant optimisation for a model you host.

Both apps are linked above for quick access.

What are the limitations of TempQuant?

The authors acknowledge that TempQuant relies on having a full‑precision reference model to compute the KL target, which may not be available in every deployment scenario. Moreover, per‑step temperature prediction adds runtime overhead, and its benefits diminish when quantization error is minor (e.g., 4‑bit models). Finally, the method focuses on sampling‑based decoding; it does not address beam search or deterministic greedy decoding.

Source: https://arxiv.org/abs/2611.22128

Frequently asked questions

Does TempQuant require retraining the quantized model?

No. TempQuant operates entirely at decoding time, learning only temperature and top‑p values to align the quantized model’s output distribution with that of the original full‑precision model.

Can TempQuant be used with any quantization level?

The paper demonstrates strong gains for aggressive 3‑bit quantization and modest improvements for milder settings. Effectiveness scales with how much quantization distorts the token probabilities; the more distortion, the larger the potential benefit.

Is per‑step TempQuant practical for real‑time applications?

Per‑step TempQuant provides the highest fidelity but incurs additional computation for each token. For latency‑sensitive services, the global variant offers a good trade‑off, delivering most of the accuracy recovery with a single parameter set.


Source: Grafy Research Lab — TempQuant: Learning Adaptive Temperatures for Quantized Language Models