bf16 Rounding Discards Most RL Updates – Fair Cast Offers One‑Line Fix

Reinforcement‑learning (RL) fine‑tuning of large language models typically keeps a full‑precision (fp32) master copy of the weights for the optimizer, while the policy used for rollouts is a bf16 version created by round‑to‑nearest conversion. According to the authors, this conversion silently discards the vast majority of per‑step updates because, at common learning rates, the changes are smaller than half the spacing of the bf16 format. On the Qwen3‑1.7B model only 0.4‑0.5 % of the served weights change each step, leaving the policy that generates training data lagging the optimizer by 50‑150 steps.

Why bf16 rounding matters for RL

The mismatch between the optimizer’s master weights and the policy that interacts with the environment creates a hidden training‑inference gap. Existing correction methods compare two copies that have already undergone the same rounding, so they cannot detect the lag. The authors point out that this lag acts like an unintended regularizer: the policy is effectively trained on stale parameters, which can slow convergence and reduce final performance.

The Fair Cast solution

To eliminate the discrepancy, the paper proposes Fair Cast, a one‑line change at the export step. Instead of the default round‑to‑nearest, the master weights are exported using an unbiased rounding rule—either stochastic rounding or error‑feedback rounding. The training loss is then computed on the same bf16 copy that will be served. This adjustment adds no new hyper‑parameters and incurs only 22‑38 ms of overhead per export for a 1.7 B‑parameter model.

Practical impact and performance gains

Across 25 matched seed pairs ranging from Qwen3‑0.6B to 4B and using both AdamW and Muon optimizers, Fair Cast outperformed the standard rounding on 18 runs, tied on 6, and fell behind on just 1, yielding a mean gain of 6.1 points. On a separate benchmark of zebra‑puzzle tasks, Fair Cast achieved a score of 0.57 ± 0.16 over four runs, whereas round‑to‑nearest collapsed to 0.33 ± 0.10. Additionally, the approach aligns the saved fp32 checkpoint with the served bf16 model, making checkpoints easier to audit and reuse.

How to try this in Grafy

Grafy provides the tools needed to experiment with Fair Cast without leaving the workspace:

• Grafy Code – Use the integrated coding studio to edit the model‑export script. Replace the standard conversion call with an unbiased rounding option (e.g., stochastic_rounding=True). The change is a single line, matching the paper’s implementation.
• Grafy Reader – Load the PDF of the arXiv paper directly into the Reader app to explore the methodology, figures, and experimental details with AI‑assisted search and question‑answering.
• Grafy Chat – Ask the multi‑model chat to explain stochastic rounding, suggest code snippets for different frameworks, or compare the performance impact on your own models.

By chaining these apps, a researcher can replicate the experiments on their own LLMs, observe the reduction in policy lag, and quantify any downstream gains on downstream tasks.

Limitations and future work

While Fair Cast demonstrates clear benefits, the paper notes that the improvement magnitude varies with model size, optimizer choice, and learning‑rate schedule. The one‑line fix also relies on the availability of unbiased rounding implementations in the underlying deep‑learning library; not all runtimes expose stochastic rounding out of the box. Future work could explore adaptive rounding strategies that switch between unbiased and deterministic modes based on observed update magnitudes.

Frequently asked questions

Does Fair Cast change the optimizer’s learning rate?

No. Fair Cast only modifies how the master fp32 weights are converted to bf16 for serving. The optimizer continues to use the original learning‑rate schedule; the rounding change simply ensures that each update is reflected in the policy copy.

Can I use Fair Cast with other low‑precision formats like int8?

The paper focuses on bf16 and the specific issue of half‑spacing rounding. While the principle of unbiased conversion could apply to other formats, the authors did not evaluate int8, so additional testing would be required.

How much extra compute does the unbiased rounding add?

According to the authors, exporting a 1.7 B‑parameter model with Fair Cast costs between 22 and 38 ms per export, which is a negligible overhead compared to typical RL training loops.

Source: https://arxiv.org/abs/2610.41073

---

Source: arXiv Machine Learning — bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix