Extra passes beat extra bits at 6-bit for small local LLMs

A new study posted to arXiv reports that small local language models can drop to 6-bit weights without losing accuracy, and that, for a fixed inference budget, spending what remains on a few shuffled-answer passes beats spending it on more precision — provided the weights already sit at 6 bits or more. That gives developers a concrete rule for a common problem: how to allocate limited compute when running a model on their own hardware rather than a cloud API. According to the paper, by Mohammed Nayeem Khan, Logesh L and Aliyas Ahmad Misri, the trade-off resolves differently above and below 6 bits, and the authors reduce their results to a short decision table.

What did the study set out to test?

The authors frame two linked questions: does storing weights in fewer bits make a model more sensitive to the order of its answer options, and which is the cheaper way to spend a fixed inference budget — more precision, or more reorderings? To get at this, they ran two small models, Qwen3-0.6B and Qwen3-1.7B, and simulated round-to-nearest weight quantization at 8, 6, 4 and 3 bits, with unquantized fp16 as the reference point. Instead of scoring free-form text, they read the logits the models assigned to the letters A through D on multiple-choice questions, which exposes how strongly each option is favored. The evaluation covered ARC-Easy, ARC-Challenge, MMLU and HellaSwag, with 300 items taken from each.

Does using fewer bits make a model more order-sensitive?

One of the clearest results is that 6-bit quantization cost no accuracy at all. According to the paper, 6-bit weights actually scored higher than fp16 — by 1.8 points (interval 1.0 to 2.5) on the 0.6B model and 0.9 points (0.3 to 1.5) on the 1.7B model — a gain the authors say they cannot explain. At 8 bits, accuracy barely moved, changing by -0.7 points (-1.1 to -0.3) and -0.4 points (-0.7 to -0.2) for the two models. The real shift shows up at 4 bits, where order sensitivity climbs. The authors measure this with a quantity they call answer-slot skew, which captures how unevenly a model's preference is spread across the four letter positions; at 4 bits it rose from 0.15 to 0.53 on the 0.6B model and from 0.07 to 0.12 on the 1.7B model. They stop short of blaming rounding alone. When they injected Gaussian noise matched to the same weight signal-to-noise ratio, they measured skew of 0.15 and 0.37 across two draws, and 0.19 on a single draw — close enough to the quantization figures that the paper calls its control inconclusive on whether rounding adds slot skew beyond what equal-power noise would. Flip rates, meaning how often the chosen answer changes, rose under quantization but stayed below a baseline of independent uniform errors matched for accuracy.

Is it cheaper to add bits or to add answer-order passes?

To compare the two strategies on equal footing, the authors price them in a single machine-independent unit: weight bytes read multiplied by the number of passes. The unit is deliberately abstract — it counts data movement rather than wall-clock time — so the comparison is meant to hold across different machines even though it ignores any one device's quirks. That framing lets them ask, for the same total cost, whether it is better to load higher-precision weights once or to load lower-precision weights several times with the options reordered. Averaging over reorderings pays off most when precision is low. According to the paper, averaging four cyclic orders recovered 12.6 points at 4 bits on the 0.6B model, but only 5.3 points at fp16, where there was less to recover. In cost-matched comparisons, extra orders beat extra bits only once the weights already had 6 or more bits. The whole decision, the authors write, collapses to a short rule: use 6-bit weights and spend the rest of the budget on up to four orders, one pass each.

What are the caveats?

The authors are direct about the limits of the work. Everything was simulated on two small models using the simplest round-to-nearest quantizer, so the findings may not transfer to larger models, other quantization schemes, or real hardware, where memory bandwidth and caching behave differently from a byte-count cost model. The unexplained 6-bit accuracy gain is flagged as unexplained rather than presented as a general effect. And a title-level literature search, the authors report, found no earlier study that looked at weight quantization and option order together, so they position the paper as a first look rather than a settled result.

How does this connect to Grafy?

This is a research paper about running models efficiently on your own hardware, and the most direct way to engage with it in Grafy is Reader, which lets you load a PDF or paper and question it with AI — handy for pulling out the exact confidence intervals, the precise definition of answer-slot skew, or the cost-matched tables that any summary flattens. If you would rather explore the underlying ideas than the paper itself — how weight quantization works, or why multiple-choice order bias appears — you can open Grafy Chat and compare how different models explain them side by side.

Frequently asked questions

Does 6-bit quantization reduce accuracy on these small models?

According to the study, no — 6-bit weights scored slightly higher than fp16 on both Qwen3-0.6B and Qwen3-1.7B, by 1.8 and 0.9 points respectively, an improvement the authors say they cannot explain. A clear accuracy loss appeared only at 4 bits, where order sensitivity also rose.

When do extra answer-order passes beat extra bits?

According to the paper, extra reorderings beat extra precision only when the weights already have 6 or more bits. Below that threshold, spending the budget on more bits helps more. The authors' decision rule follows from this: use 6-bit weights, then spend whatever remains on up to four order-averaged passes, one pass each.

Are these findings confirmed on real hardware?

No. The authors state that everything was simulated on two small models using the simplest round-to-nearest quantizer, with costs priced in a machine-independent byte-count unit rather than measured on real memory. They present the work as a first look, noting that a title-level search found no earlier study examining weight quantization and option order together.

---

Source: Grafy Research Lab — Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models