HADVQ introduces Hessian‑aware directional vector quantization for ultra‑low‑bit LLMs
A new vector‑quantization technique called HADVQ tackles ultra‑low‑bit compression of large language models by separating direction and magnitude.
Source: Grafy Research LabHADVQ – Hessian Aware Directional Vector Quantization – was presented by a team of researchers on arXiv on November 18, 2026. The method targets the most aggressive compression regimes for large language models, where standard quantization often destroys a handful of highly sensitive weight channels.
What is HADVQ and why it matters
According to the authors, HADVQ is a vector‑quantization approach designed specifically for ultra‑low‑bit settings. It aims to keep the functional performance of a model while shrinking its weight footprint to a few bits per parameter. This matters because deploying massive language models on edge devices or in cost‑constrained inference environments has been blocked by the memory and bandwidth demands of full‑precision weights.
How does the method work
The pipeline consists of three stages. First, the weights are passed through a random preconditioning matrix that regularizes the weight space, making it more uniform and easier to quantize. Next, each weight vector is expressed in polar coordinates – a radius representing magnitude and an angle representing direction. Finally, the magnitude and direction are quantized independently. Directional quantization leverages the structure introduced by the random preconditioning, allowing the authors to use a coarser codebook for angles without incurring large errors. By decoupling the two components, the method avoids the distortion that typically arises when both are forced into a single low‑bit representation.
Which parts of LLMs benefit most
The authors highlight that a small subset of channels in transformer layers is disproportionately sensitive to quantization noise. Because HADVQ treats direction and magnitude separately, it can preserve the orientation of these critical channels even when the magnitude is heavily compressed. Experiments in the paper show that the technique reduces the degradation of perplexity and downstream task accuracy compared with conventional uniform quantization at the same bit budget.
Comparison with existing quantization approaches
Traditional low‑bit quantization methods apply a scalar rounding operation to each weight, often after a simple linear scaling. Some recent works introduce learned codebooks or mixed‑precision schemes, but they still treat each weight as a single scalar value. HADVQ differs by introducing a geometric perspective – the polar decomposition – and by using random preconditioning to shape the distribution of angles. The authors report that, for the same bit width, HADVQ achieves lower reconstruction error and higher downstream performance than uniform quantization and recent Hessian‑aware scalar methods.
Practical steps to try HADVQ
- Pre‑conditioning – generate a random orthogonal matrix (or use a fast sketching transform) and multiply it with the weight tensor of the target layer.
- Polar conversion – compute the Euclidean norm and the unit direction for each weight vector.
- Quantize magnitude – apply a standard low‑bit scalar quantizer (e.g., 2‑bit uniform) to .
- Quantize direction – map the unit vectors to a small set of directional codewords using a spherical‑cap clustering that respects the preconditioning‑induced geometry.
- Re‑assemble – reconstruct quantized weights as and optionally fine‑tune the model for a few epochs to recover any remaining gap.
Developers can prototype these steps in Python using NumPy or PyTorch. The code required is modest: a matrix multiplication for preconditioning, norm calculations, and a k‑means‑like clustering on the unit sphere for direction codebooks. The authors provide a reference implementation in the supplementary material of the arXiv paper.
Relevance for Grafy users
Grafy’s Reader app lets you load the original PDF, ask the model to explain sections, and export annotations – a convenient way to explore the HADVQ paper without leaving the platform. For developers who want to experiment, the Code studio offers a ready‑to‑run notebook environment where you can import the authors’ reference code, apply the quantization pipeline to a public LLM checkpoint, and immediately test inference speed and accuracy. If you are tracking multiple quantization experiments, the Graph app can record each variant’s hyper‑parameters and results, giving you a visual branching history of your compression research. Finally, if you need to discuss findings with teammates, the Chat interface lets you bring in different model back‑ends side by side to compare explanations of the method.
Source: Grafy Research Lab, arXiv: https://arxiv.org/abs/2611.22721
Frequently asked questions
How does random preconditioning improve quantization?
Random preconditioning spreads the weight distribution more evenly across dimensions, reducing the variance of individual channels. This makes the angular component of the weights more uniform, allowing a small directional codebook to approximate the true directions with lower error.
Does HADVQ require fine‑tuning after quantization?
The paper reports that a short fine‑tuning phase (a few thousand steps) helps close the remaining performance gap, but the method already delivers competitive accuracy without any additional training.
Can HADVQ be applied to models other than language models?
Yes. The technique operates on any high‑dimensional weight tensor, so it can be used for vision transformers, diffusion models, or other architectures where ultra‑low‑bit compression is desirable.
Source: Grafy Research Lab — HADVQ: Hessian Aware Directional Vector Quantization


