Grafy News
Try Grafy free

Model Compression Beyond Quantization: Pruning, Structured Pruning, and Distillation Explained

Adrian Tam details pruning, structured pruning, and knowledge distillation as alternatives to quantization.

Illustration for the news story: Model Compression Beyond Quantization: Pruning, Structured Pruning, and Distillation ExplainedSource: MachineLearningMastery

Model compression techniques beyond quantization—pruning, structured pruning, and knowledge distillation—are gaining attention as developers look to shrink AI models for faster, cheaper inference. Adrian Tam’s November 7, 2026 post on MachineLearningMastery explains how these methods work and what trade‑offs to expect.

What is pruning and how does sparsity affect execution?

Pruning removes selected parameters from a neural network, either by setting them to zero (unstructured pruning) or by deleting whole structures (structured pruning). The simplest form, magnitude pruning, assumes that weights with small absolute values contribute little to the output. Tam demonstrates this with a PyTorch snippet that masks the lowest‑magnitude half of a 1024×1024 weight matrix, converts the result to compressed sparse‑row (CSR) format, and then multiplies a dense batch by the sparse matrix:

import torch

torch.manual_seed(0)

def magnitude_mask(weight, sparsity):
    assert 0.0 <= sparsity < 1.0
    if sparsity == 0.0:
        return torch.ones_like(weight, dtype=torch.bool)
    threshold = torch.quantile(weight.abs().float(), sparsity)
    return weight.abs() > threshold

weight = torch.randn(1024, 1024)
mask = magnitude_mask(weight, sparsity=0.5)
pruned_weight = weight * mask
print('remaining fraction:', mask.float().mean().item())

sparse_weight = pruned_weight.to_sparse_csr()

x = torch.randn(8, 1024)
sparse_output = torch.sparse.mm(sparse_weight, x.T).T
dense_output = x @ pruned_weight.T
torch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4)

The script prints a remaining fraction of roughly 0.5, confirming that half the weights were removed. After pruning, the tensor is still stored densely; converting to CSR saves memory only when the metadata overhead is lower than the space reclaimed. Speed gains depend on the matrix shape, sparsity level, data type, and whether the hardware provides an efficient sparse kernel, as Tam notes.

How does structured pruning reshape model architecture?

Structured pruning goes a step further by deleting whole units—attention heads, MLP channels, or even entire transformer layers. Because the removed pieces are complete, the resulting model can often be represented as a smaller dense network, which is easier to accelerate on standard hardware.

Consider a transformer MLP that first expands a hidden vector with an "up" projection and then contracts it with a "down" projection:

y=Wdownϕ(Wupx)\mathbf{y} = \mathbf{W}_{\text{down}} \phi(\mathbf{W}_{\text{up}} \mathbf{x})

Each intermediate channel corresponds to a row of Wup\mathbf{W}_{\text{up}} and a matching column of Wdown\mathbf{W}_{\text{down}}. Removing a low‑norm row from the up projection therefore requires removing the associated column from the down projection. Tam provides a concise PyTorch function that keeps the highest‑norm channels and rebuilds the two linear layers:

import torch
import torch.nn as nn

def shrink_mlp(up_proj, down_proj, keep_channels):
    if not 1 <= keep_channels <= up_proj.out_features:
        raise ValueError('keep_channels must be within the MLP width')
    if up_proj.out_features != down_proj.in_features:
        raise ValueError('projection widths must match')
    scores = up_proj.weight.norm(dim=1)
    keep = scores.topk(keep_channels).indices.sort().values

    smaller_up = nn.Linear(
        up_proj.in_features,
        keep_channels,
        bias=up_proj.bias is not None,
    ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype)

    smaller_down = nn.Linear(
        keep_channels,
        down_proj.out_features,
        bias=down_proj.bias is not None,
    ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype)

    with torch.no_grad():
        smaller_up.weight.copy_(up_proj.weight[keep])
        smaller_down.weight.copy_(down_proj.weight[:, keep])
        if up_proj.bias is not None:
            smaller_up.bias.copy_(up_proj.bias[keep])
        if down_proj.bias is not None:
            smaller_down.bias.copy_(down_proj.bias)

    return smaller_up, smaller_down

up = nn.Linear(16, 32)
down = nn.Linear(32, 16)
smaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)
print(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape)

The keep_channels argument is an integer count, not a sparsity fraction, and must respect the original projection dimensions. While the code guarantees shape compatibility, it does not evaluate whether the pruned channels are truly unimportant; a production‑grade pipeline would include fine‑tuning after the structural change.

Removing attention heads follows a similar pattern, requiring coordinated edits to the query, key, value, and output projections. In models that share query‑key‑value (GQA) heads, careless removal can break the grouping, so careful bookkeeping is essential. Pruning entire transformer layers changes the residual depth and may need additional mapping layers to preserve learned representations.

What is knowledge distillation and when should you use it?

Knowledge distillation trains a compact "student" model to mimic the behavior of a larger "teacher" model. Instead of learning only from hard labels, the student receives the teacher’s soft output distribution, intermediate activations, or even synthetic data generated by the teacher. This richer signal helps the student capture nuanced decision boundaries while using far fewer parameters.

Tam’s article outlines several distillation variants:

  • Logits‑based distillation – the student minimizes a KL‑divergence loss between its logits and the teacher’s softened logits.
  • Feature‑based distillation – intermediate hidden states are aligned, encouraging the student to learn similar internal representations.
  • Data‑free distillation – the teacher generates pseudo‑inputs, allowing the student to be trained without the original dataset.

The key advantage is that inference speed and memory footprint improve dramatically, often with only a modest drop in accuracy. Distillation is especially useful when the target deployment platform cannot exploit sparse kernels efficiently, or when the engineering effort to restructure a model (as required for structured pruning) is prohibitive.

Practical steps to apply pruning or distillation in your workflow

  1. Choose a baseline model – Start with a pre‑trained checkpoint that meets your accuracy target.
  2. Select a compression goal – Decide whether you need a smaller memory footprint (pruning), a hardware‑friendly shape (structured pruning), or the fastest possible inference (distillation).
  3. Apply magnitude pruning – Use a sparsity schedule (e.g., 30 % → 70 % over several epochs) and monitor validation loss. If the hardware supports sparse kernels, you may gain speed directly.
  4. Perform structured pruning – Identify low‑importance heads or channels using norm‑based scores, then rebuild the model as shown in the shrink_mlp example. Follow with a short fine‑tuning phase to recover any lost performance.
  5. Distill to a student – Define a student architecture (e.g., half the hidden size, fewer layers). Train with a combined loss: cross‑entropy on ground‑truth labels plus KL‑divergence on teacher logits. Optionally add feature‑matching losses.
  6. Validate on target hardware – Measure latency, throughput, and memory usage on the actual deployment device. Adjust sparsity or student size until you hit the desired trade‑off.

These steps can be scripted in a notebook or CI pipeline, allowing repeatable experiments across multiple models.

How Grafy lets you experiment with model compression

Grafy provides a dedicated coding environment where you can run the exact pruning and structured‑pruning scripts shown above. In Grafy Code you can create a notebook, install PyTorch, load a model checkpoint, and iterate on sparsity schedules or channel‑selection heuristics without leaving the browser. The platform also supports GPU‑accelerated kernels, so you can benchmark sparse matrix multiplication on the same hardware you plan to deploy.

If you prefer to explore the academic literature behind these techniques, Grafy Reader lets you upload PDFs of recent distillation papers, search for specific sections (e.g., “feature‑based distillation”), and ask follow‑up questions to an AI assistant that extracts the relevant formulas and experimental results. This dual workflow—coding in Code and researching in Reader—covers the full lifecycle from theory to production.

Frequently asked questions

Does magnitude pruning always improve inference speed?

Magnitude pruning reduces the number of non‑zero weights, but speed gains depend on whether the runtime can exploit sparse kernels. On CPUs or GPUs without efficient sparse support, a pruned model may still run at dense speed, though memory usage will be lower.

Can structured pruning change the model’s accuracy dramatically?

Structured pruning removes whole computational units, which can impact the model’s capacity more than unstructured pruning. Accuracy loss varies; fine‑tuning after pruning is essential to recover performance, and aligning channel counts with hardware tile sizes can mitigate degradation.

Is knowledge distillation limited to transformer models?

No. Distillation works for any architecture—CNNs, RNNs, and even reinforcement‑learning agents. The core idea is to transfer the teacher’s learned behavior to a smaller student, regardless of the underlying model type.


Source: MachineLearningMastery (Adrian Tam, “Model Compression Beyond Quantization”, November 7, 2026).


Source: MachineLearningMastery — Model Compression Beyond Quantization