Grafy News
Try Grafy free

SCALAR Cuts PEFT Costs: Same Accuracy as LoRA with 4% of the Parameters

Researchers introduce SCALAR, a new PEFT method that matches LoRA performance while using only 4% of its trainable parameters.

Illustration for the news story: SCALAR Cuts PEFT Costs: Same Accuracy as LoRA with 4% of the ParametersSource: Grafy Research Lab

What is SCALAR and how does it differ from LoRA?

SCALAR (Subspace‑Calibrated Adaptation via Layerwise Atom Reweighting) is a parameter‑efficient fine‑tuning (PEFT) technique that replaces LoRA’s two trainable low‑rank matrices with a single set of scalar coefficients per layer. The method first builds a fixed input basis from the leading singular vectors of task‑specific activations and pairs each input direction with a random output dictionary. During fine‑tuning, only the scalar weights that re‑scale each input‑direction‑output‑atom pair are updated. In contrast, LoRA learns two full low‑rank factors (often denoted A\bm{A} and B\bm{B}) that together require R (m+n)R\,(m+n) trainable parameters for an m×nm\times n weight matrix. SCALAR reduces this to R HR\,H scalars, where RR is the number of input directions and HH the number of output atoms per direction, cutting the trainable state to a few percent of LoRA’s size.

How does SCALAR achieve lower memory usage and higher throughput?

Because only scalar values are optimized, the gradient and optimizer‑state storage shrink dramatically. The forward pass aggregates the HH output atoms for each of the RR directions before applying the adapter to token activations, so the matrix multiplication operates on an effective rank of RR rather than on the full m×nm\times n matrix. Additionally, SCALAR can share the random output dictionaries across compatible layers, further reducing the fixed‑basis memory footprint. System measurements on the Qwen2.5‑7B model report 20 % lower peak training memory and 6 % higher throughput compared with LoRA, making the approach attractive for on‑device personalization where resources are limited.

What benchmarks demonstrate SCALAR’s performance?

The authors evaluated SCALAR on several downstream tasks:

  • Commonsense reasoning – Across eight benchmark datasets, SCALAR reached accuracy levels comparable to rank‑8 LoRA while using only about 4 % of LoRA’s trainable parameters.
  • Mathematical reasoning and code generation – Results showed competitive scores relative to LoRA, NOLA, and RandLoRA, confirming that the scalar‑only adaptation does not sacrifice capability on more complex generation tasks.
  • Knowledge evaluation after instruction tuning – SCALAR maintained strong performance when fine‑tuned on instruction data, indicating that the method works beyond single‑task adapters.

Experiments were run on both Qwen2.5‑7B and LLaMA3‑8B, and the paper includes scaling studies up to 32 B‑parameter models, with additional large‑model results in the appendix.

How can developers integrate SCALAR into their workflows?

SCALAR’s design fits naturally into existing PEFT pipelines. Developers can:

  1. Collect task activations – Run a few forward passes on the downstream dataset and compute the top singular vectors (the paper suggests a fast SVD implementation).
  2. Construct the fixed basis – Store the input basis and generate a random output dictionary for each direction. The dictionaries can be shared across layers that have matching dimensions.
  3. Fine‑tune only scalars – Use any optimizer that supports scalar parameters; the gradient and optimizer‑state buffers will be tiny.
  4. Deploy the adapted model – Because the adapter consists of a small lookup of scalars and a lightweight aggregation step, inference overhead is minimal.

The process can be scripted in common deep‑learning frameworks (PyTorch, JAX) and wrapped in Grafy’s Code studio to experiment with custom adapters. For those who prefer a no‑code approach, the Reader app lets you upload the arXiv PDF, ask the AI to extract the implementation steps, and generate a runnable notebook.

What are the limitations and future directions?

While SCALAR shows impressive efficiency, the method relies on a task‑calibrated input basis that must be computed from a representative activation set. If the calibration data are not diverse enough, the fixed subspace may miss useful directions, potentially limiting performance on out‑of‑distribution inputs. The paper’s ablations (Section C.6) indicate that the choice of RR and HH influences both accuracy and memory savings, so hyper‑parameter tuning remains necessary. Future work could explore adaptive basis updates during training or combine SCALAR with other compression techniques such as quantization.

Grafy perspective

For researchers and developers wanting to explore SCALAR without rebuilding the entire pipeline, Grafy’s Reader app offers a convenient entry point. Upload the SCALAR paper, ask the built‑in chatbot to summarise the method, extract the SVD‑based calibration steps, and generate code snippets that you can copy into the Code studio. This workflow lets you prototype the adapter on your own models and immediately measure memory and throughput gains.


Source: arXiv preprint "Direction or Magnitude? Learning Scalars over Task‑Calibrated Directions" (2026‑11‑16) https://arxiv.org/abs/2611.17784

Frequently asked questions

Does SCALAR train both input and output matrices like LoRA?

No. SCALAR freezes both the input basis (derived from activation SVD) and the random output dictionaries, training only scalar coefficients that re‑weight each input‑direction‑output‑atom pair.

How much memory can SCALAR save compared with LoRA?

On the Qwen2.5‑7B model, SCALAR reduced peak training memory by roughly 20 % while also delivering a 6 % increase in training throughput.

Can SCALAR be used with models larger than 32 B parameters?

The paper reports scaling experiments up to 32 B parameters and includes additional large‑model results in the appendix, suggesting that the approach is applicable to even larger models, though practical limits will depend on hardware and calibration data quality.


Source: Grafy Research Lab — Direction or Magnitude? Learning Scalars over Task-Calibrated Directions