Stratified Policies Boost Inference-Time Sampling for Large Language Models
A new study shows that allocating inference samples across multiple learned LoRA‑adapted policies improves LLM output utility.
Source: Grafy Research LabInference‑time scaling has become a common trick for squeezing extra performance out of large language models (LLMs). Instead of running a single prompt once, practitioners often generate multiple responses and pick the best, or combine them for higher utility. A fresh arXiv paper titled Learning Stratified Policies for Inference‑Time Objectives proposes a different angle: rather than drawing all samples from one policy, split the sample budget across a small population of learned policies.
What are stratified policies and why they matter?
Stratified policies refer to a set of distinct policies, each trained to generate one response that will be part of a ‑sample inference‑time objective. The idea, introduced by Mahajan, Zhang, and Pacchiano, is to treat the inference budget as a resource that can be allocated across multiple specialized generators instead of a single, monolithic one. By diversifying the policy pool, the collective output can cover a broader portion of the response space, potentially increasing the chance of a high‑quality answer without extra compute.
How do they differ from single‑policy sampling?
Traditional multi‑sample approaches repeatedly query the same policy, relying on stochasticity (temperature, nucleus sampling, etc.) to produce variation. This method often yields redundant or low‑diversity results, especially when the underlying policy is already near‑optimal for the given prompt. Stratified policies, by contrast, purposefully train each adapter to explore a different region of the output distribution. In practice, this means that a ‑sample run draws from separate adapters, each contributing a unique perspective, rather than draws from the same statistical engine.
How are the policies implemented using LoRA adapters?
The authors realize the policy population by attaching separate LoRA (Low‑Rank Adaptation) adapters to a shared language‑model backbone. LoRA adds a lightweight set of trainable matrices that modify the base model’s weights during inference while keeping the original parameters frozen. Each adapter is trained on a distinct objective or data slice, resulting in lightweight extensions that can be swapped in and out with minimal overhead. Because LoRA adapters are small (often a few megabytes), maintaining a whole suite of policies is far cheaper than storing full model replicas.
Practical implications for developers and creators
For developers building AI‑augmented products, stratified policies open a new lever for quality control. A chatbot could query three adapters—one tuned for factual accuracy, another for creativity, and a third for brevity—and then apply a simple utility function (e.g., a weighted sum of relevance and diversity) to pick the final reply. Creators generating story ideas, code snippets, or design concepts can benefit from the broadened creative palette without paying for larger model instances. Moreover, the approach preserves the existing inference pipeline; the only addition is a step that loads the appropriate LoRA adapter before each sample.
Trying stratified policies with Grafy
Grafy provides the tools needed to experiment with this idea without leaving the platform.
- Grafy Code – Use the integrated coding studio to fine‑tune LoRA adapters on a base LLM of your choice. Sample notebooks illustrate how to attach multiple adapters to the same model checkpoint and export each as a separate endpoint.
- Grafy Chat – Once the adapters are ready, configure the chat interface to rotate through them for a single user query. Grafy Chat supports side‑by‑side model calls, so you can see each policy’s response in real time.
- Grafy Graph – Track the lineage of prompts, adapter versions, and resulting outputs in a branching graph. This visual history helps you evaluate which combination of adapters yields the highest utility for a given task.
- Grafy Reader – If you want to dive deeper into the methodology, upload the arXiv PDF and ask the Reader to summarize sections, extract equations, or compare this work to earlier multi‑policy research.
By chaining these apps, a developer can prototype a stratified‑policy pipeline, measure its performance against a baseline single‑policy setup, and iterate quickly—all within Grafy’s unified workspace.
To explore this yourself, open Grafy Code: an AI coding studio.
Frequently asked questions
How many policies should I use for a given inference budget?
The optimal number depends on the task and the compute budget. The paper demonstrates improvements with modest values (e.g., to ) and suggests diminishing returns beyond that. In practice, start with three distinct adapters—accuracy‑focused, creativity‑focused, and speed‑focused—and evaluate whether adding more yields measurable gains.
Do LoRA adapters interfere with each other when loaded sequentially?
LoRA adapters are applied as additive updates to the base model’s weights. When you load an adapter, the base weights remain unchanged, and the adapter’s low‑rank matrices are overlaid for that inference pass. As long as you reset to the base model (or load a different adapter) before the next sample, there is no cross‑contamination.
Can I combine stratified policies with existing ensemble techniques like voting or reranking?
Yes. Stratified policies generate a diverse set of outputs that can be fed into downstream ensemble methods such as majority voting, rank‑based selection, or utility‑driven reranking. The diversity introduced by separate adapters often improves the effectiveness of these post‑processing steps, leading to higher overall quality.
Source: Grafy Research Lab — Learning Stratified Policies for Inference-Time Objectives


