RAW Method Reveals How Refusals Push Users Away from Deployed LLMs
A new causal measurement technique called RAW quantifies the drop‑off caused by LLM refusals, finding a 3‑to‑17‑point loss in re‑engagement for many models.
Source: Grafy Research LabA new causal measurement method called RAW quantifies how often users disengage after a large language model refuses a request. The technique leverages stochastic refusals in existing conversation logs, offering the first large‑scale estimate of refusal aversion without needing A/B tests.
What is refusal aversion and why does it matter?
Refusal aversion is the tendency of a user to stop interacting with a conversational system after the system declines to comply with a request. While refusals are essential for safety and policy compliance, they can also act as a churn driver for consumer‑facing products such as chat assistants, role‑play bots, and customer‑service agents. According to the paper, a single non‑progress error precedes ≈45% of long‑term abandonments, and three or fewer such errors precede ≈90% of users leaving the platform (Li et al., 2020). Measuring this effect is crucial for operators who must balance safety with user retention, yet traditional A/B testing is either ethically prohibited or technically infeasible on closed‑source APIs.
How does the RAW method work?
RAW treats the inherent randomness of temperature‑sampled LLMs as a natural experiment. When the same prompt appears multiple times in a log, the model sometimes refuses (label T = 1) and sometimes complies (T = 0). By matching near‑identical prompts—using a 1025‑dimensional embedding (mxbai‑embed‑large) combined with the OpenAI moderation max_concern_score (X)—the method isolates variation in T that stems from the sampler rather than from user intent. The matched pairs are then fed into a cross‑fit augmented inverse‑propensity weighting (AIPW) estimator to recover the average treatment effect on the treated (ATT). Four complementary estimators (score‑matching, embedding+score, Crump‑trimmed AIPW, and the primary AIPW) provide robustness checks, and sensitivity diagnostics such as Rosenbaum curves assess hidden‑confounder risk. The entire pipeline, illustrated in Figure 1 of the paper, can be run on any log for roughly $$0.30$ per conversations.
What did the authors find across deployed models?
The study applied RAW to 15 deployed LLMs drawn from the LMSYS‑Chat‑1M and WildChat datasets, yielding 19 distinct "cells" (model‑dataset combinations). Thirteen cells contained enough data for precise inference. In seven of those, the ATT was statistically significant and ranged from to percentage points (pp) loss in turn‑3 re‑engagement after a refusal. The remaining six cells showed no distinguishable effect, including the only pre‑training‑only model. One cell even displayed a positive point estimate, which the authors traced to a cohort of persistent users who repeatedly retried the refused request. Figure 2 visualizes the cross‑model ATT estimates, while Figure 3 presents a forest plot of all 19 cells, clearly highlighting which deployments suffer noticeable refusal aversion.
How can developers apply RAW to their own services?
Operators who already store conversation logs can adopt RAW in four practical steps:
- Label refusals – Use an LLM‑as‑judge (the same model or a trusted auxiliary model) to assign a binary refusal label to each response.
- Encode prompts – Generate the 1025‑dimensional prompt embedding and capture the moderation max_concern_score.
- Match near‑duplicate prompts – Apply cosine similarity (≥ 0.7) and a small threshold to create paired clusters where one side refused and the other complied.
- Estimate ATT – Run the cross‑fit AIPW estimator (code and data are released under CC BY 4.0) to obtain the causal effect. Grafy users can streamline steps 2 and 3 with the Grafy Graph app to visualise prompt embeddings and similarity clusters, and they can explore the released code directly in Grafy Code. For quick validation, the Grafy Chat interface lets you sample the same prompt multiple times to see stochastic refusals in action.
What are the limitations and ethical considerations?
RAW relies on the selection on observables assumption: after conditioning on prompt content and moderation risk, the remaining variation in refusal must be independent of unobserved user intent. The authors address this by reporting propensity overlap diagnostics and Rosenbaum sensitivity curves (see Figure 8 in the appendix). Still, hidden confounders—such as transient network latency or user mood—could bias estimates. The method also requires explicit user consent for log analysis and mandates aggregate‑only reporting to protect privacy, as outlined in the deployment guide section of the paper. Finally, because the measurement reveals a potential trade‑off between safety and retention, operators must weigh the business incentive to reduce refusals against the risk of exposing users to harmful content.
Grafy perspective
While RAW is a research contribution, Grafy provides tools that let creators and researchers explore its findings without building the pipeline from scratch. The Grafy Reader app can ingest the RAW corpus (labels, embeddings, moderation scores) and let you query specific cells or run custom aggregations via natural‑language prompts. If you want to prototype your own refusal‑aversion study, you can use Grafy Chat to generate stochastic responses from any hosted model and then feed the outputs into a small‑scale version of the RAW pipeline using Grafy Code. For visualising matched prompt clusters, the Grafy Graph interface offers an interactive way to see how near‑duplicate prompts are grouped and where the treatment variation occurs. Together, these apps let you experiment with causal measurement techniques on your own data while staying within Grafy’s secure, multimodal workspace.
Frequently asked questions
How does RAW differ from a traditional A/B test of refusal behavior?
Traditional A/B testing requires deliberately exposing users to different model versions, which can be ethically risky when refusals involve safety concerns. RAW bypasses the need for live experiments by exploiting natural stochastic variation in existing logs, allowing operators to infer causal effects without ever presenting a user with a deliberately risky prompt.
Can RAW be applied to models that always respond deterministically?
No. RAW depends on temperature‑sampled randomness that occasionally leads the same prompt to be refused and complied with. A fully deterministic model would produce identical outputs for identical inputs, eliminating the quasi‑random treatment variation needed for identification.
What cost should a small startup expect to incur when running RAW on its logs?
The authors report an average compute cost of about $$0.30$ per conversations, covering embedding generation, moderation scoring, and the AIPW estimation. For a startup handling a few hundred thousand daily chats, the marginal expense would be a few dollars per day, making the method financially feasible for most production workloads.


