Grafy News
Try Grafy free

RAW Method Reveals How Refusals Push Users Away from Deployed LLMs

A new causal measurement technique called RAW quantifies the drop‑off caused by LLM refusals, finding a 3‑to‑17‑point loss in re‑engagement for many models.

Illustration for the news story: RAW Method Reveals How Refusals Push Users Away from Deployed LLMsSource: Grafy Research Lab

A new causal measurement method called RAW quantifies how often users disengage after a large language model refuses a request. The technique leverages stochastic refusals in existing conversation logs, offering the first large‑scale estimate of refusal aversion without needing A/B tests.

What is refusal aversion and why does it matter?

Refusal aversion is the tendency of a user to stop interacting with a conversational system after the system declines to comply with a request. While refusals are essential for safety and policy compliance, they can also act as a churn driver for consumer‑facing products such as chat assistants, role‑play bots, and customer‑service agents. According to the paper, a single non‑progress error precedes ≈45% of long‑term abandonments, and three or fewer such errors precede ≈90% of users leaving the platform (Li et al., 2020). Measuring this effect is crucial for operators who must balance safety with user retention, yet traditional A/B testing is either ethically prohibited or technically infeasible on closed‑source APIs.

Figure 2: Cross-model ATT of refusal on user re-engagement, 13-cell main panel, split by source so each panel reads as a same-estimand comparison. Panel A (LMSYS, 4 cells…
Figure 2. Cross-model ATT of refusal on user re-engagement, 13-cell main panel, split by source so each panel reads as a same-estimand comparison. Panel A (LMSYS, 4 cells): cross-model heterogeneity on one prompt distribution. Panel B (WildChat, 9 cells): teal diamonds are general-traffic cells, dark red risky-enriched; the three dated GPT-4 checkpoints (orange, connected by a dotted line) show the ATT movi… Figure 2 of “RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs” by Pavel Kireyev (arXiv.org), CC BY 4.0.
Figure 3: Forest of all 19 cells, ordered by AIPW point estimate (the all-cells companion to Figure 2, which restricts to the 13 main-panel cells). x-axis: ATT of refusal…
Figure 3. Forest of all 19 cells, ordered by AIPW point estimate (the all-cells companion to Figure 2, which restricts to the 13 main-panel cells). x-axis: ATT of refusal on turn-3 user re-engagement, percentage points; y-axis: cell. Diamond = AIPW with 95% CI (LMSYS in blue, WildChat-general in teal, WildChat-risky in dark red); main-panel CIs use the default clustered regime of Table 6, excluded-cell CIs… Figure 3 of “RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs” by Pavel Kireyev (arXiv.org), CC BY 4.0.

How does the RAW method work?

RAW treats the inherent randomness of temperature‑sampled LLMs as a natural experiment. When the same prompt appears multiple times in a log, the model sometimes refuses (label T = 1) and sometimes complies (T = 0). By matching near‑identical prompts—using a 1025‑dimensional embedding (mxbai‑embed‑large) combined with the OpenAI moderation max_concern_score (X)—the method isolates variation in T that stems from the sampler rather than from user intent. The matched pairs are then fed into a cross‑fit augmented inverse‑propensity weighting (AIPW) estimator to recover the average treatment effect on the treated (ATT). Four complementary estimators (score‑matching, embedding+score, Crump‑trimmed AIPW, and the primary AIPW) provide robustness checks, and sensitivity diagnostics such as Rosenbaum Γ\Gamma curves assess hidden‑confounder risk. The entire pipeline, illustrated in Figure 1 of the paper, can be run on any log for roughly $$0.30$ per  1,000\,1{,}000 conversations.

What did the authors find across deployed models?

The study applied RAW to 15 deployed LLMs drawn from the LMSYS‑Chat‑1M and WildChat datasets, yielding 19 distinct "cells" (model‑dataset combinations). Thirteen cells contained enough data for precise inference. In seven of those, the ATT was statistically significant and ranged from 33 to 1717 percentage points (pp) loss in turn‑3 re‑engagement after a refusal. The remaining six cells showed no distinguishable effect, including the only pre‑training‑only model. One cell even displayed a positive point estimate, which the authors traced to a cohort of persistent users who repeatedly retried the refused request. Figure 2 visualizes the cross‑model ATT estimates, while Figure 3 presents a forest plot of all 19 cells, clearly highlighting which deployments suffer noticeable refusal aversion.

How can developers apply RAW to their own services?

Operators who already store conversation logs can adopt RAW in four practical steps:

  1. Label refusals – Use an LLM‑as‑judge (the same model or a trusted auxiliary model) to assign a binary refusal label to each response.
  2. Encode prompts – Generate the 1025‑dimensional prompt embedding and capture the moderation max_concern_score.
  3. Match near‑duplicate prompts – Apply cosine similarity (≥ 0.7) and a small ΔXs\Delta X_s threshold to create paired clusters where one side refused and the other complied.
  4. Estimate ATT – Run the cross‑fit AIPW estimator (code and data are released under CC BY 4.0) to obtain the causal effect. Grafy users can streamline steps 2 and 3 with the Grafy Graph app to visualise prompt embeddings and similarity clusters, and they can explore the released code directly in Grafy Code. For quick validation, the Grafy Chat interface lets you sample the same prompt multiple times to see stochastic refusals in action.

What are the limitations and ethical considerations?

RAW relies on the selection on observables assumption: after conditioning on prompt content and moderation risk, the remaining variation in refusal must be independent of unobserved user intent. The authors address this by reporting propensity overlap diagnostics and Rosenbaum Γ\Gamma sensitivity curves (see Figure 8 in the appendix). Still, hidden confounders—such as transient network latency or user mood—could bias estimates. The method also requires explicit user consent for log analysis and mandates aggregate‑only reporting to protect privacy, as outlined in the deployment guide section of the paper. Finally, because the measurement reveals a potential trade‑off between safety and retention, operators must weigh the business incentive to reduce refusals against the risk of exposing users to harmful content.

Grafy perspective

While RAW is a research contribution, Grafy provides tools that let creators and researchers explore its findings without building the pipeline from scratch. The Grafy Reader app can ingest the RAW corpus (labels, embeddings, moderation scores) and let you query specific cells or run custom aggregations via natural‑language prompts. If you want to prototype your own refusal‑aversion study, you can use Grafy Chat to generate stochastic responses from any hosted model and then feed the outputs into a small‑scale version of the RAW pipeline using Grafy Code. For visualising matched prompt clusters, the Grafy Graph interface offers an interactive way to see how near‑duplicate prompts are grouped and where the treatment variation occurs. Together, these apps let you experiment with causal measurement techniques on your own data while staying within Grafy’s secure, multimodal workspace.

Frequently asked questions

How does RAW differ from a traditional A/B test of refusal behavior?

Traditional A/B testing requires deliberately exposing users to different model versions, which can be ethically risky when refusals involve safety concerns. RAW bypasses the need for live experiments by exploiting natural stochastic variation in existing logs, allowing operators to infer causal effects without ever presenting a user with a deliberately risky prompt.

Can RAW be applied to models that always respond deterministically?

No. RAW depends on temperature‑sampled randomness that occasionally leads the same prompt to be refused and complied with. A fully deterministic model would produce identical outputs for identical inputs, eliminating the quasi‑random treatment variation needed for identification.

What cost should a small startup expect to incur when running RAW on its logs?

The authors report an average compute cost of about $$0.30$ per  1,000\,1{,}000 conversations, covering embedding generation, moderation scoring, and the AIPW estimation. For a startup handling a few hundred thousand daily chats, the marginal expense would be a few dollars per day, making the method financially feasible for most production workloads.


Source: Grafy Research Lab — RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs