Monte Carlo Study Reveals Inconsistent LLM Performance on Data‑Science Tasks

What is DS‑QA and why it matters

A team from the University of Edinburgh has released DS‑QA, a 100‑question benchmark that draws directly from UK university data‑science curricula. The questions span 16 public datasets and cover nine task dimensions – from data wrangling to statistical modelling – at three difficulty levels (easy, medium, hard). By grounding the test set in real coursework, the authors aim to move beyond synthetic prompts and provide a more realistic gauge of how large language models (LLMs) handle end‑to‑end data‑science workflows.

How the Monte Carlo protocol evaluates consistency

Instead of a single prompt‑response pair, the study runs each question 100 times under identical settings and records every output. The authors then score the results with a three‑layer framework:

1. General properties – runtime, verbosity (both word‑ and token‑level), and textual similarity across runs.
2. Coarse‑grained metrics – overall accuracy, whether generated code executes, and completeness of the answer.
3. Task‑specific metrics – up to 35 fine‑grained measures tailored to each of the nine question types (e.g., regression‑score for linear models, visual fidelity for plots).

All of this is captured in a Monte Carlo analysis that highlights both quality and stability of LLM outputs.

Key findings on LLM reliability

The authors applied the protocol to several OpenAI models—gpt‑4o‑mini, gpt‑4o, gpt‑4.1‑nano, and gpt‑4.1‑mini—and uncovered four notable patterns:

• High variability on visualisation and supervised‑learning tasks – especially at medium and hard difficulty, where answer deviation was far larger than for descriptive‑statistics or data‑cleaning questions (see Figure 4 in the paper).
• Verbosity correlates with token cost – a rise of 0.1 in the word‑verbosity ratio leads to an increase of roughly 0.013 in the token‑verbosity ratio, offering a concrete way to estimate API cost from answer length.
• gpt‑4o‑mini shows the best text‑similarity – it produces the most consistent wording while keeping token usage reasonable, outperforming the larger gpt‑4o and the gpt‑4.1‑nano variant.
• Temperature matters for gpt‑4.1‑mini – settings above 1.0 introduce noticeable randomness, making the model less reliable for repeatable pipelines.

These results suggest that a single run cannot reliably certify an LLM’s suitability for production data‑science pipelines; consistency must be measured explicitly.

Practical implications for developers and researchers

For anyone building AI‑augmented analytics tools, the study offers three actionable take‑aways:

1. Benchmark with repetition – Incorporate multiple runs into your evaluation suite. A single accuracy figure can mask large swings in code executability or plot quality.
2. Watch verbosity settings – If cost is a concern, monitor the word‑verbosity ratio; small reductions can noticeably cut token consumption without harming answer quality.
3. Tune temperature deliberately – For tasks that demand deterministic outputs (e.g., generating reproducible notebooks), keep temperature at or below 1.0, especially when using gpt‑4.1‑mini or similar models.

By aligning model selection and prompting strategies with these insights, teams can reduce downstream debugging and improve the reliability of AI‑driven data‑science pipelines.

How Grafy users can explore the DS‑QA findings

Grafy offers several tools that map directly onto the workflow described in the paper:

• Grafy Reader lets you upload the DS‑QA PDF and ask natural‑language questions about the benchmark, the evaluation framework, or specific results. This is ideal for quickly surfacing the 35 task‑specific metrics without scrolling through the entire paper. (See the Reader app at /reader.)
• Grafy Code provides an AI‑assisted coding environment where you can script a Monte Carlo loop, call any of the OpenAI models, and automatically capture runtime, verbosity, and execution success. The platform’s multi‑model support mirrors the paper’s side‑by‑side comparison of gpt‑4o‑mini, gpt‑4o, and others. (Access the studio at /code.)
• Grafy Graph can be used to visualise the distribution of answer similarities across the 100 runs for a given question. By branching a prompt node into 100 child nodes, you obtain a live graph of variability that echoes the authors’ Figure 4.

If you prefer a higher‑level overview, you can simply drop the DS‑QA PDF into Grafy Files and let the built‑in converters extract tables and figures for further analysis.

---

Source: “One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks,” arXiv 2611.09729 (2026).

Frequently asked questions

Does the Monte Carlo protocol require special hardware?

No. The study runs each LLM call sequentially using the cloud APIs provided by OpenAI. The only requirement is enough API quota to issue 100 calls per question; the authors performed the experiments on standard workstation hardware.

Which LLM showed the most stable performance on code generation?

According to the paper, gpt‑4o‑mini achieved the highest text‑similarity and reasonable token usage across repeated runs, making it the most stable for generating executable code in the evaluated set.

Can I use Grafy to replicate the entire DS‑QA benchmark?

Yes. By combining Grafy Reader (to ingest the benchmark PDF), Grafy Code (to script the 100‑run Monte Carlo loops), and Grafy Graph (to visualise variability), you can recreate the authors’ evaluation pipeline within Grafy’s multimodal workspace.

---

Source: Arxiv ML2 Recent — One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks