Study Reveals LLM Agents Mostly Browse Instead of Acting in Public‑Service Tasks

Researchers from the University of St.Gallen released a new benchmark that measures how large language model (LLM) agents behave when given underspecified instructions in public‑service settings. The work matters because public bodies are already deploying generative AI, yet there is little evidence of how such agents decide what to do when the prompt is vague.

What was the study and why it matters

The paper, titled Read and Ignore: What Self‑Directed LLM Agents Do in Public Service Scenarios, builds four expert‑validated scenarios – governance, healthcare, education and environmental monitoring – and runs ten different LLMs under four prompt conditions, generating a total of 2,400 trajectories. A separate set of 1,000 runs in a context‑free sandbox serves as a baseline. According to the arXiv preprint, the study aims to fill a gap in existing agent benchmarks, which either prescribe a concrete task or place the model in an empty world, neither of which reflects the discretion public‑service staff may grant to an AI.

How was the benchmark constructed

Each scenario is a static file tree that contains the records a real team would maintain (budget proposals, ward schedules, course catalogs, water‑quality logs). The environment changes only when the agent writes a file, ensuring that any subsequent read can be attributed to the model itself. Prompt conditions range from an open‑ended "Do what you want" to a directed summary request. The researchers also released a rule‑based taxonomy that labels every turn (e.g., boundary probing, artifact creation, re‑reading) without needing any labeled training data.

Key findings on agent behavior

• Reading is almost universal – agents opened at least one domain file in $96.9\%$ of open‑ended runs.
• Productive output is rare – only $18.7\%$ of those runs produced an artifact or a self‑chosen task, whereas an explicit summary prompt raised production to $86.0\%$.
• Domain context does not boost work – the presence of real records changed the production rate only marginally ($18.7\%$ vs. $16.0\%$, $p=0.11$).
• Model rankings are stable for output, not for reading – the Spearman correlation of production rankings between the sandbox and the domain scenarios was $0.79$, but reading rankings correlated at only $0.19$.
• Exploration dominates – $84.1\%$ of the 94 k turns were pure exploration, and a third of those were re‑reads of files already opened.
• Some artifacts contradict the records – a subset of generated files cited budget numbers that did not match the source data, highlighting a concrete risk.

Figure 1 visualises the output space of all 2,400 runs, showing that runs from the same model cluster together, confirming that model behavior outweighs prompt wording. Figure 5 breaks down, per model, the share of runs that read records, produce artifacts, end with a grounded file, or end with a contradicting file across the three prompt types.

Implications for public‑service AI deployments

The results suggest that simply handing an LLM a folder of files and a loose instruction is unlikely to yield actionable work. Agencies that already see $45\%$ of teams using generative AI (as reported in a UK survey cited by the authors) should therefore enforce explicit task specifications if they expect concrete deliverables. The benchmark also offers a low‑cost diagnostic: institutions can run the same test on their own operational records before procurement to see whether a candidate model tends to read‑only or produce.

How to explore the study with Grafy

Grafy users can reproduce parts of the experiment without writing code. Upload the scenario file trees to the Grafy Files tool, then use Grafy Reader to ask an LLM to "have a look at the files and see what needs doing". The chat interface will show you which files the model opens (you can copy the transcript into Grafy Graph to visualise the turn‑by‑turn actions). This hands‑on approach lets developers and researchers see the same reading‑dominant behaviour reported in the paper and experiment with more explicit prompts to improve production.

Limitations and next steps

The benchmark operates in a static environment, so it cannot capture how agents might react to external events such as new file arrivals or colleague messages. The authors note that the taxonomy was validated on a separate sandbox benchmark, achieving a Fleiss' $\kappa=0.88$, but real‑world deployments could introduce additional failure modes. Future work could extend the test to dynamic file streams or combine it with tool‑use APIs to see whether agents shift from browsing to acting when equipped with write‑back capabilities.

Frequently asked questions

Do LLM agents read the records when given an open‑ended instruction?

Yes. In the study, agents opened at least one domain file in $96.9\%$ of open‑ended runs, showing a strong tendency to explore the provided documentation.

Does providing a concrete task increase the chance of an artifact being produced?

Absolutely. When the prompt explicitly asked for a summary, $86.0\%$ of runs generated an artifact, compared with only $18.7\%$ under the "do what you want" condition.

Can public institutions use this benchmark before buying an AI system?

The authors designed the test to be reusable without any labeled data or fine‑tuning. Agencies can copy their own file structure into Grafy, run the same prompts, and compare the observed reading‑versus‑producing rates against the paper’s baselines.

---

Source: Grafy Research Lab — Read and Ignore: What Self-Directed LLM Agents Do in Public Service Scenarios