Grafy News
Try Grafy free

FieldGuide Cuts Token Use for AI Agents Exploring Journalism Corpora

Researchers introduced FieldGuide, a corpus‑map tool that lets AI agents navigate document collections more efficiently.

Illustration for the news story: FieldGuide Cuts Token Use for AI Agents Exploring Journalism CorporaSource: Grafy Research Lab

FieldGuide, a new corpus‑mapping system for generative AI agents, was released on arXiv on 18 Nov 2026. It promises to lower the token and time costs of exploratory document work, a frequent bottleneck for journalists handling large public‑record collections.

What is FieldGuide and why does it matter?

FieldGuide is an agent‑facing “corpus map” that organizes a set of documents into a hierarchical topic structure. By exposing this map through a set of bounded tools, an autonomous agent can first get a high‑level view of the collection, then drill down to specific passages only when needed. The motivation is that exploratory reporting often starts with vague questions; agents that ingest whole files waste tokens on irrelevant text, inflating costs and latency. Reducing those expenses makes AI‑assisted journalism more affordable and scalable, especially as frontier LLMs remain pricey.

How does FieldGuide work?

The system has two phases. During offline indexing, it extracts page‑aligned text from each file (using OCR when necessary), normalizes it, and builds a topic hierarchy based on the content. Metadata such as entities, monetary amounts, and document quality flags are also recorded. At runtime, an agent can invoke seven bounded operations: (1) get an overview of top‑level topics, (2) expand a branch to see sub‑topics, (3) inspect document‑level metadata, (4) perform scoped keyword search within a chosen branch, (5) read a bounded window of text, (6) retrieve entity lists, and (7) request quality warnings. These primitives keep the agent’s context small and focused, separating navigation from evidence gathering.

What were the experimental results?

Nick Hagar evaluated FieldGuide on five journalistic exploration tasks that used 38 public‑record PDFs from the Chicago Police Department. The study compared three setups: raw‑file access (the agent reads whole PDFs), a LanceDB‑based retrieval‑augmented generation (RAG) pipeline, and FieldGuide. Across the five prompts, FieldGuide used 27 % fewer tokens and 41 % less elapsed time than raw‑file access. Compared with RAG, it saved 38 % of tokens and 21 % of time. An accuracy audit found only four minor issues in FieldGuide’s outputs, a rate comparable to the other conditions. In blind quality ratings of generated news leads, FieldGuide’s average scores were higher than RAG’s but lower than the raw‑file condition’s, suggesting a trade‑off between efficiency and open‑ended discovery.

What does this mean for journalists and AI developers?

The findings indicate that giving agents a structured map of a corpus can substantially cut inference costs without dramatically harming answer quality. For newsroom teams, this could translate into lower cloud bills and faster turnaround on investigative pieces that rely on hundreds of PDFs. For developers building agentic tools, the work highlights the value of separating orientation (finding where to look) from evidence (reading the exact text). However, the modest dip in lead‑quality scores also warns that overly restrictive navigation might miss serendipitous connections that human reporters often uncover.

How can Grafy users explore similar workflows?

Grafy does not yet ship a dedicated corpus‑map feature, but the concepts can be experimented with using existing apps. The Reader app lets you upload PDFs and query them with AI, providing a simple retrieval interface. Pairing Reader with Chat, you can manually build a topic hierarchy by asking the model to summarize collections and then issue scoped searches within the hierarchy. For a more visual approach, the Graph app can capture the sequence of prompts you use to navigate the corpus, preserving the reasoning chain for later review. These combinations let creators, developers, and researchers prototype the orientation‑then‑evidence pattern that FieldGuide formalizes.


Source: https://arxiv.org/abs/2611.21647

Frequently asked questions

How much token reduction did FieldGuide achieve compared with raw‑file access?

In the evaluation, FieldGuide reduced token consumption by 27 % relative to the raw‑file baseline. This saving stems from the system’s ability to limit the amount of source text the agent reads at each step.

Does FieldGuide completely replace retrieval‑augmented generation?

No. FieldGuide complements RAG by providing a higher‑level navigation layer. While it uses fewer tokens than the RAG pipeline in the reported tests, it may also constrain the breadth of discovery, so both approaches can be useful depending on the reporting goal.

Can I use FieldGuide directly inside Grafy without coding?

Grafy does not currently offer a built‑in FieldGuide module. However, you can approximate its workflow by uploading documents to Reader, using Chat to ask for topic summaries and scoped searches, and tracking the interaction path in Graph. This manual setup mirrors the orientation‑then‑evidence steps that FieldGuide automates.


Source: Grafy Research Lab — FieldGuide: A Corpus Map for Token-Efficient Agentic Document Exploration