FilingLoop Benchmark Shows Self‑Evolving LLM Agents Improve SEC Filing Review
Researchers unveil FilingLoop, an interactive benchmark for LLM agents that review SEC filings.
A new benchmark called FilingLoop evaluates how large language model (LLM) agents can review and revise corporate disclosures, and demonstrates a self‑evolving framework that modestly improves compliance rates. The work matters because public filing repositories such as SEC EDGAR are critical for investors, yet automating their review has remained an open problem.
What is the FilingLoop benchmark?
FilingLoop is an interactive benchmark built from historical correspondence between the U.S. Securities and Exchange Commission (SEC) and companies filing for initial public offerings (IPOs). The authors reconstructed 380 review tasks drawn from 96 companies, each task containing a set of unstated deficiencies, missing facts, and revision requirements. Across the whole dataset there are 6,099 eligible requirements that agents must discover, satisfy, or correctly flag as unsupported. An external evaluator, separate from the acting agents, scores the final filings and the agents' responses against these requirements, auditing for unsupported claims and harmful revisions. The benchmark therefore goes beyond static question‑answering by requiring iterative discovery, fact‑gathering, and evidence‑based revision.
How does FilingLoop differ from existing financial QA evaluations?
Traditional financial QA benchmarks focus on single‑turn question answering against a static knowledge base. They do not test an agent’s ability to:
- Identify hidden deficiencies that are not explicitly asked for.
- Seek out missing company facts from auxiliary documents or web sources.
- Validate that a suggested revision meets regulatory evidence.
FilingLoop introduces a looped workflow where the agent first reviews a filing, raises comments, acquires additional data, and then produces a revised document. The evaluation records feedback at the requirement level and feeds it back into the agent’s procedural knowledge for the next case, allowing researchers to measure procedural learning without changing the underlying model weights.
What were the results of the self‑evolving agent framework?
The authors compared two variants:
- Fixed‑policy agents that follow a static set of instructions throughout all cases.
- Self‑evolving agents that incorporate requirement‑level feedback into role‑specific guidance for subsequent cases.
On the evaluated stream, the evolving agents achieved:
- Requirement fulfillment: 27.96 % (up from 24.28 % for the fixed policy).
- Full comment resolution: 20.75 % (up from 16.90 %).
- Total cost increase: 5.8 % relative to the fixed‑policy baseline.
These figures show that procedural adaptation can raise compliance without a proportional explosion in token usage, albeit with a modest cost penalty. Importantly, the learning occurs without any weight updates; the system only adjusts its procedural prompts for later tasks, making the improvement measurable in a single pass.
How can developers and researchers use FilingLoop?
- Benchmarking – By running a custom LLM or chain of tools against the public FilingLoop dataset, teams can quantify how well their agents discover and satisfy regulatory requirements.
- Procedural Prompt Engineering – The benchmark encourages the design of role‑specific guidance that can be updated between cases, offering a low‑cost alternative to fine‑tuning.
- Cost‑Benefit Analysis – Since the evaluation tracks token consumption, developers can experiment with different model selections (e.g., cheaper open‑router models vs. higher‑accuracy Vertex AI models) to find an optimal cost‑performance sweet spot.
- Integration with Existing Workflows – The looped nature of the task aligns with multi‑step pipelines common in compliance automation, making FilingLoop a realistic testbed for end‑to‑end systems.
The authors have released the benchmark code and data alongside the paper, enabling immediate replication and extension.
How does FilingLoop relate to Grafy?
Grafy provides several tools that can help creators, developers, and researchers experiment with the FilingLoop benchmark:
- Grafy Reader lets you upload the PDF of the arXiv paper, ask natural‑language questions about the methodology, and retrieve exact passages without leaving the workspace.
- Grafy Chat enables you to run multi‑model conversations against the same filing task, swapping between models (e.g., OpenRouter’s Claude, Vertex AI Gemini) to compare performance.
- Grafy Graph can visualise the iterative review‑revise loop, branching each step of the agent’s reasoning, the evidence it gathers, and the final compliance score.
By combining these apps, a team can prototype a FilingLoop‑compatible agent, record its prompt history in Graph, and immediately query the underlying paper for guidance—all within a single, multimodal workspace.
What are the limitations and next steps identified by the authors?
The benchmark focuses on U.S. IPO filings, which may not capture the full diversity of corporate disclosures (e.g., quarterly reports, foreign filings). The procedural learning mechanism relies on hand‑crafted guidance updates, which could be further automated with meta‑learning or reinforcement‑learning‑based policy adaptation. Finally, the modest improvement (≈3 percentage points) suggests that more sophisticated knowledge‑retrieval pipelines or hybrid symbolic‑neural systems may be needed to close the gap to human‑level compliance.
Frequently asked questions
How does FilingLoop measure whether an agent’s revision is evidence‑backed?
The external evaluator checks each revised statement against the 6,099 curated requirements. A revision receives a positive score only if the agent provides verifiable evidence (e.g., a cited SEC comment letter) that satisfies the regulatory wording. Unsupported or speculative revisions are flagged as errors.
Does the self‑evolving framework require fine‑tuning the LLM?
No. The framework updates procedural prompts between cases based on requirement‑level feedback, leaving the underlying model weights unchanged. This design isolates the impact of prompt engineering from model training.
Can I try FilingLoop without building a full agent?
Yes. Using Grafy Reader you can explore the benchmark data and the paper’s methodology, then employ Grafy Chat to manually step through a single filing task, asking the model to identify deficiencies and propose revisions. This hands‑on approach lets you gauge feasibility before committing to a full implementation.
More news
Seq@K Benchmark Highlights Need for Feedback‑Aware Model Evaluation
A new arXiv paper introduces Seq@K, an evaluation protocol that measures a language model’s ability to solve tasks within K sequential attempts using feedback.
GAUSS Introduces Workload‑Aware Simulation for LLM Inference Serving
A new paper presents GAUSS, a simulator that predicts full latency distributions for LLM inference.
New AI Method Estimates Multi-Agent Risk Propagation
A new arXiv study proposes IE-Prop, an interaction-conditioned model that estimates propagated risk in LLM multi-agent systems. Researchers report AUROC gains.