AOS-Bench Tests Whether AI Agents Can Handle Streams of Work

A new benchmark called AOS-Bench asks whether AI coding agents can do more than finish isolated tasks. According to the paper and its authors, the tool examines whether LLM agents can act as persistent managers of queued work instead of single-task responders. The source is the arXiv paper at https://arxiv.org/abs/2610.33178.

The shift under study matters because coding agents are moving from one request at a time toward persistent assistants that receive a stream of work. A build, a data-processing job and a bug fix might all be pending on the same machine. The agent then has to decide when to take on new tasks, how to delegate and overlap them, and how to stay within CPU and memory limits.

The paper's authors describe a shift from isolated requests to persistent assistants. That means an agent may be responsible for multiple kinds of work over time. The benchmark's abstract says coding agents are moving from handling one request at a time to serving as persistent assistants to which users hand a stream of work.

How did AOS-Bench test dynamic workloads?

AOS-Bench, described according to arXiv:2610.33178, evaluates an agent system as a workload manager. The benchmark uses verified tasks that arrive over time, the source says. Those tasks must be delivered under shared CPU and memory limits. The same tasks run one at a time serve as a per-system reference.

The study examined five model-harness pairs. Its design was meant to test persistent operation, not a single isolated coding problem. According to the source, the benchmark's central question is whether an agent can manage dynamic workloads. The paper frames the problem around task intake, delegation, overlap and resource limits.

The benchmark's method matters because it does not only ask whether an agent solves a task. It asks whether the system can decide what to do first, when to start, how to overlap work and how to respect limits.

Why did isolated task success fail to predict stream success?

Across five model-harness pairs, isolated task success did not carry over evenly, the source reports. On heavy workloads, every system lost 13 to 23 points. On medium workloads, systems spanned 11 points in isolated success but 26 points in stream success. Two systems 0.6 points apart in isolation delivered 54% and 68% of the same tasks.

The source says existing benchmarks cannot measure this because they evaluate each task in isolation. The gap between isolated and stream success suggests that task context, coordination and resource awareness may be missing. For researchers, the finding raises a question about how agent performance should be aggregated when work arrives over time.

What did the benchmark reveal about execution policies?

The source says execution-policy interventions traced much of the gap to task intake. Serial dispatch left 73% of heavy-workload tasks unclaimed. Yet more elaborate management does not necessarily help. A fixed concurrency of two came within four points of the best agent-managed policy on both medium and heavy workloads.

Under every policy, median CPU use stayed below 30% of the provisioned cores while tasks waited, the source notes. These details suggest that intake, concurrency and waiting behavior were part of the evaluation. The paper presents the results as evidence that workload management is a distinct capability.

What does AOS-Bench mean for persistent coding agents?

The results identify workload management as a distinct capability for agents, according to the source. AOS-Bench is presented as a way to measure it. For developers, the implication is that agent evaluation should include queued tasks, shared compute limits and delivery behavior, not only isolated task success.

The source also cautions that more elaborate management does not automatically solve the problem. The benchmark's findings point to task intake and execution policy as areas to study. For AI builders, persistent agents may need explicit queues, prioritization and resource budgets.

How could Grafy users explore this?

Grafy's Code app is the closest listed surface for readers interested in coding agents. Grafy describes Code as an AI coding studio that installs packages and runs code. That is a genuine but narrower connection to the paper's focus on coding agents under resource limits.

A Grafy user curious about the idea could use Code to generate or modify code, then use Reader to study the arXiv paper and related agent research. The Grafy News Desk's constructive take: even without a dedicated AOS-Bench workload manager, the story argues for building agent workflows with clear task queues and compute constraints.

Frequently asked questions

What is AOS-Bench measuring?

It measures whether an agent system handles queued, time-staggered tasks under shared compute limits. According to the source, it evaluates workload management, not just isolated coding success. The benchmark gives verified tasks over time and checks delivery under CPU and memory constraints. It also uses one-at-a-time runs as a reference.

Why did stream performance differ from isolated performance?

The source reports that isolated success did not carry over evenly. Heavy workloads caused losses of 13 to 23 points. Medium workloads showed 11 points isolated versus 26 in stream success, and two systems close in isolation delivered 54% and 68% of the same tasks. This points to gaps in task intake and coordination.

Do better execution policies always fix agent performance?

No, according to the source. Serial dispatch left 73% of heavy-workload tasks unclaimed. More elaborate management did not necessarily help. A fixed concurrency of two came within four points of the best agent-managed policy on medium and heavy workloads. Median CPU use stayed below 30% of provisioned cores while tasks waited.

---

Source: arXiv AI — AOS-Bench: Can LLM Agents Manage Dynamic Workloads?