Seq@K Benchmark Highlights Need for Feedback‑Aware Model Evaluation

Seq@K, a freshly proposed evaluation protocol, challenges the conventional way we rank language‑model agents by measuring how well they incorporate sequential feedback. The work, posted on arXiv on October 31 2026, argues that Pass@1 and Pass@K miss a core ability of modern agents: the capacity to refine answers after each piece of natural‑language critique.

What is Seq@K and why does it matter?

Seq@K stands for Sequential-at‑K and asks a model to solve a problem in at most K attempts, where after every unsuccessful try the model receives human‑style feedback. Unlike multi‑turn benchmarks that test long‑term planning, Seq@K isolates the refinement loop: can the model understand the comment, adjust its reasoning, and arrive at the correct solution within the allotted attempts? According to the authors, this mirrors real‑world agentic workflows where users iteratively correct outputs—think code assistants, data‑analysis bots, or conversational tutors.

How does Seq@K differ from traditional Pass@K metrics?

Pass@K simply records whether a model produces a correct answer in any of K independent samples drawn from its output distribution. It does not expose the model to any feedback, nor does it require the model to remember or build on prior attempts. Seq@K, by contrast, creates a feedback‑driven trajectory: each failure is followed by a textual hint, and the model must incorporate that hint in the next try. The paper’s experiments on seven established benchmarks show that rankings shift when models are judged by Seq@K rather than Pass@K, indicating that some models excel at raw generation while others are better at learning from corrections.

What were the key findings across the evaluated models?

The authors evaluated 16 frontier models spanning different architectures and scaling regimes. Their analysis uncovered three practical takeaways:

1. Model rankings change – A model that tops Pass@K on a coding benchmark may fall behind a smaller model that more effectively uses feedback, and vice‑versa.
2. Pareto frontiers become richer – When plotting cost versus performance under Seq@K, the frontier includes configurations that were invisible under Pass@K, allowing practitioners to pick cheaper models that still meet a feedback‑driven success threshold.
3. Gains from optimal configuration are volatile – The performance gap between the best and second‑best model can widen or shrink depending on whether the evaluation uses Pass@K or Seq@K, suggesting that cost‑benefit analyses need to be benchmark‑specific.

How does the nature of feedback influence model behavior?

The study experimented with several feedback styles: terse error messages, detailed explanations, and even contradictory hints. Results indicate that models are sensitive to feedback granularity. Detailed feedback tends to accelerate convergence for larger models, while concise hints are sometimes more effective for compact models that struggle with lengthy context. Moreover, the authors observed a spectrum of "rigidity": some models cling to their original answer despite clear corrective cues, whereas others quickly abandon a flawed line of reasoning. This suggests that designing the feedback loop—its tone, length, and specificity—is as important as selecting the underlying model.

What does Seq@K mean for developers building agentic systems?

If you are constructing a chatbot that helps users write code, a research assistant that iterates over literature summaries, or any system that expects users to point out mistakes, Seq@K offers a more realistic yardstick. Instead of optimizing for a single‑shot success rate, you can benchmark how many feedback cycles your pipeline needs on average, and whether a cheaper model can meet your latency or cost constraints when feedback is available. The protocol also encourages systematic testing of prompt designs for feedback ingestion, which can be automated through tools that simulate user critiques.

How to try Seq@K‑style evaluation with Grafy

Grafy does not ship a dedicated Seq@K module, but the workflow can be assembled with existing apps:

• Use Grafy Chat to run a model against a task prompt, capture the output, and feed it into a second turn where you manually (or via a scripted prompt) provide corrective feedback.
• Record each turn in Grafy Graph, which automatically branches prompts and responses, letting you visualise the refinement trajectory across K attempts.
• For a deeper dive into the benchmark papers themselves, upload the PDF to Grafy Reader and ask the assistant to extract the Seq@K methodology, compare results, or generate a summary. By chaining these apps, creators can reproduce the Seq@K experiment on their own data, compare model behaviours, and even build a custom dashboard that tracks success rates per K.

What are the broader implications for AI research?

Seq@K pushes the community to acknowledge that many practical AI deployments are interactive rather than one‑shot. It highlights a gap in current leaderboards, which often reward raw generative power but overlook the ability to learn from dialogue. Future research may extend Seq@K to longer horizons, integrate reinforcement‑learning‑from‑human‑feedback loops, or combine it with automated error detection to create fully self‑correcting agents. As the paper notes, careful design of feedback mechanisms will be a critical factor in the next generation of trustworthy AI assistants.

Frequently asked questions

Does Seq@K replace Pass@K as the new standard metric?

Seq@K complements rather than replaces Pass@K. It measures a different capability—iterative refinement—while Pass@K still captures raw generation quality. Both metrics together give a fuller picture of a model’s suitability for feedback‑driven tasks.

Can Seq@K be applied to non‑textual modalities such as images or code?

The protocol is modality‑agnostic in principle; any task where a model can receive textual feedback about a previous output (e.g., image captions, generated code snippets) can be evaluated with Seq@K. The original paper demonstrated it on code, math, and reasoning benchmarks.

How many sequential attempts (K) are typically needed for a model to succeed?

The paper reports that success rates improve as K grows, but the exact number varies by task and model. Some models resolve the problem within two attempts, while others may need the full K budget to converge. Practitioners should tune K based on latency and cost constraints.

---

Source: arXiv AI — Seq@K: Evaluating Language Models Under Sequential Feedback