AI agents waste tokens—7 patterns and practical fixes

Production‑grade AI agents often start as quick prototypes, but by the third month their token bills can explode. Vinod Chugani explains in a MachineLearningMastery tutorial that seven common patterns are responsible for most of that waste, and each has a concrete architectural remedy.

---

What token‑waste patterns do production AI agents fall into?

The article lists seven recurring mistakes:

1. Context window accumulation – appending every turn to the prompt array.
2. Uncompressed tool result ingestion – feeding raw, verbose JSON directly to the model.
3. Retrieval thrash ("grep‑and‑dump") – pulling whole files when only a snippet is needed.
4. Static tool‑schema bloat – injecting all tool definitions on every call.
5. Unbounded chain‑of‑thought loops – allowing retries to run indefinitely.
6. Ignoring KV cache and prompt prefixing – placing dynamic data before static content, breaking provider caching.
7. Monolithic routing to frontier models – using the most expensive model for trivial tasks. These patterns inflate both input and output token counts, leading to higher latency and larger API bills.

---

Why does an ever‑growing context window hurt latency and cost?

Treating the prompt array as a database causes the model to reread the entire conversation on each inference step. As the number of turns grows, attention costs increase quadratically while token usage climbs linearly. The fix, per the tutorial, is a sliding context window combined with semantic rolling summarization: keep the most recent N turns verbatim, compress older turns into a dense summary, and store the full state in an external store such as SQLite or Redis. This reduces the payload sent over the wire and keeps inference time roughly constant.

---

How can uncompressed tool results be trimmed before reaching the model?

Third‑party APIs (Stripe, Jira, AWS, etc.) return deeply nested JSON designed for machines, not language models. Dumping that payload can consume thousands of tokens. The recommended approach is a middleware transformation layer that:

• Strips null values and non‑essential keys.
• Converts the remaining structure to a compact format like YAML or Markdown.
• Removes repetitive metadata (timestamps, thread IDs) unless required for pagination or state tracking. The article warns that over‑filtering can drop needed fields such as next_token, so developers should tailor the filter to each tool’s contract.

---

What is retrieval thrash and how does chunked AST‑aware search help?

When agents use naïve text matching to locate a function, they often pull the entire source file (e.g., a 4,000‑line Python module) into the prompt. This is the retrieval equivalent of reading an encyclopedia for a single fact. The fix is chunked, AST‑aware retrieval: during indexing, parse code into an abstract syntax tree, then query the index for the exact function signature, docstring, and immediate dependencies. Libraries such as Tree‑sitter make this feasible. The trade‑off is the need for an asynchronous indexing pipeline to keep the vector store in sync with the codebase.

---

How does static tool‑schema bloat affect token budgets, and how can it be pruned?

Many frameworks inject the full definition of every available tool on each LLM call. With dozens of tools, this can exceed 1,500 tokens, even when only a handful are relevant. The article suggests a finite‑state‑machine‑driven tool pruning strategy: based on the agent’s current phase (e.g., discovery vs. execution), only the necessary subset of tool schemas is sent. While this saves tokens, it can reduce prompt‑caching efficiency because the static prefix changes whenever the tool set changes.

---

Which safeguards stop runaway chain‑of‑thought loops?

Autonomous agents sometimes enter infinite retry cycles: a failed tool call triggers another reasoning pass, which repeats the same call, and so on. Since output tokens are billed at a higher rate, a runaway loop can quickly become expensive. The recommended guardrail is an orchestrator‑level circuit breaker that caps both the number of retries and the output token budget per action. When the limit is hit, the system should either fall back to a deterministic routine or escalate to a human‑in‑the‑loop. Frameworks like LangChain and LlamaIndex already expose retry hooks that can be repurposed for this.

---

How can developers implement these fixes with Grafy’s apps?

Grafy offers several tools that map directly to the fixes above:

• Grafy Code (/code) lets you build, test, and run code‑centric agents. You can embed the sliding‑window summarizer or the middleware that compresses API payloads directly into your agent’s orchestrator logic.
• Grafy Graph (/graph) provides a visual branching graph of prompts, outputs, and edits. Use it to prototype the finite‑state‑machine that prunes tool schemas, visualizing which schemas are active in each branch.
• Grafy Chat (/chat) supports multiple models side‑by‑side, making it easy to route simple tasks to a cheaper model while reserving frontier models for complex reasoning—addressing the monolithic routing issue.
• Grafy Bots (/bots) lets you embed a lightweight agent on a site. By leveraging the bot’s built‑in cache‑aware prompt builder, you can ensure static content sits at the top of the request, preserving KV‑cache hits. If you are not building a full agent stack, you can still explore the concepts with Grafy Reader (/reader) to experiment with summarization of long conversation logs, or use Grafy Chat to manually test how different prompt structures affect token counts.

---

What does this mean for creators, developers, and researchers?

Understanding token waste is no longer an academic exercise; it directly impacts monthly cloud spend and user experience. By applying the seven fixes—sliding windows, payload compression, AST‑aware retrieval, state‑driven tool pruning, circuit breakers, cache‑friendly prompting, and model‑level routing—teams can keep inference latency low, avoid surprise bills, and maintain deterministic behavior. Grafy’s multimodal workspace makes it straightforward to prototype each pattern, measure token usage, and iterate quickly, turning cost‑optimization from a post‑mortem activity into a design‑time habit.

---

Sources

• Vinod Chugani, 7 Ways Your AI Agent Is Wasting Tokens (And What to Do Instead), MachineLearningMastery, Oct 30 2026. https://machinelearningmastery.com/7-ways-your-ai-agent-is-wasting-tokens-and-what-to-do-instead/

---

Frequently asked questions

How much can I actually save by applying these token‑waste fixes?

Savings vary by workload, but developers report linear reductions in input token count (often 30‑60%) and a corresponding drop in latency. Output token caps from circuit breakers prevent runaway costs, typically keeping monthly bills within a predictable range.

Do I need to rewrite my entire agent to use sliding windows and summarization?

No. Most frameworks let you replace the prompt‑assembly step with a summarizer function. You keep the original reasoning logic unchanged while only the context‑building layer changes.

Will pruning tool schemas break the caching benefits of KV‑cache?

Pruning does shift the static prefix, which can reduce cache hits. However, the token savings from sending fewer schemas usually outweigh the slight increase in latency, especially for agents that handle many concurrent requests.

---

Source: MachineLearningMastery — 7 Ways Your AI Agent Is Wasting Tokens (And What to Do Instead)