Researchers expose symbolic vocabulary weaknesses in LLM planners
A new paper from Grafy Research Lab shows that large language model planners falter when symbolic action and object names are obfuscated.
Source: Grafy Research LabWhat the new study reveals
The paper titled Adversarially Attacking Symbolic Vocabulary Vulnerabilities In LLM Planners was posted on arXiv on 16 November 2026 by the Graf Grafy Research Lab. According to the authors, as large language models (LLMs) are increasingly used as autonomous agents, their ability to solve automated planning problems is a core capability. The study demonstrates that when the symbolic vocabulary – the names of actions, predicates, and objects that define a planning task – is deliberately obfuscated, LLM‑based planners lose reliability and fall short of traditional symbolic planners.
Why symbolic vocabularies matter for LLM planning
Symbolic vocabularies act as a bridge between a model’s internal language understanding and the formal structure of a planning problem. In conventional settings, the model can lean on commonsense associations tied to familiar words (for example, recognizing that move implies a change of location). The paper points out that when those cues are removed or replaced with arbitrary tokens, the model can no longer draw on its world knowledge, exposing a gap between linguistic fluency and logical reasoning. This gap is crucial because many emerging AI agents rely on LLMs to generate or adapt plans on the fly, often without a human‑in‑the‑loop.
How the authors crafted adversarial attacks
To probe the vulnerability, the researchers designed an adversarial procedure that systematically replaces the original symbolic names with synthetic tokens that preserve the underlying planning semantics but hide any semantic hint. The attack does not alter the problem definition itself; rather, it manipulates the surface language that the LLM sees. By feeding the obfuscated task description to several state‑of‑the‑art reasoning models, the authors measured the drop in planning success compared with a baseline that uses the original vocabulary. The paper notes that recent reasoning models have shown improved performance on tasks with such obfuscation, yet the new adversarial method still manages to degrade their effectiveness.
Key findings and their implications
The empirical results indicate a consistent decline in planning accuracy across the tested models when symbolic vocabularies are attacked. The authors conclude that LLM planners are currently over‑reliant on lexical cues rather than robust logical inference. This finding suggests that deploying LLM‑driven agents in safety‑critical domains—such as robotics, logistics, or autonomous software orchestration—could be risky if an attacker can manipulate the symbolic naming scheme. Moreover, the work highlights a broader research direction: developing planning architectures that separate language understanding from the symbolic reasoning core, or that can recover the original semantics even when the surface symbols are scrambled.
What this means for developers of autonomous agents
For practitioners building autonomous systems, the study serves as a warning to audit the robustness of any LLM‑based planning component. Developers should consider adding a verification layer that checks the logical consistency of generated plans independent of the wording used. Techniques such as symbolic grounding, constraint checking, or fallback to classical planners can mitigate the exposure highlighted by the paper. In addition, the research encourages the community to explore training regimes that explicitly randomise symbolic vocabularies during fine‑tuning, thereby strengthening the model’s ability to reason beyond word‑level patterns.
Exploring the topic with Grafy
Grafy provides tools that let you experiment with the concepts described in the paper without writing extensive code. You can upload the arXiv PDF to Grafy Reader to ask the AI to summarise sections, extract the experimental setup, or compare different planning models. If you want to test how a current LLM reacts to obfuscated vocabularies, you can craft prompt variations in Grafy Chat and observe the generated plans side by side. By switching between multiple models available in the chat interface, you can gauge which ones are more resilient to the lexical attacks described in the study. This hands‑on approach helps developers assess the safety of their own LLM‑driven pipelines.
Frequently asked questions
Does the paper prove that all LLM planners fail with obfuscated vocabularies?
The research shows a systematic drop in performance for the models tested, but it does not claim universal failure. The authors suggest that many existing planners are vulnerable, while acknowledging that future architectures could mitigate the issue.
Can the adversarial technique be applied to any planning domain?
The attack targets the symbolic naming layer, so it can be adapted to any domain where tasks are expressed with action and predicate names. Its effectiveness depends on how much the underlying planner relies on lexical cues.
How can I use Grafy to evaluate my own LLM‑based planner?
Upload your planner’s output and the original task description to Grafy Reader for analysis, then recreate the obfuscated version of the task and query the same model via Grafy Chat. Comparing the two responses will reveal any sensitivity to symbolic vocabulary changes.
Source: Grafy Research Lab — Adversarially Attacking Symbolic Vocabulary Vulnerabilities In LLM Planners


