Vision-as-a-Tool paradigm lets coding agents call external visual utilities
A new Vision-as-a-Tool paradigm lets multimodal and text‑only coding agents offload visual reasoning to external utilities.
Source: Grafy Research LabMultimodal agentic coding has just gained a new architectural option: the Vision-as-a-Tool (VaaT) paradigm, which treats visual reasoning as an on‑demand utility rather than an intrinsic capability of the coding agent. The approach, introduced in a paper titled Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding (arXiv:2611.17797v1), promises to break the performance ceiling imposed by an agent’s built‑in perception and to bring text‑only coders into the visual domain.
What is the Vision-as-a-Tool (VaaT) paradigm?
The VaaT paradigm reframes a large multimodal language model (MLLM) from being a permanent visual sensor into a callable external service. Instead of feeding images directly into the coding model at every iteration—a practice the authors call the “Thinking‑with‑Images” paradigm—agents can now invoke a set of visual utilities only when they need perceptual feedback. This decoupling means that a text‑only coding agent, which has no native vision, can still perform visual reasoning by delegating the task to an external visual module. Conversely, a multimodal agent can keep its intrinsic vision lightweight and augment it on demand, avoiding the bottleneck where the quality of generated code is bounded by the agent’s own visual fidelity.
How does VHarness implement VaaT?
The authors instantiate VaaT in a framework they call VHarness. VHarness offers five atomic actions that act as general‑purpose visual tools:
- transcribe – extracts a holistic structural description from an image.
- grep – locates visual elements matching a query.
- inspect – returns fine‑grained attributes of a selected region.
- align – checks whether a rendered artifact matches a target visual assertion.
- diff – compares two images and highlights differences.
These actions are deliberately analogous to classic shell utilities (grep, diff) but operate on visual data. During an agentic coding loop, the coding model decides which action to call, passes the relevant image(s), and receives a textual response that informs the next code edit. The process is iterative: the agent writes or refines code, renders the output, calls a visual utility if the render looks off, and repeats until the visual and code artifacts align.
Which tasks and models were evaluated?
To test the generality of VHarness, the researchers benchmarked three vision‑to‑code domains:
- Image‑to‑HTML – reconstructing a webpage layout from a screenshot.
- Image‑to‑SVG – generating vector graphics that match a raster illustration.
- Image‑to‑3D – producing a procedural 3D model that reproduces the shape of an input image.
Four multimodal coding agents with built‑in vision were evaluated: GPT‑5.5, GPT‑5.6‑sol, Claude Opus 4.8, and GLM‑5.3‑Flash. Additionally, a text‑only agent, GLM‑5.2, was tested to demonstrate that VaaT can bring a purely textual model up to speed with multimodal baselines.
What performance improvements were observed?
Across all three tasks, VHarness consistently raised both visual fidelity and geometric accuracy. The paper reports that multimodal agents using VHarness produced higher‑quality renders than the same agents operating under the traditional Thinking‑with‑Images workflow. More strikingly, the text‑only GLM‑5.2, when equipped with VaaT utilities, achieved results competitive with the native multimodal GPT‑5.5 baseline. Qualitative examples in the supplementary material show smoother refinement trajectories: agents iteratively call diff to spot mismatches, then use inspect to query missing attributes, leading to concise code edits that converge faster than a purely vision‑driven loop.
How can Graf by users experiment with VaaT?
Graf by already provides the building blocks needed to prototype a VaaT‑style workflow. Developers can use the Grafy Code app to write and execute code snippets, while the Grafy Reader app can ingest the VaaT paper itself, extract the five atomic actions, and let a language model generate the appropriate prompts for each utility. A practical experiment might look like this:
- Upload an image (e.g., a UI mock‑up) to the Code studio.
- Write a simple rendering script that turns HTML or SVG code into a preview image.
- Use the Reader to ask the model to transcribe the layout description from the original image.
- Feed that description back into the Code studio as a comment, prompting the model to generate initial code.
- Iteratively call the Reader again with a diff prompt (“compare the rendered preview with the original image and list differences”) and let the model edit the code accordingly.
While Graf by does not yet ship a dedicated VaaT service, the combination of Code and Reader lets users emulate the five actions manually. As the platform continues to integrate more multimodal models, a native VaaT plug‑in could become a first‑class feature.
Frequently asked questions
Does VaaT require a vision‑capable language model to work?
No. VaaT is designed precisely to let text‑only models perform visual reasoning by calling external visual utilities. The paper shows that a text‑only GLM‑5.2 can achieve performance comparable to multimodal baselines when equipped with VaaT.
What are the five atomic actions in VHarness?
The actions are transcribe, grep, inspect, align, and diff. They cover structure extraction, element localization, attribute querying, assertion checking, and image comparison, respectively, and together form a minimal yet expressive visual toolset.
Can I use VaaT with Graf by’s existing apps?
Yes. You can combine Grafy Code to generate and run code with Grafy Reader to issue visual‑utility prompts and retrieve textual feedback. This manual composition mimics the VaaT workflow until a dedicated integration arrives.
Source: https://arxiv.org/abs/2611.17797
Source: Grafy Research Lab — Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding


