VisualArtifactSpec 1.0 benchmark tests AI on presentations, graphics
Researchers unveiled VisualArtifactSpec 1.0, a 211‑task benchmark spanning presentations.
Source: Grafy Research LabWhat is VisualArtifactSpec 1.0 and why it matters?
The Grafi Research Lab released VisualArtifactSpec 1.0 (VAS) on 17 November 2026 as a new benchmark for multimodal AI agents that create visual artifacts. According to the authors (Ouyang et al., 2026), visual‑artifact tasks probe an agent’s ability to turn a written specification and source material into a concrete output – a slide deck, a chart, an animation, or a web interface – that satisfies real‑world needs. By covering four distinct artifact families, VAS aims to reveal whether a model’s authoring skill is broadly applicable or limited to a single medium, a gap that earlier benchmarks left largely unaddressed.
How was the benchmark built and what tasks does it include?
VAS comprises 211 creation and modification tasks spread across presentations (P), static graphics (G), animations/videos (M) and web interfaces (W). Each task lists:
- an authoring objective (e.g., “summarise research findings for a non‑technical audience”),
- source materials such as PDFs, images, or raw data,
- constraints (page limits, color palettes, interaction requirements), and
- deliverables (a PowerPoint file, an SVG chart, an MP4 clip, or an HTML page). The construction process had three stages: (1) gathering feasible source assets, (2) drafting task descriptions and reference outputs, and (3) running pilot agents to verify that the tasks were executable. The authors note that the benchmark deliberately mixes audiences (technical vs. consumer) and purposes (educational, promotional, analytical) to test how well agents adapt content, organization, visuals, and delivery to context.
How are the tasks evaluated?
VAS separates evaluation into two orthogonal dimensions:
- Task Fulfillment – a binary check that the agent respects explicit requirements (correct file type, functional links, required data fields, etc.).
- Artifact Performance – a rubric‑based score on four criteria: Content, Organization, Visuals, and Delivery. The rubric is instantiated for each task with rating anchors that reflect the intended audience, purpose, and medium. For example, a slide deck aimed at senior executives is judged on concise messaging and high‑impact visuals, whereas a tutorial web page is judged on navigability and step‑by‑step clarity. Judgments are grounded in three evidence streams: the original source materials, the rendered artifact (static image, video frame, or interactive preview), and any interaction logs when the artifact is executable. This evidence‑based approach lets evaluators distinguish a missed requirement from a sub‑optimal design choice.
What did the experiments reveal about current AI models?
The authors tested nine model‑harness configurations – combinations of foundation models and prompting strategies – on the full set of 211 tasks. Overall scores ranged from 42.12 to 70.52 (out of 100). Two patterns stood out:
- Task Fulfillment was generally high, indicating that most agents could produce files of the correct type and include required data.
- Artifact Performance accounted for 73.4 % of the performance gap between the best and worst configurations, showing that fine‑grained authoring decisions (layout, visual hierarchy, interaction flow) remain a major weakness. Cross‑family analysis (Section 4.3 of the paper) showed that a model that excelled at static graphics often lagged on video timing or web interaction, confirming that strengths do not automatically transfer across media. The case studies in Section 5 illustrate concrete failures: a mobile‑reading version of a research paper omitted crucial headings, an animation lost temporal consistency, and a web prototype failed to preserve key navigation cues. These findings suggest that future work should focus on context‑aware prompting and post‑generation refinement rather than solely on raw generation capability.
How can creators, developers, and researchers use this benchmark with Grafy?
- Explore the benchmark data – Use the Grafy Reader app to upload the VAS PDF or the HuggingFace dataset link and ask the model to summarize task categories, view sample specifications, or extract rubric criteria.
- Prototype individual tasks – The Creator app can generate static graphics or short video clips from a VAS prompt, letting you compare your output against the reference artifact.
- Iterate on presentation or web tasks – Combine Film Studio for animation snippets and Graph to track prompt variations and visual edits across attempts, mirroring the evidence‑based evaluation pipeline described in the paper.
- Benchmark your own agents – By following the three‑stage task construction (source, specification, reference) you can build a private VAS‑style suite inside Grafy’s Code studio, run your model, and record fulfillment and performance scores using the same rubric dimensions. While Grafy does not yet ship a turnkey VAS scorer, the modular nature of the platform lets you recreate the evaluation flow with existing tools, turning the academic benchmark into a practical development loop.
Frequently asked questions
What artifact families does VisualArtifactSpec 1.0 cover?
VAS includes four families: presentation decks, static graphics (charts, posters), animations or short videos, and interactive web interfaces. Each family is represented by multiple tasks that vary in audience and purpose.
How does the benchmark separate requirement compliance from design quality?
The evaluation uses two layers: Task Fulfillment checks that explicit specifications (file type, required fields, functional links) are met, while Artifact Performance scores the final output on Content, Organization, Visuals, and Delivery using task‑specific rating anchors.
Can I use Grafy to run the VAS benchmark on my own models?
Yes. You can import the VAS dataset via the Reader app, generate artifacts with Creator, Film Studio, or Code, and log prompt‑output chains in Graph. Although Grafy does not provide an automated scorer, you can apply the published rubric manually or build a custom scoring script in the Code studio.
Source: Grafy Research Lab — VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications


