Does Format Matter? WebTaskBench Results

HTML was designed for human eyes.

AI agents have no eyes.

The page an agent receives was built to be rendered — laid out with CSS, painted with gradients, animated with transitions. Agents skip all of that. They parse the raw markup, wade through thousands of tokens of presentation noise, and try to extract meaning from a format that was never designed for them.

The average web page delivers 33,000 tokens of HTML to an AI agent. Three-quarters of those tokens encode visual presentation — CSS classes, layout wrappers, decorative elements — that is completely irrelevant to agent reasoning.

The companion paper modeled possible annual industry waste between $1 billion and $5 billion under its stated traffic, preprocessing, and pricing assumptions. That is a scenario estimate, not observed industry spend.

An open benchmark designed to isolate the effect of format on agent performance. Same tasks, same models, same websites — only the input representation changes.

Task categories:

1. Extraction — pull specific data from a page
2. Comparison — compare information across page sections
3. Navigation — identify correct links and actions
4. Summarization — produce accurate page summaries
5. Adversarial — resist noise, deceptive markup, hidden content
6. Interactive — identify clickable elements and form fields

The historical runner recorded lower SOM latency than HTML for the two displayed model treatments. This was a runner-specific observation, not a general speed guarantee. But here was the surprise:

This suggests that explicit structure reduces model reasoning overhead. The model spends less time figuring out what the page contains and more time answering the question.

We identified four types of agent hallucination when reading web pages:

Structural hallucination — the agent invents page elements that don't exist, like buttons or sections. Most common with HTML input where the model must infer structure from markup noise.

Content hallucination — the agent misquotes or fabricates text from the page. Occurs across all formats but is amplified by token-heavy inputs that push content outside the model's attention window.

Attribution hallucination — the agent attributes information to the wrong section or element. Common when HTML nesting is deep and relationships between elements are ambiguous.

Inference hallucination — the agent draws conclusions not supported by page content. Format-independent but triggered more often when the model has to reason about noisy inputs.

The SOM format includes provenance metadata — every structured element traces back to its source in the original HTML. This enables something raw HTML and markdown cannot: programmatic verification of agent claims.

When an agent makes a claim about page content, the provenance chain lets you verify whether that claim is grounded in an actual page element — or hallucinated. In this example, the price is verified but "unlimited API calls" has no source element, flagging it as potentially fabricated.

WebTaskBench demonstrates that format is not a minor optimization — it is a fundamental infrastructure problem. The gap between HTML and structured representations is too large to close with better models. We need better formats.

The Semantic Object Model, the Agent Web Protocol, and cooperative content negotiation are the building blocks. The benchmark, evaluation harness, and all data are open source.

Everything is available:

• WebTaskBench — the full benchmark dataset, evaluation harness, and results
• Plasmate — the headless browser engine that produces SOM representations
• SOM Specification — the Semantic Object Model format definition
• Agent Web Protocol — the communication protocol for agent-web interaction
• All papers — published at dbhurley.com/papers