Making Sense of the Agentic Web

I have spent two decades building software that connects people and organizations to the web. Marketing automation, CRM systems, open source communities. In every case, the work came down to helping one type of consumer interact with web content more effectively.

Today, the consumer is changing. AI agents are browsing the web on behalf of humans. They read articles, fill forms, compare products, book appointments, and synthesize information across dozens of sources. They do this autonomously, at scale, and increasingly as the default way people access online content.

This shift is what I call the agentic web. It is not a product or a protocol. It is a structural change in who consumes web content and how. Understanding it does not require a computer science degree. It requires stepping back and seeing the pattern that is already emerging.

This post is my attempt to make sense of that pattern. If you are a publisher wondering why your server logs show unfamiliar traffic, a product manager evaluating how AI will affect your roadmap, or simply someone curious about where the web is heading, this is where I would start.

What we mean by "agentic"

The word "agent" gets used loosely in AI discussions. Let me be specific about what I mean.

An AI agent is software that acts on behalf of a human to accomplish a task. The human says "find me flights to Tokyo under $800" or "summarize what happened in Congress this week" or "schedule a dentist appointment for next Tuesday." The agent does the work: browsing websites, reading content, making decisions, and taking actions.

This is different from a chatbot, which waits for you to ask questions and responds with text. It is different from a search engine, which returns a list of links for you to click. An agent does the clicking, the reading, and the deciding. You get the result.

Why this matters now

You might reasonably ask: agents have existed for years. Why does this matter now?

The answer is scale.

Consider what happens when someone asks Claude, ChatGPT, or Gemini a question that requires current information. The model cannot answer from its training data alone. It needs to browse the web, find relevant pages, read them, and synthesize an answer. This happens millions of times per day. Each of those browsing sessions is an agent consuming web content.

The same pattern applies to more complex tasks. When an agent books a flight, it visits airline websites, navigates search forms, compares options, and selects the best result. When an agent researches a company for due diligence, it reads news articles, SEC filings, LinkedIn profiles, and review sites. When an agent monitors competitors, it checks pricing pages, product announcements, and job postings across dozens of sites daily.

The problem with how agents browse today

Here is where things get interesting, and where I have spent the last two years building.

AI agents today browse the web by fetching HTML pages, just like a regular browser. But unlike a browser, agents do not render pixels. They do not see the page. They process the text.

The problem is that HTML was designed for visual rendering. It contains enormous amounts of information about how a page should look: CSS class names, style attributes, layout containers, decorative elements, tracking scripts, advertising code. None of this matters to an agent trying to understand what the page says or what actions are available.

I wrote about this in detail in The Billion Dollar Tax on AI Agents. The short version: a typical web page contains 30,000 to 60,000 tokens of HTML markup. The actual content, the words and actions that matter, might be 5,000 tokens. The rest is presentation noise.

Agents pay to process all of it. Language models charge by the token. Every CSS class name, every tracking pixel, every decorative div consumes tokens that contribute nothing to the agent's understanding. At scale, this waste adds up to billions of dollars per year across the industry.

But cost is not the only problem. The noise also degrades quality. I explored this failure mode in When AI Reads the Web Wrong, looking at cases like the Air Canada chatbot that invented a refund policy. These failures are not random. They follow predictable patterns, and many trace back to the fundamental mismatch between how web content is formatted and how agents need to consume it.

What publishers are facing

If you run a website, you are already experiencing the agentic web whether you realize it or not.

Your server logs contain requests from AI crawlers. Some identify themselves clearly. Others do not. The traffic consumes your bandwidth and server resources. Unlike search engine crawlers, which visit periodically to update an index, agent traffic can be continuous and unpredictable. An AI assistant might fetch your pages every time a user asks a question about your content.

Publishers are responding in different ways. Some block agent traffic entirely, treating it like unwanted scraping. Others tolerate it passively, accepting the load without optimizing for it. A few are starting to cooperate actively, recognizing agents as a legitimate audience worth serving well.

I explored these three responses in The Publisher's Dilemma. The choice matters because it affects how your content appears (or fails to appear) when people use AI assistants to find information. Publishers who block agents become invisible to that channel. Publishers who serve agents well become preferred sources.

The fourth state of the web

To understand where this is heading, it helps to see the pattern across web history.

In The Web's Fourth State, I argued that the web evolves in distinct states, each defined by the arrival of a new consumer class that existing infrastructure cannot serve well.

The first state was the human web: browsers rendering HTML as pixels for people to read. Every feature of early web development, from HTML to CSS to JavaScript, was designed around this consumer.

The second state was the index web: search engines crawling pages to build rankings. This consumer could not render pixels. It needed structured signals about content. The web responded with sitemaps, robots.txt, and structured data (Schema.org, JSON-LD, meta tags). Publishers who adopted these formats were rewarded with better search visibility.

The third state was the application web: software consuming data through APIs. This consumer did not want HTML at all. The web responded with REST APIs, GraphQL, and webhooks. Data that had been locked in rendered pages became available as structured endpoints.

The fourth state is the agent web. AI agents are a new consumer class with distinct needs: structured content (not pixels), token efficiency (not visual fidelity), semantic understanding (not DOM selectors), and interactive affordances (what can I click, type, or select?).

Building for the fourth state

This is where my work comes in. After two decades of building infrastructure for web consumers, I recognized the pattern: agents need their own infrastructure layer, just as search engines and applications did before them.

That recognition led me to build Plasmate. I described the technical architecture in Why I'm Building a Browser No Human Will Ever Use. The short version: Plasmate is an open source headless browser that converts web pages into a structured format called SOM. It skips everything agents do not need (pixel rendering, visual layout) and preserves everything they do (content, structure, interactive elements, semantic regions).

But Plasmate is not just a tool. It is part of an ecosystem designed to serve the agentic web. If you are evaluating how to engage with this shift, here is how the pieces fit together.

The ecosystem: what exists and where to find it

Lessons from building Mautic

I founded Mautic in 2014. It became the world's first open source marketing automation platform before Acquia acquired it in 2019. Building Plasmate, I keep seeing the same patterns repeat.

I wrote about this in What I Learned Building Mautic That Applies to the Agentic Web. The core lesson: when a new consumer class emerges that existing tools cannot serve well, the opportunity is building purpose-built infrastructure for that consumer.

Open source matters in both cases. Mautic grew to millions of installations because organizations could try it without a sales call, deploy it without procurement, and extend it without permission. Plasmate follows the same distribution model: open source core, commercial services on top for those who need operational convenience.

Standards matter too. Mautic contributed patterns that other marketing tools adopted. With Plasmate, I am investing in specifications (SOM, AWP, robots.txt extensions) that can become web standards, not just Plasmate features. I participate in the W3C Web Content for Browser and AI Community Group because standards bodies matter even when they move slowly.

What to do from here

If you have read this far, you are probably wondering what concrete steps you can take. That depends on who you are.

The transition is happening

Every transition between web states has been driven by economics, not ideology. Publishers adopted sitemaps because search visibility drove traffic. They built APIs because application integrations drove revenue. They will adopt agent-oriented infrastructure for the same reason: agents are becoming a primary channel for content discovery.

I am not saying this to sell you something. Plasmate is open source. The specifications are open. The research is public. My interest is in seeing the agentic web develop well, with infrastructure that serves both agents and publishers, rather than a chaotic land grab that benefits neither.

If that interests you, the resources are available:

I know which I prefer.

David Hurley is the founder of Plasmate Labs and the creator of Mautic, the world's first open source marketing automation platform. He writes about web infrastructure, AI agents, and the agentic web at dbhurley.com and publishes research at dbhurley.com/papers.