How to Optimize Your Website for AI Agents: A Complete Setup Guide

Share Now

AI agents are no longer just a future possibility. They are actively browsing websites right now, on behalf of users, carrying out tasks like researching products, filling out forms, comparing services, and gathering information. Google has responded by adding a new Agentic Browsing audit inside Google Chrome’s Lighthouse tool to help website owners measure exactly how well their sites perform for these non-human visitors.

This guide walks you through the complete setup and execution process, what the results mean, and how to fix every issue the audit surfaces.

Table of Contents

What is the Agentic Browsing audit?
Step-by-step setup and execution
Understanding the results
Fix 1: Accessibility tree
Fix 2: llms.txt
Fix 3: WebMCP (transactional sites only)
What to do after the audit
FAQs

What is the Agentic Browsing Audit?

Google’s Lighthouse has long been the go-to tool for measuring page performance, accessibility, and SEO. The Agentic Browsing audit is an experimental addition that adds a completely different lens: how well can AI agents actually read, understand, and interact with your site?

Unlike traditional Lighthouse audits that produce a score between 0 and 100, the Agentic Browsing audit produces a Pass/Fail checklist. There is no score to optimise for. Each check either passes or it does not, and a failure means an AI agent visiting that page will struggle to do its job.

The audit evaluates four things: the health of your accessibility tree, the presence of an llms.txt file, layout stability during page load, and optionally your WebMCP implementation. The first three apply to every website. The fourth is only relevant for sites with transactional flows.

Step-by-Step Setup and Execution

Step 1: Update your browser

You need Google Chrome version 146 or later to access the Agentic Browsing audit. If your stable release of Chrome does not yet show the option, switch to the Beta or Canary channel temporarily to access the experimental feature.

You can check your current Chrome version by going to chrome://settings/help.

Step 2: Enable the Agentic Testing flag

Open a new tab and type the following into the address bar:

chrome://flags/#enable-webmcp-testing

Find the flag in the list, switch its toggle from Default to Enabled, and click the Relaunch button that appears at the bottom of the screen. Chrome will restart with the flag active.

Step 3: Open Developer Tools on your target page

Navigate to the page you want to audit. Right-click anywhere on the page and select Inspect, or press F12 on Windows or Cmd+Option+I on Mac. Developer Tools will open in a panel alongside or below the page.

Step 4: Run the audit

Inside DevTools, find the Lighthouse tab. If you cannot see it immediately, click the >> arrows at the top of the DevTools panel to reveal hidden tabs.

Once inside Lighthouse, do the following:

Select either Mobile or Desktop depending on your primary audience. Check that Agentic Browsing is ticked in the categories list. Click the Analyse page load.

The audit will take roughly 30 to 60 seconds to complete. When it finishes, scroll down to the Agentic Browsing section of the report.

Understanding the Results

Unlike the performance, accessibility, or SEO audits you may be familiar with, there is no numeric score here. The Agentic Browsing report gives you a simple checklist where each item is either passing or failing.

The three checks are:

Accessibility tree confirms whether interactive elements on your page have proper names, labels, and valid semantic roles. This is what agents use to understand your page structure.

llms.txt confirms whether you have placed a concise plain text summary file at your root domain that agents can read to understand your site.

WebMCP API verifies whether your site logic and forms are explicitly exposed so AI agents can execute actions programmatically, without needing to simulate human clicks through your UI.

A failing check does not crash your site or hurt your Google search ranking today. But it does mean AI agents visiting your site are either guessing, failing silently, or giving up. As agent-driven traffic grows, these failures translate directly into missed opportunities.

Fix 1: Repair the Accessibility Tree

This is the most foundational fix. If your accessibility tree is broken, agents cannot reliably interpret any page, regardless of how well everything else is set up.

What the accessibility tree is

When a browser loads your page, it builds two separate trees from your HTML. The DOM tree contains everything. The accessibility tree is a stripped-down version containing only semantically meaningful elements: headings, labels, buttons, links, form fields, and their hierarchical relationships. Screen readers use this tree. AI agents use this tree.

How to inspect your current accessibility tree

Open Chrome DevTools, click on any element in the Elements panel, and then open the Accessibility tab in the right-hand sidebar. You will see the computed accessibility role, name, and properties for that element. For a full-page view of issues, run the standard Accessibility audit in Lighthouse alongside the Agentic Browsing audit.

The most common failures and how to fix them

Unlabelled interactive elements. Every button, input, link, and form control needs a label that describes what it does. A button containing only an icon with no text is invisible to an agent. Fix it with aria-label, aria-labelledby, or by adding visible text inside the element.

Generic containers instead of semantic HTML. Agents understand <nav>, <main>, <article>, <section>, and heading tags <h1> through <h6>. They do not understand <div class=”nav-wrapper”> regardless of how it looks visually. Audit your templates and replace generic containers with the correct semantic elements.

Broken heading hierarchy. Headings should descend in order: one <h1> per page, then <h2> for subsections, then <h3> below those. Skipping levels or using heading tags purely for visual sizing breaks the document outline that agents rely on to navigate your content.

Images without alt text. Informational images should have a descriptive alt attribute. Decorative images should have alt=”” so agents correctly skip them rather than trying to interpret them.

Form fields without associated labels. Use <label for=”field-id”> paired with the field’s id, or use aria-labelledby. Placeholder text alone is not sufficient and disappears once a user begins typing.

A free browser extension like axe DevTools or WAVE will surface exactly which elements are failing and give you actionable fix guidance for each one.

Fix 2: Add an llms.txt File

What llms.txt is

llms.txt is a plain text file placed at the root of your domain, at yourdomain.com/llms.txt. It is a community-driven standard inspired by robots.txt, but with a completely different purpose. Rather than restricting crawlers, it gives AI agents a curated, human-authored summary of what your site is about and which pages matter most.

Without it, agents visiting your site must guess. With it, you are in control of what they read first.

What to include

A well-structured llms.txt file contains four things:

A brief description of your business or site written in plain, natural language. Keep it factual and direct, not promotional.

A list of your most important URLs, each with a one-line description of what that page contains. Focus on pages that explain your product or service, your pricing, your documentation, and your contact or conversion pages.

Optionally, a second file called llms-full.txt is linked from the main file, containing richer content like detailed product descriptions or plain-text FAQs for agents that want more depth.

A short note about what agents are and are not permitted to do on your site.

A template to get you started

# About this site

[Business name] provides [what you do] for [who you serve].

[One or two additional sentences of context.]

## Key pages

– /: Homepage, introduces the product and main value proposition

– /pricing: Pricing plans and feature comparison

– /docs: Full product documentation

– /about: Company background and team

– /contact: Contact form and support options

## Extended content

For more detailed information, see /llms-full.txt

## Agent instructions

AI agents may read and summarise content from this site.

Please do not submit forms or create accounts without explicit user confirmation.

Save this as a plain .txt file with no HTML and upload it to your web root. Verify it is publicly accessible by visiting the URL directly in a browser.

Fix 3: Implement WebMCP (Transactional Sites Only)

Who needs this right now?

WebMCP is a protocol that exposes your site’s actions directly to AI agents as a machine-readable manifest. Instead of an agent navigating your UI to search for a product or submit a form, WebMCP gives the agent a structured list of actions it can call directly.

If your site is primarily a content or lead generation site, skip this for now. The accessibility tree and llms.txt fixes will give you far more return for the effort.

If you run an e-commerce store, a SaaS platform, or any site where users complete multi-step transactional flows, WebMCP belongs on your roadmap.

How it works

WebMCP works by exposing a manifest file at /.well-known/mcp.json. This file describes the available actions on your site, each with a name, a plain English description, and a schema defining its parameters and expected responses. An agent visiting your site discovers this manifest and can invoke actions directly rather than simulating user clicks.

A basic manifest example

{

  “actions”: [

    {

      “name”: “search_products”,

      “description”: “Search the product catalogue by keyword or category”,

      “parameters”: {

        “query”: { “type”: “string”, “required”: true },

        “category”: { “type”: “string”, “required”: false }

      }

    },

    {

      “name”: “add_to_cart”,

      “description”: “Add a product to the user’s shopping cart”,

      “parameters”: {

        “product_id”: { “type”: “string”, “required”: true },

        “quantity”: { “type”: “integer”, “required”: true }

      }

    }

  ]

}

Start with your two or three highest-value actions. Product search and product detail retrieval are strong starting points for e-commerce. Quote request submission works well for service businesses. Expand the action library once those are tested and stable.

What to Do After the Audit

After your first run, you will have a mix of passes and failures across the four checks. Work through them in this order:

Fix the accessibility tree first. It is foundational. A broken tree means agents cannot reliably interpret any page, regardless of how good everything else is.

Add llms.txt next. This is a single file, with no code dependencies, and can be live within an hour. There is no good reason to delay it.

Address CLS issues third. Check your current Lighthouse Performance score for CLS, identify the top offenders using the report’s diagnostic detail, and fix them in order of impact.

Plan WebMCP only if you have transactional flows and have already handled the first three. Build a case based on the agent traffic you are already receiving before investing engineering time here.

Re-run the audit quarterly. Agentic browsing standards are evolving quickly, and what the audit checks today may expand significantly over the coming months.

FAQs

Does the Agentic Browsing audit affect my regular Google search ranking?

Not directly, not right now. The audit is in an experimental Lighthouse category and is not currently a confirmed ranking signal in standard Google Search. However, many of the underlying fixes, particularly the accessibility tree and CLS improvements, do influence your standard SEO and accessibility scores, which do matter for ranking. Fixing these issues helps both human visitors and AI agents simultaneously.

Will adding llms.txt affect how search crawlers index my site?

No. Traditional search crawlers like Googlebot do not read or act on llms.txt. It is specifically designed for AI agents and language model systems. There is no downside to adding it, and no risk of it interfering with your existing SEO setup.

Do I need a separate llms.txt file for each subdomain?

Yes. A file at yourdomain.com/llms.txt only covers the root domain. If you host content on docs.yourdomain.com or blog.yourdomain.com, each subdomain needs its own llms.txt pointing agents to the relevant content on that subdomain. You can cross-reference between them within the file.

How do I know if AI agents are already visiting my site?

Check your server logs or analytics platform for traffic from known AI agent user-agent strings. Agents from major AI providers typically identify themselves. Many website owners are surprised to discover this traffic is already substantial, particularly on sites that publish informational, research-oriented, or product-focused content.

My site is on WordPress. Is there a faster way to set all of this up?

Yes. For the accessibility tree, plugins like WP Accessibility or Yoast SEO (which surface some accessibility issues) give you a starting point. For llms.txt, you can create the file manually and upload it to your root directory via FTP or your hosting file manager, or use a WordPress plugin built specifically for llms.txt generation. For CLS, a caching plugin combined with lazy loading settings will handle the most common causes. WebMCP on WordPress currently requires a custom plugin or developer work.

Share Now

Leave a Comment

Your email address will not be published. Required fields are marked *

Hire a machine, don’t be one!

Need a custom AI-powered solution to any marketing problem? We help build bespoke AI-driven solutions to help marketers automate processes and be more productive.

Contact Us