Make your WordPress site ready for AI agents
AI assistants now read business websites, fill in contact forms and book appointments for the people who ask them. WordPress gives you more control over how that goes than any hosted site builder, but plugins you installed years ago can quietly get in the way. Here’s a checklist, in the order we’d work through it.
Updated
1. Let the right AI crawlers in
Start at yoursite.com/robots.txt. This file asks automated visitors to keep out of parts of your site, and AI companies publish the names (user agents) their crawlers and assistants use, so you can treat each one differently.
It helps to sort them into three groups. Assistants fetch a page because a person just asked them something (ChatGPT-User, Claude-User, Perplexity-User). AI search crawlers index your site so it can turn up in AI answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Training crawlers collect text to train future models (GPTBot, ClaudeBot, CCBot). Many businesses happily welcome the first two and opt out of the third. That’s a reasonable choice, and it costs you nothing in AI answers today.
WordPress gives you three ways to change robots.txt:
- Leave it to WordPress. With no robots.txt file on the server, WordPress makes a virtual one that blocks
/wp-admin/and nothing else. Separately, check that Settings → Reading → “Discourage search engines from indexing this site” is clear on a live site; it’s meant for sites still being built. - Yoast SEO: Yoast SEO → Tools → File editor, then “Create robots.txt file”. That creates a real file that replaces WordPress’s virtual one. The menu doesn’t appear if your site has file editing turned off; then you’d edit the file over FTP or your host’s file manager.
- Rank Math: Rank Math SEO → General Settings → Edit robots.txt. The option may not appear if a real robots.txt file already exists or another plugin or your theme makes one.
Here’s a starting point. Swap in your own sitemap address (Yoast’s is /sitemap_index.xml; WordPress’s own is /wp-sitemap.xml):
# AI assistants and AI search: welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /wp-admin/
# Optional: opt out of AI training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
# Everyone else: WordPress's usual rules
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
A few cautions. A bot that matches a named group follows only that group, so repeat any rules you still want it to obey. robots.txt is a request, not a lock. And caching plugins can keep serving an old copy, so clear the cache after you change it. Our robots.txt guide covers the full list of AI user agents.
2. Check security plugins and firewalls
robots.txt is only half the story. Security plugins, your host’s firewall and services like Cloudflare can block or challenge requests before WordPress sees them, and they never tell the assistant why. The visitor just hears “I couldn’t load that site.”
- Look at what’s being blocked. Most security plugins and hosts keep a log of blocked requests. Search it for the AI user agents above.
- Check bot and rate-limit rules. Rules that block “unknown bots”, empty referrers or fast repeated requests catch AI agents as well as scrapers.
- Keep the WordPress REST API reachable. Some hardening settings switch it off for anyone not logged in. Plugins that talk to outside services, Salesman Hal included, need it.
- On Cloudflare, look at its AI crawler and bot settings, which can block AI bots at the edge whatever your robots.txt says. See our Cloudflare guide.
The quick test: ask ChatGPT, Claude or Perplexity to read a specific page on your site and tell you what it says. If it can’t, something is in the way.
3. Caching: fast pages, hidden visits
Page caching (from a plugin or built into your host) saves a finished copy of each page and serves it without running WordPress. That’s why it’s fast. It also means anything that runs inside WordPress on each request doesn’t run for cached pages, and that includes plugins that log visits on the server. On a busy, well-cached site, most AI agent visits can go unseen.
- Don’t turn caching off. Speed matters to agents too. Count visits another way instead: a tracking image (pixel) that loads from outside the cache catches agents that fetch images, and server or CDN logs catch the rest. Our guide to measuring AI agent traffic compares the options.
- Clear the cache after changes to robots.txt, structured data or head tags, or agents keep seeing the old version.
- Watch out for files at the site root. If
/llms.txtor a file under/.well-known/returns your theme’s 404 page or an old copy, exclude it from the cache.
4. Clean up your forms
Whether an agent fills in your form in a browser or sends the details some other way, it works from what the form says about itself. Most of what helps agents also helps people on phones.
- Give every field a visible label that says what goes in it. A placeholder that vanishes when you type isn’t a label.
- Use the right field types: email fields for email, phone fields for phone numbers, drop-downs for fixed choices. Gravity Forms, Contact Form 7 and WPForms all have them.
- Name fields sensibly. In Contact Form 7 the name is in the form tag (
[email* your-email]), so preferyour-phonetotext-412. Gravity Forms and WPForms number fields internally, so their labels matter even more. - Mark required fields honestly. Ask only for what you need to reply. Every extra required field is another question the assistant has to go back and ask.
- Write error messages that say what’s wrong (“Enter a phone number with area code”), not just “There was a problem with your submission.”
- Leave consent boxes for people. Marketing and SMS consent should be ticked by the visitor, never by software acting for them.
- One form, one job. A form called “Request a quote” with a clear success message beats a catch-all “Contact us” with ten optional fields.
5. CAPTCHAs and agents
reCAPTCHA, hCaptcha and Cloudflare Turnstile exist to tell people from bots, and an AI agent working in a browser is a bot. Invisible and score-based CAPTCHAs may let it through or may not; puzzles usually stop it. That’s the trade-off: the same check that stops spam stops a genuine customer’s assistant.
Don’t rip your CAPTCHA out. Spam is real. Instead, give agents a route that doesn’t go through the browser: an MCP server that hands inquiries to your form with its own signature standing in for the CAPTCHA (this is what the Salesman Hal plugin does), plus a phone number and email address in plain text on the page. Keep the honeypot fields and server-side spam filtering your form plugin offers; they don’t bother agents.
6. Booking embeds
Booking widgets from Calendly, Cal.com, Acuity and others usually load in an iframe or with JavaScript. Some browser agents can work through them, slowly; agents that only read your HTML can’t see the calendar at all.
- Put a plain link next to the embed (“Book a 30-minute call”) pointing at the scheduler’s own page. Agents that don’t run scripts can follow it.
- Say what the appointment is in text on the page: length, price, what happens, who it’s with.
- Keep event types tidy. Fewer, clearly named appointment types are easier for an agent to choose from.
- Offer booking through an API too. The most reliable path is a tool that checks real availability and books directly. Hal does this for Calendly, Cal.com, SavvyCal, Acuity and HubSpot Meetings, and every booking it makes is tagged so you know an AI assistant sent it.
7. Structured data with Yoast or Rank Math
Structured data (schema.org markup in JSON-LD) states facts about your business in a form machines read without guessing: your name, address, phone, opening hours, services and answers to common questions. Both big SEO plugins output it.
- LocalBusiness. In Yoast, LocalBusiness markup comes from its separate Local SEO plugin, which turns your primary location into a
LocalBusiness(or a more specific type, such asDentist) with address, phone, opening hours and price range. In Rank Math, fill in the Local SEO module; its docs list Local SEO schema in the free version. - Service. Rank Math’s free Schema Generator includes a Service type, so a page per service can describe exactly what you offer.
- FAQ. Yoast’s FAQ block is in the free plugin; in Rank Math’s free version the FAQ block is the only way to add FAQ schema. Both note that Google stopped showing FAQ rich results, but the markup still helps search engines and AI tools understand the questions and answers.
Whatever you mark up must match what’s visible on the page, and the facts must agree everywhere: website, Google Business Profile, directories. An assistant that finds two different phone numbers trusts neither.
8. llms.txt
llms.txt is a proposed plain-text file at /llms.txt that gives AI tools a short summary of your site and links to its most useful pages. No major assistant has said it relies on it, so treat it as cheap and optional. Our llms.txt guide goes into the evidence.
- Yoast SEO: Yoast SEO → Settings → Site features, then switch on llms.txt under AI tools. By default Yoast picks the pages; “Customize llms.txt file” lets you choose them by hand. Not available on multisite.
- Rank Math: switch on the LLMS Txt module in Rank Math SEO → Dashboard, then choose post types, taxonomies and a page limit under General Settings → Edit llms.txt. Posts set to noindex are left out.
Use one plugin for it, not both, and open the file after saving to check it isn’t a cached 404.
9. Discovery files
If your site has an MCP server (a service AI assistants can connect to and call tools on, like “request a quote”), agents need a way to find it. The emerging convention is a small JSON file at /.well-known/mcp describing the server, a fuller “server card” at /.well-known/mcp/server-card.json, and a <link rel="mcp"> tag in each page’s head for agents that read HTML. Our MCP discovery guide explains the formats.
On WordPress there’s a wrinkle: some managed hosts answer requests under /.well-known/ themselves, so WordPress never sees them. A plugin that serves these files has to cope with that, for example by also writing real files to disk where it can. After setting them up, open each address in a private browser window and check you get JSON, not a 404.
10. WebMCP attributes on forms
WebMCP is a proposed browser feature that lets a page offer tools to an AI agent working in the visitor’s browser. Its declarative form turns an existing form into a tool with two attributes on the <form> tag, toolname and tooldescription. It runs in Chrome and Edge origin trials, so only a few visitors’ browsers use it today; our WebMCP guide has the details.
Form plugins don’t have a setting for these attributes, but Gravity Forms and WPForms both have documented filters for the form tag. In a small custom plugin or your child theme’s functions.php, for WPForms:
add_filter( 'wpforms_frontend_form_atts', function ( $atts, $form_data ) {
if ( absint( $form_data['id'] ) !== 11 ) {
return $atts; // Only the quote form.
}
$atts['atts']['toolname'] = 'request_quote';
$atts['atts']['tooldescription'] = 'Ask for a quote on a plumbing job. A person replies by phone or email within one business day.';
return $atts;
}, 10, 2 );
And for Gravity Forms:
// gform_form_tag_3 runs for form 3 only.
add_filter( 'gform_form_tag_3', function ( $form_tag, $form ) {
return str_replace(
'<form ',
'<form toolname="request_quote" tooldescription="Ask for a quote on a plumbing job. A person replies within one business day." ',
$form_tag
);
}, 10, 2 );
Contact Form 7 has no documented equivalent, so ask your developer. One caution: if you install Salesman Hal, its script already registers inquiry, quote and booking tools with WebMCP on every page. Adding attributes to the same form gives the agent two tools for one job, so pick one.
11. Installing the Salesman Hal plugin
The free Salesman Hal plugin covers the discovery, form and visit-logging parts of this checklist. It needs WordPress 6.2 or later and PHP 7.4 or later.
- Install it. Plugins → Add New, search for “Salesman Hal”, then Install and Activate.
- Fill in your business details. Settings → Salesman Hal: name, description, services and service area, phone, email and address. The business email matters: if your site can’t take a request, it’s emailed there. Choose which post types AI assistants should see, then click “Save Changes”.
- Connect. Click “Connect to Salesman Hal”. The service checks you own the domain by fetching a code from
/.well-known/salesman-hal-verification.txt, or from the plugin’s REST route if your host keeps/.well-known/to itself. If that fails, make sure the site is public and nothing blocks the REST API, then click “Verify again”. - Open the dashboard from the plugin’s settings to see AI activity and connect your calendar.
- Check it. Clear your page cache, then open
/.well-known/mcpin a private window and look for<link rel="mcp"in a page’s source.
Once connected, the plugin:
- Gives your business a hosted MCP server and publishes
/.well-known/mcp,/.well-known/mcp.json, the server card and the<link rel="mcp">tag, writing them as real files too where your host answers/.well-known/itself. It doesn’t change your theme or pages. - Syncs your content: published posts of the types you chose (never password-protected pages or pages set to noindex in Yoast or Rank Math), the fields of your lead forms (never past submissions), and booking links it finds on your pages.
- Sends inquiries and quote requests into your forms. Gravity Forms, Contact Form 7 and WPForms are supported, and requests arrive as normal entries, so your notifications and CRM hooks fire. Requests are signed, so they skip the CAPTCHA; consent boxes are never ticked; a missing required answer goes back to the assistant to ask the visitor.
- Logs AI agent visits by user agent (ordinary visitors are never logged) and shows them in wp-admin, with a tracking pixel (on by default) that catches agents on cached pages if they load images.
- Loads a small WebMCP script, deferred, which does nothing unless the browser supports WebMCP.
It doesn’t edit robots.txt, write llms.txt or add schema markup; keep using your SEO plugin for those. An optional footer line, “MCP for AI assistants, powered by Salesman Hal”, is off unless you turn it on.
How Salesman Hal handles WordPress
One plugin from WordPress.org: it serves the discovery files, keeps your hosted MCP server up to date with your pages and forms, replays agents’ inquiries into Gravity Forms, Contact Form 7 or WPForms so they land like any other entry, and shows which AI agents visited, and why, right in wp-admin. Bookings go straight to Calendly, Cal.com, SavvyCal, Acuity or HubSpot Meetings once you connect one in the dashboard.
FAQ
Should I block GPTBot and other training crawlers?
It’s your call, and it’s separate from being found. Blocking training crawlers doesn’t stop assistants fetching a page for a user or AI search indexing your site, as long as you leave those user agents allowed.
Will my CAPTCHA stop AI agents?
It can stop agents that fill in the form in a browser, which is what it’s for. Keep it for spam, and give agents another route: a phone number and email in plain text, and an MCP server that sends inquiries into your form without going through the browser.
Do I need llms.txt?
No. It’s optional and cheap if your SEO plugin makes it for you. Clear pages, accurate structured data and crawler access matter more.
My booking form is a WordPress plugin, not Calendly. Will Hal book it?
Not today. Hal books through Calendly, Cal.com, SavvyCal, Acuity and HubSpot Meetings. If your bookings live in a WordPress booking plugin, agents can still send an inquiry through your contact form.
Does the Salesman Hal plugin slow my site down?
Barely. Checking a user agent is a string match, visits are reported after the page has been delivered (or hourly as a fallback), and the WebMCP script is a few kilobytes and deferred.
What happens if I deactivate the plugin?
Syncing stops and the discovery files go away. Deleting it also removes its settings and visit log. What was already sent to the service stays there until you ask for it to be deleted.
Sources
- The robots.txt file in Yoast SEO, Yoast
- How to enable llms.txt with Yoast SEO, Yoast
- FAQ block, Yoast
- Yoast Local SEO for WordPress: Schema output, Yoast developer portal
- Rank Math General Settings, Rank Math
- How to use llms.txt in Rank Math SEO, Rank Math
- Configuring Schema Markup in Rank Math, Rank Math
- gform_form_tag, Gravity Forms documentation
- wpforms_frontend_form_atts, WPForms developer documentation
Further reading
- Salesman Hal on the WordPress plugin directory, WordPress.org
- WebMCP: a practical guide for business websites, Salesman Hal
- robots.txt and AI crawlers, Salesman Hal
- llms.txt, Salesman Hal
- MCP discovery, Salesman Hal
- Cloudflare and AI agents, Salesman Hal
- Measuring AI agent traffic, Salesman Hal
Let AI assistants book your WordPress business.
The free plugin adds an MCP server, discovery files and AI visit stats in about ten minutes. Free during early access.
Or run the free AI-readiness scan first.