How AI Reads and Recommends Your Website

Somewhere this week, a potential customer asked ChatGPT to recommend a company like yours. The answer named three businesses and gave a one-line reason for each. There was no second page of results, no ad slot next to the answer, no chance to be the fourth suggestion. Whoever was left out lost a customer in a place they cannot see: a conversation.

We build websites for a living, which means we spend more time than most people reading server logs. Those logs have changed character. Alongside Googlebot, names like GPTBot, ClaudeBot and PerplexityBot now show up every day, and by June 2026 Cloudflare was measuring automated crawlers at just over half of all web requests. Every website now publishes to a second audience: machines that read on behalf of people, and increasingly decide what those people get told.

This article is about that audience. Which bots visit and why, what they can and cannot see, how an AI answer is actually put together, and whether the models run something like Google's E-E-A-T when they decide whom to trust. At the end you will find the fifteen-minute audit we apply to our own work, and a short list of things you can safely ignore.

Who is reading your website

Three kinds of AI visitors call on your site, and they want different things.

Training crawlers collect text to teach future models. OpenAI's GPTBot, Anthropic's ClaudeBot and Common Crawl's CCBot are the best known. What they gather shapes what next year's models know about the world, including your company. Blocking them is a legitimate business decision, and it removes exactly nothing from today's AI answers, because answers are not assembled from training data alone.

Search-index crawlers build the retrieval indexes behind live AI answers. OAI-SearchBot feeds ChatGPT search, Claude-SearchBot feeds Claude, PerplexityBot feeds Perplexity. When an assistant cites sources, the citations come through this pipeline. Block these and you disappear from the answers themselves.

User-triggered fetchers act on a person's request. Paste a link into ChatGPT and ChatGPT-User goes to read it. Ask Claude to compare two pricing pages and Claude-User fetches both. The browsing agents that click buttons and fill forms belong in this family too. These are the closest thing to a human visitor that is not one.

Each operator documents its bots publicly, and the practical picture looks like this:

User agent	Run by	Job	If you block it
GPTBot	OpenAI	Training data	Future models know less about you. No effect on today's answers
OAI-SearchBot	OpenAI	Index behind ChatGPT search	You drop out of ChatGPT's cited answers
ChatGPT-User	OpenAI	Fetches pages a user asks about	ChatGPT cannot open your pages on request
ClaudeBot	Anthropic	Training data	Same trade as GPTBot
Claude-SearchBot	Anthropic	Index behind Claude's web search	You fade from Claude's search results
Claude-User	Anthropic	Fetches pages a user asks about	Claude cannot read your pages when asked
PerplexityBot	Perplexity	Index behind Perplexity answers	You disappear from Perplexity citations
Google-Extended	Google	Opt-out switch for Gemini training	No effect on Search, AI Overviews or AI Mode
Googlebot	Google	Classic search plus AI Overviews and AI Mode	You leave Google entirely
CCBot	Common Crawl	Open dataset many models train on	Less of you in future public corpora

The consequence hiding in that table: robots.txt is no longer one decision. It is at least three. A lot of sites wrote a blanket "block everything with AI in the name" rule during the 2023 panic and are quietly absent from AI answers in 2026 without anyone having decided that on purpose. The reverse mistake exists too: publishers who wanted to keep their text out of training datasets but never touched the search-index bots, or vice versa. The first thing to know about your own robots.txt is which of the three doors it actually closes.

Google deserves its own paragraph, because its naming actively misleads. Google-Extended sounds like the master switch for everything AI at Google. It only controls whether your content trains Gemini models. AI Overviews and AI Mode are classified as search features and fed by plain Googlebot, so switching Google-Extended off changes nothing there. For years the only real lever was nosnippet, which also strips your snippets from classic results. In June 2026 Google began testing a Search Console setting that removes a site from AI Overviews and AI Mode without touching rankings; at the time of writing it is only open to a small group of UK publishers.

One more thing worth knowing: the whole arrangement runs on trust, and the trust is under renegotiation. In 2025 Cloudflare publicly accused Perplexity of fetching blocked pages through undeclared crawlers; Perplexity's defense was that a fetch made on a user's behalf is not crawling. In 2026 Cloudflare announced it will block mixed-use crawlers by default on ad-supported pages from September 15, and is extending its pay-per-crawl system into a model where publishers get paid when their content appears in an AI answer. The plumbing under all of this is being rebuilt while you read.

What the machine actually sees

Here is a test you can run right now. Open your homepage, press Ctrl+U to view the raw source, and search for the first sentence of your own text. If you cannot find it, most AI systems cannot either.

The reason is unglamorous: nearly all AI crawlers and fetchers read the initial HTML response and stop there. They do not run JavaScript. Measurement studies through 2026 found GPTBot requesting script files in roughly one request out of nine and executing none of them, and ClaudeBot downloading scripts in about a quarter of requests without ever running one. Gemini is the notable exception, because it rides on Googlebot's rendering infrastructure.

This produces the strangest visibility split on the modern web. A site built as a client-side application, where the server sends an empty shell and JavaScript paints the content afterwards, can rank fine in classic Google and be a blank page to ChatGPT, Claude and Perplexity at the same time. The owner sees a working site in the browser and healthy Google traffic in the dashboard, and never learns that a growing class of readers gets nothing.

The fix has been standard engineering advice for a decade: render your content on the server, keep real text in real HTML, use headings that describe what follows them, and never store information that matters only inside images. What changed is the price of ignoring it.

How an AI answer is assembled

When someone asks an assistant for a recommendation, a small pipeline runs. Understanding it matters, because you can fall out at each stage for a different reason, with a different fix.

Stage one: the question becomes several searches. The assistant rewrites "who should build our clinic's website" into a handful of queries. Google openly calls this query fan-out in its AI Mode documentation; the other platforms do the equivalent.

Stage two: an index supplies candidates. Those queries hit a search index, the same kind of infrastructure classic SEO has always fed. If your pages are not indexed, or you blocked the search-index bot, the pipeline ends here and nothing downstream can save you.

Stage three: the assistant reads the pages. Candidate pages get fetched and parsed. This is where the rendering gap bites: a page whose content lives behind JavaScript arrives as an empty shell and is silently dropped.

Stage four: passages get selected and cited. The model skims what survived, pulls the passages that answer the question, and writes a reply naming a handful of sources. Pages that state facts plainly, with numbers and dates, get quoted. Pages of adjective soup do not, even when the company behind them is genuinely good.

Notice what this means for the loud claims that "AI optimization" replaces SEO. The entry ticket is still ordinary search hygiene, because stage two is a search engine. What is new are the two gates after it, and most sites have never been checked against either.

Machine trust, or the E-E-A-T nobody wrote down

Google's E-E-A-T is a written rubric: experience, expertise, authoritativeness, trust, spelled out in guidelines that human quality raters apply. So the natural question is whether AI models run their own version before they recommend a business.

The short answer is no. There is no E-E-A-T module inside GPT-5 or Claude, no checklist the model consults before citing you.

The longer answer is more interesting, because models end up behaving as if such an analysis ran. Trust enters the pipeline at three separate layers, and the three together do the job the rubric does at Google.

Inherited trust. The search index that supplies candidate pages has spent twenty years scoring link authority and filtering spam. Every AI answer built on retrieval inherits those judgments wholesale. Your domain's standing in ordinary search quietly becomes your standing in AI answers. This layer moves over months and years.

Remembered trust. Models carry priors about brands from training data. If your company has been written about, reviewed, linked and discussed across the web, the model arrives already knowing roughly what you are and how you are regarded. A brand with no footprint starts every conversation cold and depends entirely on what retrieval happens to surface. This layer moves at the pace of model releases, which is why consistent public presence pays off long before anyone can measure it.

Demonstrated trust. At answer time, the model judges the page in front of it. Does it state checkable claims or vague ones? Are there dates, a named author, sources? Does it agree with the other retrieved pages, and when sources conflict, which one reads like the primary? These judgments are learned statistical habits rather than rules, but they run every single time. This layer you can change today, this afternoon, on any page you own.

Two honest caveats. First, all three layers judge by proxy, and proxies can be gamed: confident, well-structured prose reads as authoritative to a model, which is exactly why so much AI-facing spam is confident and well-structured. The platforms know this and keep adjusting, so pages built on checkable facts age better than pages that perform authority. Second, the gaming has an embarrassing failure mode. Hiding white-on-white text that instructs the model to recommend you is not optimization; it is a screenshot waiting to happen, and retrieval systems increasingly detect and penalize it.

The experience problem

Of Google's four letters, the first E is the one machines have the hardest time with. A model reading your website cannot visit your workshop, taste the food, or watch you handle a difficult project. The only experience it can detect is the kind that leaves marks in text.

What leaves marks: numbers you measured yourself, decisions you can date, named projects, and problems described at a level of detail that secondhand writing never reaches. "We build fast, reliable, user-friendly websites" carries zero evidence weight; every competitor's page says the same words. "We rebuilt a dental clinic's booking flow in March and appointment requests doubled within two months" is checkable in principle, specific in fact, and exactly the kind of sentence that survives being lifted into an answer.

This cuts against the cheap version of E-E-A-T advice. Adding an author box and sprinkling credentials is decoration a model can read past. Publishing what you actually know, with the numbers attached, is the expensive version, and it is the one that compounds across all three trust layers at once.

What we changed in how we build websites

None of what follows is exotic, and that is the point. The sites AI systems recommend are mostly just well-built sites. These are the defaults we now apply to every project that leaves our studio:

1. Content arrives in the first HTML response. Server-rendered or statically generated, so the view-source test passes on every page that matters.
2. robots.txt is written with intent. We decide per bot family. Search-index and user-triggered bots stay open unless the client has a specific reason; training bots are the client's call, made knowingly rather than by accident.
3. Pages answer first and elaborate second. The first paragraph under each heading answers that heading in two or three sentences that would survive being quoted alone. Detail and nuance follow for the human who keeps reading.
4. The company is machine-legible. The same name, description and address everywhere; Organization and LocalBusiness schema with sameAs links; an about page that states in plain words what the company does, for whom, where, and since when.
5. Every service page carries something only we could have written. A real number, a dated project, a decision and its reason. If a paragraph could appear unchanged on a competitor's site, it is not finished.
6. Dates mean something. Publication and update stamps reflect actual changes, because "fresh" signals that turn out to be cosmetic are the kind of trick the demonstrated-trust layer learns to discount.

Then we measure. AI referrals appear in analytics under referrers like chatgpt.com, perplexity.ai and gemini.google.com, and the numbers are small but strange in a good way. Benchmark studies in 2026 put visitors arriving from ChatGPT converting at around sixteen percent against under two percent for Google organic; other studies land lower but point the same direction. The caveat matters: ChatGPT still sends a fraction of Google's volume, and click-through from AI answers runs below one percent. Treat it as a high-intent channel that is growing, not as a replacement for search traffic.

The fifteen-minute audit

You do not need tools or budget for a first look. You need a browser and a stopwatch.

1. View source. Ctrl+U on your homepage, search for your own opening sentence. Present in the raw HTML, or not? (2 minutes)
2. Read your robots.txt. Open yoursite.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot. Work out which of the three doors you have closed, and whether anyone chose that. (3 minutes)
3. Ask the assistants. Put "what does [your company] do" and "recommend a [your category] in [your city]" to two different AI assistants. Note what they say, what they cite, and who appears instead of you. (5 minutes)
4. Check referrers. In your analytics, filter for chatgpt, perplexity, gemini and copilot. Even a handful of visits tells you the channel is live for your market. (3 minutes)
5. Read your first paragraph as a stranger. Does the homepage state what you do, for whom and where, in words a machine could quote? Or does it open with a slogan? (2 minutes)

Fifteen minutes, and you will know more about your AI visibility than most of your competitors know about theirs.

What you can safely ignore

llms.txt, mostly. The proposed standard (a markdown map of your site for language models) grew nearly ninefold in adoption over the past year, yet 97 percent of the files never received a single request from an AI crawler, and no major provider has committed to reading it. Google states plainly that Search does not use it. It costs five minutes and does no harm, so add one if you like. Just do it after the rendering and robots work, not instead of it.

Miracle "AEO" tooling. A gold rush of dashboards now promises to optimize you into AI answers. The monitoring ones have genuine uses. The optimization ones are mostly selling the contents of this article back to you with a subscription attached.

Hidden instructions to the model. Covered above, worth repeating: it gets detected, it gets screenshotted, and it torches the demonstrated-trust layer you were trying to fake.

Panic-blocking everything. Blocking every AI bot feels decisive and costs you the fastest-growing high-intent channel there is. Close doors deliberately or not at all.

The pattern across all four: the fundamentals moved, the fads did not. A site whose content is real HTML, whose claims are checkable, whose identity is consistent and whose robots policy is deliberate serves both of its audiences with the same work.

Your next customer may still find you the old way, typing keywords and scanning ten blue links. The one after that may never see a results page at all. We build for both readers now, on every site we ship, and the audit above is where every one of those projects starts. If you run it on your own site and something fails, that result is worth knowing today rather than next year.

Questions clients ask us

Does blocking GPTBot remove my site from ChatGPT's answers? No. GPTBot only collects training data. Live ChatGPT answers draw on OAI-SearchBot's index and ChatGPT-User fetches, so those two are the ones that decide whether you appear.

Do AI crawlers run JavaScript? Almost none of them. Studies through 2026 show GPTBot, ClaudeBot and PerplexityBot fetching scripts but never executing them; Gemini is the exception because it shares Google's rendering pipeline. Content that only exists after JavaScript runs is invisible to the rest.

Do I need an llms.txt file? No provider requires one, and the overwhelming majority of published files are never requested. Add one if you want; fix server-side rendering and robots.txt first, because those two demonstrably change what AI systems see.

How do I know if AI assistants send me customers? Filter your analytics referrers for chatgpt.com, perplexity.ai, gemini.google.com and copilot.microsoft.com. Expect low volume and unusually high intent; 2026 benchmarks consistently show AI-referred visitors converting at several times the rate of organic search.