Can AI engines read your site, and how do you check?

A crawler is a program an engine's company sends to read web pages under a fixed name it states in every request. Most companies run one crawler for search, one that visits a page when a user asks for it, and one that collects content for training.

The difference between the three is what matters to you. A search crawler reads your page so the engine can cite it in answers; a user crawler opens it when someone asks about it; a training crawler collects text for future models. The table lists them as the companies themselves document them.

Do not trust the name alone. Google says a user-agent string can be spoofed, which is why Google, OpenAI, Perplexity, Anthropic and Apple publish the IP addresses they crawl from. How to verify a visit is in the FAQ at the end.

This guide is part of the steps to lead AI answers: an engine reading your page is the first step, before any writing.

Allow the search and user crawlers if you want to be cited. Training crawlers are a business decision for you alone, because the companies that document the split say blocking them does not take you out of search.

OpenAI says each setting is independent of the others: you can allow OAI-SearchBot so your site appears in ChatGPT search, and block GPTBot so your content is not used for training.

Anthropic says blocking ClaudeBot excludes your future content from its training data, blocking Claude-SearchBot stops your content being indexed for its search, and blocking Claude-User stops it fetching your page when a user asks about it.

Google-Extended is a special case. It is not a crawler but a name you write in robots.txt. Google says it decides whether what Google collects from your site is used to train Gemini models and to ground their answers, and that it does not affect your inclusion or ranking in Google Search.

So control over AI Overviews and AI Mode runs through Search's own controls, Googlebot's rules in robots.txt, the snippet controls in section six and a Google Search Console setting that excludes the whole site from both, not through Google-Extended.

Our view, for a business that wants engines to recommend it to customers: allow every search and user crawler, and decide on training crawlers with your team. At Tasadr (تصدَّر) we allow them all on our own site, because we want what we write to be read and quoted.

Put a plain-text file named robots.txt at the root of your site, and write a group for each crawler you want a rule for. A crawler reads the group that carries its name and, if there is none, reads the general group.

In the illustrative example below, ruwaq-schools.sa, a group of private schools in Jeddah, allows every search crawler, blocks two training crawlers, and closes the parents' portal, which nobody needs in search.

Note that the example writes no group for OAI-SearchBot, PerplexityBot or Claude-SearchBot. They read the general group and may crawl everything but the portal, which is simpler than a group per crawler and leaves fewer copying mistakes.

Remember two things. The file is not a lock: the standard itself says its rules are not a form of access authorisation. And Google says it is not a way to keep a page out of Google: a page you want out of search needs a noindex tag or a password.

OpenAI and Perplexity say a change to the file can take about 24 hours to apply in their systems, so do not judge its effect the same day.

An llms.txt file, for its part, neither allows nor blocks any crawler: it summarises your site and holds no rules. What Google says about it is in what is llms.txt, and does your site need one?

Because some sites do not put their text in the HTML the server sends. They build it with JavaScript in the browser, and a crawler that does not run JavaScript reads a template with no text.

Google says Googlebot processes JavaScript pages in three phases, crawling, rendering and indexing, and that every page returning a 200 goes into the rendering queue. It also says server-side rendering or pre-rendering is still a great idea, because it is faster and because not all bots can run JavaScript.

The crawler pages of OpenAI, Perplexity and Anthropic, as we read them on 11 October 2026, do not say whether their crawlers run JavaScript. Apple says Applebot may render a page in a browser, and may not render it properly if JavaScript or CSS files are blocked in robots.txt.

So the safe path is to keep the important text in the HTML itself. You can check in a minute: open the page, choose View Page Source in your browser, and search for a sentence from your first paragraph. If it is not in the source, it is being built in the browser.

For Google, yes. Google says a page must be indexed and eligible to be shown in Search with a snippet to be a supporting link in AI Overviews and AI Mode, and its site must not be excluded from them in Google Search Console's settings.

That is what someone means when they search «أرشفة موقعك في جوجل» ("getting your site archived in Google"): archiving here means indexing, Google reading your page and storing it in its index to choose from when asked.

Google also says traffic from its AI features counts in Search Console's overall search traffic, under the Web search type in the Performance report. Search Console's URL Inspection tool tells you whether a page is indexed, and lets you request indexing if it is not.

Other engines have their own paths: ChatGPT, Perplexity and Claude read with their crawlers, and Copilot answers from Bing search, so submit your sitemap to Bing Webmaster Tools too, as in the site preparation checklist.

They stop or limit Google quoting your page. Google says nosnippet prevents the content being used as a direct input for AI Overviews and AI Mode, and max-snippet limits how much of it may be used.

Our view is plain: put none of them on the service pages and guides you want cited, and keep them for what you genuinely do not want quoted, such as private contract terms or internal pages.

• nosnippet: no snippet for the page in search results, and no use of its text as a direct input for AI Overviews and AI Mode
• max-snippet: a character limit on the snippet; 0 is the same as nosnippet, and -1 leaves the length to Google
• data-nosnippet: an attribute on part of the HTML to keep it out of the snippet; Google's page on AI features in Search lists it among the controls on what is shown from your page
• noindex: the page is not shown in search results at all, so it cannot enter AI Overviews or AI Mode, which need an indexed page

Open robots.txt, then the source of an important page, then Search Console, then ask whoever runs your hosting. Ten minutes later you know whether your pages reach the engines whole.

If you have a developer, ask them to request your page under the name of each crawler in the table and confirm the response is a 200 with the full text. That is what we did with our own site, in the next section.

• Minutes 1 to 2: open your domain followed by /robots.txt and look for any Disallow line under a search crawler's name, or under the general group, that covers your important pages
• Minutes 3 to 4: open your most important service page, choose View Page Source and search it for your first sentence and your main heading; if they are missing, the text is built in the browser
• Minutes 5 to 6: search the same source for noindex, nosnippet and max-snippet; if one sits on a page you want cited, ask for it to be removed
• Minutes 7 to 8: open URL Inspection in Google Search Console and paste the page's address to see whether it is indexed and how Googlebot reads it
• Minutes 9 to 10: ask whoever runs your hosting or CDN whether the firewall blocks AI crawlers or challenges them before letting them in; Perplexity says a firewall may need to allow its crawlers explicitly

On 11 October 2026 we requested the clinics guide on tasadr.ai under the names of 17 crawlers, and as a plain curl request. Every request came back with a 200 and the whole page, its main heading and its structured data included.

No request met a challenge or a verification page. The crawlers were Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, GoogleOther, Applebot, meta-externalagent, CCBot, Bytespider, DuckAssistBot and MistralAI-User.

The lesson: a site whose text is written on the server, and whose firewall blocks nobody, reaches everyone who asks for it in full. That does not mean engines cite us. A crawler reaching the page is a condition for citation, not a guarantee of it.

The test has a limit we should state. It only sends the crawler's name; it does not come from those companies' real IP addresses, so a firewall that checks addresses may treat the real crawler differently. Server logs remain the final evidence.

On 14 October 2026 we also checked the public files of 71 sites Perplexity or Claude cited in Arabic buying answers, and found no rule blocking the search crawlers in the robots.txt files we could read. The full figures are in who leads Arabic buying answers in AI? October 2026.

Nobody can guarantee that an engine will cite you because its crawler read your page. Reading is a condition; the choice after it is the engine's, and none of these companies sells a place among the sources its engine cites.

Crawler names and roles change. We took them from their owners' pages as we read them on 11 October 2026, so check them before you change your file, and do not read this guide as legal advice on protecting your content or your rights.