Can AI engines reach and read your site?

All eight AI engines against your robots.txt, your page tags, your page text and your sitemap, which AI crawlers visit, and what to fix first.

Start your free trial See the plans

Free for 7 days, no credit card

Or check your site for free first, no sign-up

each engine's access to your site: 8 engines
crawling, indexing, pages and files: 4 areas
and a task for every problem: Fix ready

The AI Readiness page in the Tasadr dashboard: a summary saying 4 of 8 engines read the site with no problem, 3 problems may stop or limit some engines reading it, and AI crawlers made 1,281 requests in the last 7 days, over the four areas, crawler access, indexing, page reading and site files, each with its problems — You do not know whether AI engines can reach your site at allHow many engines read your site, and how many problems stop the rest

Each engine

Can each AI engine read your site?

All eight engines in one table, crawling, indexing and pages for each, so you see which engine reads your site with no problem and which one a problem holds back.

The "Can each engine read your site?" table in AI Readiness: the eight engines against crawling, indexing, pages and their crawlers' requests, Gemini, Google AI Overviews, Google AI Mode and Copilot with no problem, a problem marked on Perplexity's crawling and on ChatGPT's, Perplexity's and Claude's pages, Claude, Copilot and Grok outside the plan, and Grok with no declared crawler — An engine does not cite your site, and you do not know if it reads itAll eight engines in one table, each with its access verdict

The summary firsthow many of the eight engines read your site with no problem, and how many problems may stop or limit some of them
Each engine's crawlerwhat robots.txt tells each engine's crawler, with the line number that allows or blocks it
Engines outside your planwe check all eight engines' access on every plan, and mark the ones your plan does not analyse "Not in your plan"
Confirmed or indicationevery verdict carries its strength, "Confirmed" when your files and pages settle it, "Indication" when we cannot see everything that decides it
What we cannot confirmsaid beside the result, such as protection that refuses crawlers by name, and Grok, for which xAI publishes no crawler

Four areas

Crawler access, indexing, page text and the sitemap

One check in four areas, crawler access, indexing, page reading and site files, each with its problems and its details.

Details by area in AI Readiness on the page reading tab: a row saying two pages are empty before the page runs in a browser, /oud-collections and /gift-sets, marked as an indication, a row saying 9 of 10 pages can be read, with /blog/oud-guide answering with an error, and one saying the pages are within the size Google reads, over the pages table, where the two pages are empty and the home page, /oud/cambodian and /best-oud have text — Your page looks complete in a browser, and a crawler may find it emptyEach area with its problems, and every page we checked with its state

Crawler accesswhether robots.txt blocks an engine's crawler from your whole site or from some pages, or Cloudflare's settings block it
Indexingthe tags that ask for a page not to be indexed, limit what Google quotes from it, or keep Copilot from using it in its answers
Page readingwhether your pages have text before JavaScript runs, since the ChatGPT, Perplexity and Claude crawlers do not run it, and whether a page is small enough for Google to read in full
Site fileswhether robots.txt answers and every line of it is understood, and whether your sitemap loads and is named in it
The pages we checkedyour home page, the pages engines cite, the pages with a task and pages from your sitemap, each with its state

What each engine's crawler reads is in the AI crawlers guide.

Training and search

Did you block a training crawler, or a search crawler with it?

Blocking training crawlers alone does not block search crawlers, but one line in the file can block both, so we show which line blocks which crawler.

A card in AI Readiness saying the crawler instructions file blocks Google-Extended on lines 7 and 8, with the buttons to keep the block or see how to allow it, over the training crawlers tab: GPTBot, ClaudeBot and Applebot-Extended allowed on line 2, Google-Extended on line 8 and CCBot on line 11 blocked by your choice — You blocked training crawlers, and do not know if search crawlers went tooWhich line blocks which crawler, and the Google-Extended decision in your hands

Search crawlersread your pages so the engine can cite them in its answers; they decide its access to your site
Training crawlerssuch as GPTBot and ClaudeBot collect content to train models, and blocking them is your call
The shared linewhen one line blocks a training crawler and a search crawler together, we call it a problem and give you its fix
The Google-Extended decisionit controls whether Google uses your content to train Gemini and in Gemini's answers, and does not affect Google Search results, Google AI Overviews or Google AI Mode, so it stays your decision, never a task
Other crawlersfor your information, such as the fetcher that opens a page when a person asks for it in a chat

Why the Google-Extended decision leaves Google's search answers alone is in Google AI Overviews in Saudi Arabia.

Crawler visits

Which AI crawlers actually visit your site?

Your file says who is allowed; Cloudflare says who came. If your site runs through Cloudflare, connect it in one click and see each crawler's requests beside its access verdict.

The AI crawlers reading your site card in AI Readiness, from Cloudflare: 1,281 requests in the last 7 days and 5,005 in the last 28, split into search, assistant and training, a table of each crawler with its kind and requests, GPTBot, OAI-SearchBot, Applebot, ClaudeBot and ChatGPT-User, and the most read pages, /oud-collections, /oud/cambodian and /gift-sets — Your file allows the crawlers, and you do not know if they cameEach crawler's requests from Cloudflare, beside its access verdict

Each crawler's requestshow many requests each AI crawler made, as Cloudflare logged them, and the page it asked for most
Training, search or assistanta crawler that reads to train models, one that builds an AI search engine's index, and one that opens the page because someone asked about it just now
Most read pagesthe pages crawlers ask for most
In the engines tableeach engine's crawler requests on its row, beside its access verdict
Read-onlyaggregate counts with no data about your visitors; we change nothing in your Cloudflare account, and disconnecting withdraws the permission and deletes what we read

The same connection adds your site's server-side visits to AI Referral Traffic.

Not on Cloudflare? Start with the free check.

The fix

What to fix first, and how

The problems come in order of priority, each with its fix in place, for you to apply or send to your developer, and then we check it.

The problems in order of priority in AI Readiness, the first one open: let Perplexity's crawler read your site, high impact, with the robots.txt change showing the PerplexityBot and Disallow lines in red to remove, the buttons to copy the change or the corrected file, then the problems to make two pages' text readable to ChatGPT, Perplexity and Claude, and to name the sitemap in the crawler instructions file — You know the problem, not exactly what to change in the fileThe change ready line by line, and a named task in Priority Fixes

The change, readythe robots.txt change line by line, removed and added, to copy as a change or as the whole corrected file
A task in Priority Fixeseach problem becomes a named task, such as "Let Perplexity's crawler read your site", for you to assign to your team
Closes on its ownonce we find the fix on your site the task closes, and "Check now" is there when you do not want to wait
An alert when something changesan in-app alert when robots.txt or Cloudflare's settings newly block an engine's crawler, a page newly asks not to be indexed, or the file stops answering
A comparison after you allow a crawleronce an engine's crawler is let in, you compare how often that engine cites your site before and after, a comparison, never a cause

Agencies pass the fix to their client's developer as it is, and follow its tasks in Priority Fixes.

Data and speed

And beside it, structured data and page speed

Once the engines can read your site, what they understand from it comes next. On the Site Signals page, beside AI Readiness, you find structured data, speed and your business's identity.

Structured data on your pageswhat your key pages carry and lack, with ready code built from your pages and your business profile
Page speed on a phonetime to show the main content, response to taps and layout stability, with what to do about each
Your business's identitywhether your business has a Wikidata item, and whether its name, address and phone match wherever it is named

What your pages need from it is in the structured data guide.

Your plan

A plan from your site's readiness, do, review, watch, follow up

The page ends with a plan drawn from its results, and each task opens in Priority Fixes.

Dothe highest-priority problem, such as an engine's crawler your file blocks, or a page with no text before JavaScript runs
Reviewthe Google-Extended decision, and the pages that did not answer in the last check
Watchthe last check of robots.txt and your home page, and the AI crawler requests from Cloudflare
Follow upthe next full check, and the closed tasks, including any problem that came back after a fix

AI Readiness

Where is AI Readiness in your dashboard?

In the Search and Site section, and every problem is a task in Priority Fixes.

AI Readiness
each engine's access, the four areas, the problems with their fixes, and AI crawlers from Cloudflare
Site Signals
the structured data on your pages, their speed on a phone, and your business's identity
Priority Fixes
a task for every problem, to assign to your team, closed once we find the fix on your site

Frequently asked questions

How is the free check different from AI Readiness?

The free check needs no sign-up. It reads your home page, robots.txt, your sitemap and llms.txt, and tells you which engines' crawlers can read your site. AI Readiness, in your dashboard, checks more pages, gives you each fix ready with a task for every problem, alerts you when something changes, and adds crawler visits from Cloudflare.

Does being ready mean the engines will cite my site?

No. An engine reading your site does not mean it will cite it: being read is a condition for a citation, not a guarantee, and nobody can promise you one. Answer Tracking shows who cites you and who leads instead of you.

Do you change my robots.txt or my site?

No. Tasadr does not edit your site, its files or your Cloudflare settings. TasadrBot reads your public files and follows your robots.txt. We give you the change, ready; you, your team or your agency publish it, and we check it once it is live.

Do I need Cloudflare?

No. The check works on any site. Connecting Cloudflare adds the crawlers that actually visit, and it works when your site's traffic passes through Cloudflare. Some figures depend on your domain's Cloudflare plan. You connect it in one click and disconnect it whenever you like.

Where are structured data and page speed?

On the Site Signals page, beside AI Readiness in your dashboard: the structured data on your key pages with ready code built from your pages, your pages' speed on a phone, and your business's identity on Wikidata.

Is AI Readiness on every plan?

Yes, on every plan and in the free trial. We check all eight engines' access on every plan and mark the engines your plan does not analyse. When the trial ends, your last check's result stays in your account. Each plan is on the plans page.

Find out whether the engines can read your site

Add your site and see all eight engines' access to it, with every problem and its fix. Free for 7 days, no credit card.

Start your free trial Compare the plans in detail