TasadrBot, the crawler Tasadr uses to read public web pages
Last updated:
TasadrBot is the crawler Tasadr uses to read web pages that are open to everyone. It names itself in every request, reads robots.txt before every page and never requests a page it disallows, never logs in, never fills in a form and never reads a private page, and you can allow or block it with a line in your file.
On this page (9)
What is TasadrBot?
Tasadr is an Arabic-first platform that measures how AI engines answer your customers' questions. To do that it has to read pages from the web the way any visitor would, and the crawler that does the reading is called TasadrBot.
It is a program Tasadr owns and runs itself. It requests public pages and reads their text, and it changes nothing on any site it visits. If you see it in your logs, one of the reasons below is why it came, and every one of them is a read of a page open to everyone.
What does TasadrBot read, and why?
TasadrBot reads public pages only, meaning what any visitor who is not signed in can see. Every read goes back to a Tasadr customer who asked for it or who added the site in question.
On every read without exception, TasadrBot reads the site's robots.txt before any page of it, and does not request a page the file disallows for its name. If robots.txt answers with a server error or cannot be reached, it counts that as a refusal and requests nothing.
The reasons TasadrBot reads pages Reason What it reads Checking a customer's own site and its readiness for AI engines The home page, the site's key pages, robots.txt, sitemaps and llms.txt A customer's product pages The sitemaps that list products, and the public product pages themselves Pages cited in AI answers The page an answer cited, so the customer can see who is named instead of them and why A competitor's site that a customer looks up Its home page and its publicly visible pages for services, products and prices Adding a new site to an account The home page, to confirm the site works, and the tag the customer placed on their own site when they ask to prove ownership What does TasadrBot never do?
This crawler's limits are part of how it is built, not a promise we write and forget.
- It never requests a page that robots.txt disallows for its name, on any read and any site.
- It never logs in to a site and never uses a username or a password.
- It never fills in or submits a form, never clicks a button and never changes anything on your site. It requests pages for reading only.
- It never reads a page that needs an account or a subscription to open.
- It never hides who it is: its name and the address of this page are in every request it sends.
- It only reaches public sites on the web's usual ports, and never goes near an internal network.
- It does not flood your site with requests: it reads a few pages at a time, waits for the delay your robots.txt asks for, and slows down if your site says it is busy.
How do you recognise a request from TasadrBot?
TasadrBot sends the identity below in the User-Agent header of every request, and it includes the address of this page.
The name robots.txt matches is TasadrBot alone. It is what you write on a User-agent line when you want a rule for it.
TasadrBot's identity in requests Field Value User-Agent TasadrBot/ 1.0 (+https:// www.tasadr.ai/ en/ bot) Name in robots.txt TasadrBot How do you block TasadrBot from your site?
Add the two lines below to the robots.txt file at the root of your site. TasadrBot then stops requesting any page of it.
All it still asks for is robots.txt itself, to learn what you allow.
Blocking TasadrBot from the whole site Line in the file What it means User-agent: TasadrBot This group is for TasadrBot alone Disallow: / Read nothing on the site How do you block part of your site, or ask it to wait?
In TasadrBot's group, list the folders you do not want it in, and add a Crawl-delay for the number of seconds it should leave between one page and the next, within a reasonable limit.
Whatever its group does not mention it reads, as long as it is open to everyone.
Blocking a folder and asking for a pause between pages Line in the file What it means User-agent: TasadrBot This group is for TasadrBot alone Disallow: /members/ Do not read anything under this folder Crawl-delay: 10 Leave 10 seconds between one page and the next How do you allow TasadrBot while blocking other crawlers?
A crawler reads the group that carries its own name, and if there is none it reads the general group that starts with the asterisk. So giving TasadrBot a group that allows it is enough, even if the general group closes the site.
Closing the site to crawlers but allowing TasadrBot Line in the file What it means User-agent: * The general group, for every crawler that has no group of its own Disallow: / Read nothing on the site User-agent: TasadrBot A group for Tasadr's crawler Allow: / Read the site's open pages How do you contact us about TasadrBot?
If you see requests from TasadrBot and do not know why, or you want us to stop reading your site, write to support@tasadr.ai with your site's name and the User-Agent line as it appears in your logs. We answer in Arabic or English.
Who is behind the crawler is in who is behind Tasadr, every way to reach us is in contact us, and what we keep is in the privacy policy.
Frequently asked questions
What is TasadrBot?
Does TasadrBot obey robots.txt?
Does TasadrBot log in to my site?
How do I block TasadrBot?
How do I know a request really comes from TasadrBot?
Try Tasadr free for 7 days
Add your site, write up to 10 questions your customers ask, and see who leads the answer today. No credit card needed.
Start your free trial