robots.txt

robots.txt sits at a site's root, as in example.com/robots.txt, and lists rules per crawler: User-agent names the crawler, Disallow and Allow list paths. The format is the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. Most sites also use it to point to their sitemap.

It is a request, not access control: well-behaved crawlers such as Googlebot follow it, and nothing technically stops others. Blocking a page in robots.txt also does not keep it out of search results; Google can still list a URL it has not crawled. A noindex tag does that, and only on a page Google is allowed to fetch.

Clair's Web Scraper API fetches the URL you send it, one page per request; it is not a crawler and does not read robots.txt for you. Whether to fetch a page, and what the site's terms allow, is the caller's decision.