User agent

Browsers send a User-Agent header such as Mozilla/5.0 (Windows NT 10.0; Win64; x64) … Chrome/…, and crawlers send their own, such as Googlebot. Servers use it to serve the right version of a page, for statistics, and to tell browsers from bots.

HTTP libraries send a default user agent that names the library, such as python-requests, and many sites block those outright. Changing the header alone rarely helps for long, because sites also check the rest of the request: other headers, the TLS handshake, and the IP address.

Clair's Web Scraper API sends requests itself and returns the page; you set the URL and whether to render, not headers.