Headless browser

A headless browser, such as Chrome driven by Playwright or Puppeteer, loads a page exactly as a user's browser would, runs its JavaScript, and lets a program read the result or click and scroll. Many modern sites send an almost empty HTML page and build the content in the browser, so a plain HTTP request sees nothing.

Browsers are slower and more expensive than plain requests, often by an order of magnitude, so scrapers use them only for pages that need them.

Clair decides per request whether a page needs a browser, and uses plain requests where they are enough. That is part of why response times differ between APIs.