Web Scraper API reference

Send a URL and get the page’s HTML. A direct fetch goes first. When the page blocks it or answers an error, a browser that gets past bot checks loads it instead. render=true sends the page straight to the browser, for pages that build their content in JavaScript. It returns the page, not parsed fields. For structured results from Google, Amazon, Glassdoor, and the other sites in the catalog, use their APIs. See the Web Scraper API: overview, pricing, and free tier.

Endpoint

GET /v1/scrape · 1 request

curl -G https://api.clair.im/v1/scrape \
  -H "Authorization: Bearer YOUR_KEY" \
  --data-urlencode "url=https://news.ycombinator.com/"

Parameters

Parameter	Required	Contract
`url`	yes	The page, with `http://` or `https://`. Public hostnames only: IP addresses, localhost, custom ports, and URLs with a username or password are rejected.
`render`	no	`true` to load the page in a browser first, for content built in JavaScript. Default `false`.

Response

{
  "url": "https://news.ycombinator.com/",
  "final_url": "https://news.ycombinator.com/",
  "status": 200,
  "rendered": false,
  "title": "Hacker News",
  "html": "<html lang=\"en\" op=\"news\"><head>…</head><body>…</body></html>"
}

Field	Type	Contract
`url`	string	The URL you asked for, without its fragment.
`final_url`	string	Where the page ended up after redirects.
`status`	integer	The HTTP status the page answered with.
`rendered`	boolean	`true` when a browser loaded the page.
`title`	string \| null	The page’s `<title>`, or `null` when it has none.
`html`	string	The page’s HTML. Rendered pages return the DOM after load.

status is the target page’s own status: a page that answers 404 comes back as a 200 response with "status": 404. A page neither the direct fetch nor the browser could load is a 502 scrape_failed, and a page that does not load within 60 seconds is a 504 timeout. Neither is charged.

What this API does not do

It does not parse the page or follow links. One URL is one page.
It does not log in, fill forms, or click.
It does not fetch private addresses, non-HTML files, or pages over the size limit.

Auth, plans, rate limits, and error codes: Authentication.

​Endpoint

​Parameters

​Response

​What this API does not do

Endpoint

Parameters

Response

What this API does not do