> ## Documentation Index
> Fetch the complete documentation index at: https://clair.im/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraper API reference

> Web Scraper API: any public page as HTML. A direct fetch first, then a browser that gets past bot checks; render=true for JavaScript-built pages. One request.

Send a URL and get the page's HTML. A direct fetch goes first. When the page
blocks it or answers an error, a browser that gets past bot checks loads it
instead. `render=true` sends the page straight to the browser, for pages that
build their content in JavaScript.

It returns the page, not parsed fields. For structured results from Google,
Amazon, Glassdoor, and the other sites in the catalog, use their APIs.

See the [Web Scraper API: overview, pricing, and free tier](https://clair.im/apis/scrape).

## Endpoint

`GET /v1/scrape` · **1 request**

```sh theme={null}
curl -G https://api.clair.im/v1/scrape \
  -H "Authorization: Bearer YOUR_KEY" \
  --data-urlencode "url=https://news.ycombinator.com/"
```

## Parameters

| Parameter | Required | Contract |
| - | - | - |
| `url` | yes | The page, with `http://` or `https://`. Public hostnames only: IP addresses, localhost, custom ports, and URLs with a username or password are rejected. |
| `render` | no | `true` to load the page in a browser first, for content built in JavaScript. Default `false`. |

## Response

```json theme={null}
{
  "url": "https://news.ycombinator.com/",
  "final_url": "https://news.ycombinator.com/",
  "status": 200,
  "rendered": false,
  "title": "Hacker News",
  "html": "<html lang=\"en\" op=\"news\"><head>…</head><body>…</body></html>"
}
```

| Field | Type | Contract |
| - | - | - |
| `url` | string | The URL you asked for, without its fragment. |
| `final_url` | string | Where the page ended up after redirects. |
| `status` | integer | The HTTP status the page answered with. |
| `rendered` | boolean | `true` when a browser loaded the page. |
| `title` | string \| null | The page's `<title>`, or `null` when it has none. |
| `html` | string | The page's HTML. Rendered pages return the DOM after load. |

`status` is the target page's own status: a page that answers 404 comes back
as a `200` response with `"status": 404`. A page neither the direct fetch nor
the browser could load is a `502 scrape_failed`, and a page that does not load
within 60 seconds is a `504 timeout`. Neither is charged.

## What this API does not do

* It does not parse the page or follow links. One URL is one page.
* It does not log in, fill forms, or click.
* It does not fetch private addresses, non-HTML files, or pages over the
  size limit.

Auth, plans, rate limits, and error codes: [Authentication](https://clair.im/docs/authentication.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.