> ## Documentation Index
> Fetch the complete documentation index at: https://clair.im/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How News search works

> How Clair builds a cleaned English-language index, evaluates keyword queries, orders results, and keeps pagination stable.

The News API separates acquisition from serving. Background workers discover,
fetch, clean, and index publisher material. `GET /v1/news` reads that prebuilt
index; a public request never starts a crawl or waits on a publisher.

## The indexed document

Clair builds one English full-text search document from each article's title
and cleaned body. Title matches receive more weight than body-only matches.

The public query syntax maps to PostgreSQL web-search behavior:

| Input                | Meaning                                      |
| -------------------- | -------------------------------------------- |
| `inflation`          | Require a searchable form of the term.       |
| `"central bank"`     | Match the quoted phrase.                     |
| `inflation OR rates` | Match either branch. `OR` must be uppercase. |
| `inflation -sports`  | Match inflation and exclude sports.          |

English normalization handles common inflections and removes stop words. If
normalization removes every searchable term, Clair returns `empty_query`
instead of running an empty search.

<Note>
  The public News API is keyword-first full-text search. It does not accept
  vectors, generate embeddings, or claim semantic equivalence.
</Note>

## Search and corpus-walk modes

The same endpoint supports two bounded read patterns:

1. Send `q` to search title and content.
2. Omit `q` to walk canonical articles newest first.

Source and publication-time filters work in both modes. A no-query walk is
useful for polling or maintaining your own downstream index because it does
not require a second bulk endpoint.

## Ordering

`sort_by=published_at` is the default. Results are ordered by publisher time
descending, then stable article ID descending. With `q`, each item still
includes a query-local `relevance_score`.

`sort_by=relevance` orders by text rank descending, then publisher time and ID.
It requires `q`.

The score is normalized to the interval from 0 to 1 for the current query. It
is a ranking signal, not a probability, confidence estimate, or value that
should be compared across unrelated queries.

## Stable cursors

Clair requests one extra row to decide whether another page exists. When it
does, the response encodes the final returned sort position, exact
microsecond timestamp, stable ID, sort mode, and a fingerprint of the inputs in
an opaque cursor.

On the next request, Clair verifies that the cursor still belongs to the same:

* Query.
* Source and time filters.
* Sort order.
* Revision mode.

A mismatch returns `bad_cursor`. This protects a polling loop from silently
skipping or duplicating rows after a filter changes.

There is no request-time total count. Counting every match in a broad archive
query would add latency without helping stable traversal; `next_cursor` is the
continuation contract.

## Canonical records and edits

Publishers sometimes update an article in place at the same URL. Clair retains
the records it collected, but the default read returns only the canonical
version for each URL.

Set `include_edits=true` to expose later revisions. `is_update=true` identifies
them. Keep the stable Clair `id`, publisher `url`, `published_at`, and
`collected_at` together when revision history matters.

## Content states

The acquisition pipeline derives a bounded snippet from publisher metadata or
cleaned text. If extraction does not yield usable text, Clair can fall back to
the feed excerpt. If neither is available, `content` is null. Clair does not
expose a full-article archive.

Two flags make the state explicit:

* `content_available` distinguishes absent content from an empty-looking
  consumer field.
* `content_truncated` identifies a snippet capped at the public 2,000-character
  limit.

Clair does not summarize, generate, or silently fill missing content. Keep the
publisher URL as the source of record.

## Runtime boundaries

News search depends on Clair's database and full-text index. It has no runtime
dependency on an embedding service or vector index. Platform readiness,
per-route deadlines, and concurrency limits isolate a News failure from the
Jobs product.

For semantic retrieval or RAG, use Clair as the acquisition and synchronization
layer, then chunk, embed, or rerank the returned records in your own stack.