How News search works

The News API separates acquisition from serving. Background workers discover, fetch, clean, and index publisher material. GET /v1/news reads that prebuilt index; a public request never starts a crawl or waits on a publisher.

​

The indexed document

Clair builds one English full-text search document from each article’s title and cleaned body. Title matches receive more weight than body-only matches. The public query syntax maps to PostgreSQL web-search behavior: English normalization handles common inflections and removes stop words. If normalization removes every searchable term, Clair returns empty_query instead of running an empty search.

​

Search and corpus-walk modes

The same endpoint supports two bounded read patterns:

1. Send q to search title and content.
2. Omit q to walk canonical articles newest first.

Source and publication-time filters work in both modes. A no-query walk is useful for polling or maintaining your own downstream index because it does not require a second bulk endpoint.

​

Ordering

sort_by=published_at is the default. Results are ordered by publisher time descending, then stable article ID descending. With q, each item still includes a query-local relevance_score. sort_by=relevance orders by text rank descending, then publisher time and ID. It requires q. The score is normalized to the interval from 0 to 1 for the current query. It is a ranking signal, not a probability, confidence estimate, or value that should be compared across unrelated queries.

​

Stable cursors

Clair requests one extra row to decide whether another page exists. When it does, the response encodes the final returned sort position, exact microsecond timestamp, stable ID, sort mode, and a fingerprint of the inputs in an opaque cursor. On the next request, Clair verifies that the cursor still belongs to the same:

• Query.
• Source and time filters.
• Sort order.
• Revision mode.

A mismatch returns bad_cursor. This protects a polling loop from silently skipping or duplicating rows after a filter changes. There is no request-time total count. Counting every match in a broad archive query would add latency without helping stable traversal; next_cursor is the continuation contract.

​

Canonical records and edits

Publishers sometimes update an article in place at the same URL. Clair retains the records it collected, but the default read returns only the canonical version for each URL. Set include_edits=true to expose later revisions. is_update=true identifies them. Keep the stable Clair id, publisher url, published_at, and collected_at together when revision history matters.

​

Content states

The acquisition pipeline derives a bounded snippet from publisher metadata or cleaned text. If extraction does not yield usable text, Clair can fall back to the feed excerpt. If neither is available, content is null. Clair does not expose a full-article archive. Two flags make the state explicit:

• content_available distinguishes absent content from an empty-looking consumer field.
• content_truncated identifies a snippet capped at the public 2,000-character limit.

Clair does not summarize, generate, or silently fill missing content. Keep the publisher URL as the source of record.

​

Runtime boundaries

News search depends on Clair’s database and full-text index. It has no runtime dependency on an embedding service or vector index. Platform readiness, per-route deadlines, and concurrency limits isolate a News failure from the Jobs product. For semantic retrieval or RAG, use Clair as the acquisition and synchronization layer, then chunk, embed, or rerank the returned records in your own stack.