> ## Documentation Index
> Fetch the complete documentation index at: https://clair.im/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How it works

> The stateless-vector model, the shared embedding space, and the two-stage match behind every score.

## Entities live on the client

Traditional news APIs make you pick from *their* taxonomy. Clair inverts that.
You define an entity (a short description of whatever you care about) and embed
it locally. The server never stores your entity; it only ever sees a vector.

Nothing proprietary is involved: the model is open, so you can do this with
`sentence-transformers` directly:

```python theme={null}
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-m3")
vector = model.encode(
    "Artificial Intelligence: foundation models, GPUs, inference, AI labs, chips and datacenters.",
    normalize_embeddings=True,
)  # 1024 floats; this vector is all the server ever sees
```

## The shared embedding space

Matching only works if your entity vector and Clair's news vectors live in the
**same** space. That space is pinned:

| Property      | Value                   |
| ------------- | ----------------------- |
| Model         | `bge-m3` (open weights) |
| Dimensions    | 1024, dense             |
| Normalization | L2 (unit length)        |
| Input         | instruction-free        |

Because the model is open and deterministic, `vector = f(text, bge-m3)` is
reproducible by anyone. Vectors are therefore **not secret**; the moat is the
maintained historical corpus, which can't be back-filled after the fact. Every
vector-bearing request declares its `model` so we compare correctly and can add
new spaces later without breaking old callers.

<Warning>
  Model **and** chunking, pooling, and input type all have to line up, not just
  the model name. If any of them drift when you embed yourself, your local scores
  diverge from the API's even though you're nominally in the "same" space. Text
  mode (`"text": "…"`) side-steps this entirely: the server embeds with the
  pinned settings.
</Warning>

## Scoring: best-chunk cosine

A whole-article vector mean-pools a 25-sentence story into one point, which
drowns out a topic mentioned in a single sentence. Clair avoids that dilution by
embedding **every sentence chunk** of each article.

An article's `score` is the **maximum cosine similarity** between your entity
vector and any of that article's chunk vectors. The chunk that produced the max
is returned as `best_chunk`, your explanation for the match. No model sits in
the scoring loop, so results are explainable and can't hallucinate.

## How a query is planned

Ranked retrieval and complete retrieval are different questions, so they get
different plans. Neither runs a model at request time.

### `order=relevance`: the top matches

<Steps>
  <Step title="Coarse candidates (ANN)">
    A whole-document HNSW approximate-nearest-neighbor search over
    `news_embeddings` pulls the \~500 most doc-similar articles in milliseconds.
    Your `since`/`until` filter is applied **inside** this stage, so a narrow
    window returns the best matches *in that window* rather than starving.
  </Step>

  <Step title="Exact best-chunk rerank">
    Within those candidates, the closest sentence chunk in
    `news_chunk_embeddings` gives each article its exact `score` and
    `best_chunk`.
  </Step>
</Steps>

One ANN probe plus a bounded chunk fetch: fast and cheap. It answers *"what are
the best matches?"*, so it returns the top `limit` and stops. There's no cursor:
a snapshot has no page two.

### `order=time`: everything that matched

You asked for *everything above a threshold*, which is a different question from
"top-K", so the answer can't be "the best 100, sorted by date." An article
scoring 0.52 from this morning belongs on page one even if a hundred better
matches from last month exist; cut off at 100 by score and you'd never see it.

So this mode collects the **whole** threshold set before it sorts anything. It
walks `news_chunk_embeddings` in relevance order and stops once scores fall below
your `min_score`, because everything at or above a threshold is a *prefix* of
distance order, arriving at a chunk below the line proves there's nothing left to
find. Then it sorts what it collected by time and keyset-paginates to the end.

The cost therefore tracks **how much matches**, not how big the corpus is, and
it's cheapest exactly when your query is sharpest. Pagination is stable because
HNSW search is deterministic: the same vector re-derives the same set on every
page, so pages tile it without gaps or repeats.

If the threshold set is too large to collect in one pass, you get
`422 too_many_matches` rather than a prefix of it. That's the one honest failure
mode here, and both fixes are yours: raise `min_score`, or narrow the window.

### No query: the corpus

Nothing to score, so nothing is scanned: a plain chronological keyset walk with
no cap at all. `score` and `best_chunk` come back `null`.

## One corpus, one price

There's no separate bulk endpoint. A bulk pull is `/v1/match` with no query, it
pages like every other mode, and `limit` is capped at 100 everywhere, so an
article costs the same however you asked for it.

Pulling the corpus and matching locally with the open `bge-m3` model reproduces
our scoring exactly: same model, same chunking, same math. You don't need our
vectors to do it; they're derivable from the text with public weights, which is
the whole point of pinning an open model.