Entities live on the client
Traditional news APIs make you pick from their taxonomy. Clair inverts that. You define an entity (a short description of whatever you care about) and embed it locally. The server never stores your entity; it only ever sees a vector. Nothing proprietary is involved: the model is open, so you can do this withsentence-transformers directly:
The shared embedding space
Matching only works if your entity vector and Clair’s news vectors live in the same space. That space is pinned:
Because the model is open and deterministic,
vector = f(text, bge-m3) is
reproducible by anyone. Vectors are therefore not secret; the moat is the
maintained historical corpus, which can’t be back-filled after the fact. Every
vector-bearing request declares its model so we compare correctly and can add
new spaces later without breaking old callers.
Scoring: best-chunk cosine
A whole-article vector mean-pools a 25-sentence story into one point, which drowns out a topic mentioned in a single sentence. Clair avoids that dilution by embedding every sentence chunk of each article. An article’sscore is the maximum cosine similarity between your entity
vector and any of that article’s chunk vectors. The chunk that produced the max
is returned as best_chunk, your explanation for the match. No model sits in
the scoring loop, so results are explainable and can’t hallucinate.
How a query is planned
Ranked retrieval and complete retrieval are different questions, so they get different plans. Neither runs a model at request time.order=relevance: the top matches
1
Coarse candidates (ANN)
A whole-document HNSW approximate-nearest-neighbor search over
news_embeddings pulls the ~500 most doc-similar articles in milliseconds.
Your since/until filter is applied inside this stage, so a narrow
window returns the best matches in that window rather than starving.2
Exact best-chunk rerank
Within those candidates, the closest sentence chunk in
news_chunk_embeddings gives each article its exact score and
best_chunk.limit and stops. There’s no cursor:
a snapshot has no page two.
order=time: everything that matched
You asked for everything above a threshold, which is a different question from
“top-K”, so the answer can’t be “the best 100, sorted by date.” An article
scoring 0.52 from this morning belongs on page one even if a hundred better
matches from last month exist; cut off at 100 by score and you’d never see it.
So this mode collects the whole threshold set before it sorts anything. It
walks news_chunk_embeddings in relevance order and stops once scores fall below
your min_score, because everything at or above a threshold is a prefix of
distance order, arriving at a chunk below the line proves there’s nothing left to
find. Then it sorts what it collected by time and keyset-paginates to the end.
The cost therefore tracks how much matches, not how big the corpus is, and
it’s cheapest exactly when your query is sharpest. Pagination is stable because
HNSW search is deterministic: the same vector re-derives the same set on every
page, so pages tile it without gaps or repeats.
If the threshold set is too large to collect in one pass, you get
422 too_many_matches rather than a prefix of it. That’s the one honest failure
mode here, and both fixes are yours: raise min_score, or narrow the window.
No query: the corpus
Nothing to score, so nothing is scanned: a plain chronological keyset walk with no cap at all.score and best_chunk come back null.
One corpus, one price
There’s no separate bulk endpoint. A bulk pull is/v1/match with no query, it
pages like every other mode, and limit is capped at 100 everywhere, so an
article costs the same however you asked for it.
Pulling the corpus and matching locally with the open bge-m3 model reproduces
our scoring exactly: same model, same chunking, same math. You don’t need our
vectors to do it; they’re derivable from the text with public weights, which is
the whole point of pinning an open model.