01 / THE BOTTLENECK
RAG quality is retrieval quality
Every RAG pipeline over news has the same failure budget: whatever the retriever gets wrong, the model gets wrong more fluently. Feed an LLM three off-topic articles about the wrong “Apple” and it will confidently synthesize them. Miss the article that reported the event in different words and the model answers from a world where it didn’t happen.
Most teams discover this after building: they wire a keyword news API into their agent, watch context windows fill with near-misses, and start stacking rerankers and filters on top. The fix belongs at the source: retrieval that matches meaning, returns a score, and shows its evidence. (For the general argument against keyword matching, see Clair vs NewsAPI.)
02 / THE CHECKLIST
Five things to demand from any news API you put in front of an LLM
1. Semantic retrieval
The query is a meaning, not a string. If the API can't match a paraphrase, your agent misses the story that matters.
2. Scores you can threshold
A similarity score per article turns "relevance" from a vibe into a parameter. You decide the floor; precision becomes a dial, not a hope.
3. Chunk-level provenance
Grounded generation needs the passage that matched, not just a URL. It's the difference between a citation and a link dump.
4. A deduplicated corpus
Wire-service stories arrive dozens of times under different headlines. Without dedup you pay to embed, rank, and prompt the same story repeatedly.
5. Deterministic scoring
If the same query over the same corpus returns different scores tomorrow, your eval suite measures noise. Reproducibility is what makes retrieval testable.
Clair is built to pass this checklist: semantic matching with a per-article cosine score, the matched passage on every result (best_chunk), a curated and deduplicated corpus, and scoring that is deterministic and open source, so you can rerun it yourself. But take the checklist to every vendor you evaluate; it's the list that matters.
03 / WIRING
Two ways to plug news into your pipeline
Text mode: zero infrastructure. Send the thing you’re watching for as a sentence; Clair embeds it server-side and returns scored matches. This is the curl-only path, with no embedding model, no vector store, and nothing to deploy:
POST /v1/match
{
"text": "semiconductor export controls and chip sanctions",
"min_score": 0.6,
"limit": 20
}Vector mode: your embeddings, our corpus. If your pipeline already embeds with bge-m3 (1024-dim, open model), send the vector itself. Your query text never leaves your machine, there’s no second embedding model to reconcile, and the score is reproducible end to end:
POST /v1/match
{
"model": "bge-m3",
"vector": [/* 1024 floats */],
"min_score": 0.6
}For agents that poll, add order=time and a since timestamp: you get everyarticle above your floor since your last check, keyset-paginated. It’s a monitoring loop, not a top-N sample. The quickstart shows the full loop.
04 / EVALS
Deterministic retrieval is testable retrieval
The quiet advantage of deterministic scoring: your retrieval layer becomes a fixture. Same article, same query, same score, every run, forever. That means you can write assertions against real thresholds (“this article must score ≥ 0.6 against this query”), regression-test a prompt change without retrieval noise, and audit any historical answer by rerunning the exact match that produced its context.
Because the scoring algorithm is public (Apache-2.0), an eval failure is debuggable all the way down: you can reproduce the score locally and see precisely which chunk of which article produced it. No black-box relevance ranking survives that sentence.
05 / FAQ
Common questions
Why not just use a keyword news API for RAG?
Because your pipeline inherits the retriever's errors before the model ever runs. Keyword search returns articles that contain your string but aren't about your topic, and misses articles that are about it without saying the word. The context you hand the LLM ends up both polluted and incomplete, and no amount of prompting repairs that.
Can I use my own embeddings with Clair?
Yes. Send a 1024-dimension bge-m3 vector to POST /v1/match and Clair scores the corpus against it directly: no server-side embedding model in the loop, fully reproducible, and your query text never leaves your machine. Or send plain text and Clair embeds it for you.
Does Clair generate summaries or answers?
No. There is no LLM in the serving path: matching is deterministic vector math over published articles, and every result carries its cosine score and the exact passage that produced it. Generation is your side of the pipeline; Clair's job is retrieval you can audit.