News corpus for research and RAG

1. 01Search the topic

   AP search takes q with sort=newest and up to 20 pages. Each result has the URL, publish time, and content type; keep articles and skip videos and galleries.

   GET /v1/search?engine=ap_news&q=hurricane&sort=newest&page=1 · results.0

   {
     "position": 1,
     "title": "Powerful Hurricane Polo drenches Mexico’s coast as Nolo takes aim at Hawaii",
     "url": "https://apnews.com/article/hurricane-polo-mexico-nolo-hawaii-a036e08f59c3f3842aaba2ee69bd9d37",
     "published_at": "2026-09-24T08:20:28.000Z",
     "authors": [],
     "content_type": "article",
     "premium": false
   }

2. 02Read each article

   The article endpoint returns the story as paragraphs, with authors and section. AP stories are free to read, so body_truncated is false and the text is complete.

   GET /v1/article?engine=ap_news&url=https%3A%2F%2Fapnews.com%2Farticle%2Fhurricane-polo-mexico-nolo-hawaii-a036e08f59c3f3842aaba2ee69bd9d37 · article

   {
     "title": "Powerful Hurricane Polo drenches Mexico’s coast as Nolo takes aim at Hawaii",
     "url": "https://apnews.com/article/hurricane-polo-mexico-nolo-hawaii-a036e08f59c3f3842aaba2ee69bd9d37",
     "authors": [
       {
         "name": "Desiree Brooks",
         "url": null
       },
       {
         "name": "Megan Janetsky",
         "url": null
       }
     ],
     "section": "World News",
     "access": "free",
     "body_truncated": false,
     "paragraphs": [
       "Hurricane Polo dumped heavy rain on Mexico’s Pacific coast."
     ]
   }

3. 03Store it with its source

   Keep the URL and publish time with every document. A retrieval system that cites the article, with its date, is one readers can check.

   GET /v1/article?engine=ap_news&url=https%3A%2F%2Fapnews.com%2Farticle%2Fhurricane-polo-mexico-nolo-hawaii-a036e08f59c3f3842aaba2ee69bd9d37 · article.url

   "https://apnews.com/article/hurricane-polo-mexico-nolo-hawaii-a036e08f59c3f3842aaba2ee69bd9d37"

# Build a news corpus for a topic from AP: search, then read each article,
# and write one JSON line per story for analysis or a retrieval index.
import json, os, requests

API = "https://api.clair.im"
HEADERS = {"Authorization": f"Bearer {os.environ['CLAIR_API_KEY']}"}
QUERY = "hurricane"
PAGES = 5

def get(path, **params):
    r = requests.get(f"{API}{path}", headers=HEADERS, timeout=60,
                     params={"engine": "ap_news", **params})
    r.raise_for_status()
    return r.json()

urls = []
for page in range(1, PAGES + 1):
    body = get("/v1/search", q=QUERY, sort="newest", page=page)
    urls += [r["url"] for r in body["results"] if r["content_type"] == "article"]
    if not body["next_page"]:
        break

with open("corpus.jsonl", "w") as f:
    for url in dict.fromkeys(urls):  # keep order, drop repeats
        a = get("/v1/article", url=url)["article"]
        f.write(json.dumps({
            "url": a["url"], "title": a["title"], "section": a["section"],
            "authors": [x["name"] for x in a["authors"]],
            "text": "\n\n".join(a["paragraphs"]),
        }) + "\n")

Set CLAIR_API_KEY to a key subscribed to the Associated Press API. Each call is one request against that API's monthly quota.

• Rights stay with AP. Analysis, research, and retrieval with citations are the usual uses; republishing stories needs an AP license.
• AP search goes back as far as AP's site search does, up to 20 pages per query. For longer periods, run narrower queries.
• Section and topic hub pages mix in most-read and right-rail cards. Search results are cleaner for a corpus.