I built a pipeline that turns news into events
I’ve built a news pipeline that turns thousands of URLs into structured, usable events 🗞️⚙️
At first, the problem seems simple:
🔎 Find news
📝 Summarize articles
📲 Show the results to the user
But finding information is the easy part.
The hard part is deciding:
- Is it actually relevant?
- Is it important or routine?
- Is it a new event?
- Is it another source covering an event we already know about?
- Is it an update, a correction or a duplicate?
- Can we trace every claim back to its original source?
After almost a month of running the pipeline, I’ve analyzed the real data in the database 👇
The system completed 883 of 908 discovery runs: a 97.3% completion rate ✅
Those runs generated:
🔗 9,475 candidate URLs
🌍 7,299 unique URLs
🗑️ 5,147 irrelevant results
📦 133 routine or historical results
⬆️ 4,195 candidates selected for normalization
So more than half of everything we find is noise.
Then the selected items went through an identity resolution process.
The system had to decide whether each item described something new or something that already existed in our event registry.
The explicit normalization results were:
🆕 1,884 new events
🔁 1,495 matches with existing events
🧩 332 duplicate developments
📈 3 genuinely new developments
This is the most interesting part.
Most of the pipeline’s value comes from filtering out duplicate, old or low-value events.
Without identity resolution, the same event can show up as ten different headlines published by ten different outlets.
A basic news aggregator shows ten stories.
A useful intelligence system needs to understand that all ten are about the same event.
The shared registry now contains:
📌 2,339 canonical public events
📚 2,349 event developments
📰 3,713 associated sources
Every event has at least one development and one source.
The pipeline preserves the distinction between:
- The actual event
- A new development within that event
- A source reporting on the development
- The personalized view each user gets
This separation lets us properly handle corrections, updates, deduplication, timelines and source traceability.
Contrary to what it might seem, finding more information is NOT the problem.
The problem is compressing noisy, repetitive information into a smaller set of reliable facts without losing context or traceability 🎯
That’s why most news aggregators take the easy route: they pick a small set of reliable news sources that aren’t duplicated.
That saves them from having to deal with this problem.
But in exchange, they leave a ton of important news on the table.
They give up on finding everything because they refuse to turn over every stone.
PS: The data is from an older version of the pipeline and with only a few assets. The new version is even better :)
