News Analyzer
Loading visualization...
I built this pipeline to turn raw local news into a concise daily briefing. It scrapes, extracts, indexes, and summarizes—then ships the result through my Kubernetes stack.
The particles represent batches moving between stages.
Uses Playwright for auth + navigation on dynamic e-editions. Downloads source documents and keeps sessions stable across runs.
Extracted text gets chunked + embedded, then written to Qdrant / Weaviate for retrieval and dedupe across editions.
Uses LLMs through LiteLLM to generate a compact, scannable briefing and extract key entities for follow-up queries.
The “hard” part isn’t the model—it’s everything around it: logins, flaky sources, and making sure partial data doesn’t poison the index.