primal f2bb1e72d2 Split domain processing into separate check and crawl loops ...

- StartDomainCheckLoop: DNS verification for unchecked domains (1000 workers)
- StartFeedCrawlLoop: Feed discovery on DNS-verified domains (100 workers)

This fixes starvation where 104M unchecked domains blocked 1.2M
DNS-verified domains from ever being crawled for feeds.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-02 20:35:46 -05:00

14 KiB

Raw Blame History

View Raw