primal f2bb1e72d2 Split domain processing into separate check and crawl loops ...

- StartDomainCheckLoop: DNS verification for unchecked domains (1000 workers)
- StartFeedCrawlLoop: Feed discovery on DNS-verified domains (100 workers)

This fixes starvation where 104M unchecked domains blocked 1.2M
DNS-verified domains from ever being crawled for feeds.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-02 20:35:46 -05:00

cmd/genkey

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

scripts

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.dockerignore

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.gitignore

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

.launch.sh

Use Unix timestamp for cache busting, remove version display

2026-02-01 18:01:04 -05:00

CLAUDE.md

Remove publishing code - now handled by publish service

2026-02-02 15:40:49 -05:00

crawler.go

Split domain processing into separate check and crawl loops

2026-02-02 20:35:46 -05:00

db.go

Add status column to items table

2026-02-02 15:46:33 -05:00

docker-compose.yml

Update shared references to commons

2026-02-02 15:19:48 -05:00

Dockerfile

Update shared references to commons

2026-02-02 15:19:48 -05:00

domain.go

Split domain processing into separate check and crawl loops

2026-02-02 20:35:46 -05:00

feed_check.go

Migrate to normalized FK schema (domain_host, domain_tld)

2026-02-01 22:36:25 -05:00

feed.go

Remove publishing code - now handled by publish service

2026-02-02 15:40:49 -05:00

html.go

Add JSON Feed support

2026-01-29 13:16:50 -05:00

item.go

Remove item ID column references - items now use composite PK (guid, feed_url)

2026-02-01 23:51:44 -05:00

main.go

Split domain processing into separate check and crawl loops

2026-02-02 20:35:46 -05:00

oauth.env.example

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

parser.go

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

pds.env.example

Add PDS credentials env file for service auth

2026-01-26 19:38:36 -05:00

shortener.go

Trim shortener.go - keep only URL creation, remove click tracking

2026-02-02 13:28:10 -05:00

test-app

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

tld.go

Fix TLD sync to use domain_tld column for feeds table

2026-02-01 23:52:29 -05:00

tlds.txt

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

util.go

Migrate to normalized FK schema (domain_host, domain_tld)

2026-02-01 22:36:25 -05:00

Description

No description provided

12 MiB

Languages

Go 98.8%

Shell 0.6%

Dockerfile 0.6%