crawler

primal d4a1928fa6 Increase feed check parallelism for 1Gbps bandwidth

- Workers: 1000 -> 4000
- Work channel buffer: 1000 -> 4000
- Fetch batch size: 1000 -> 4000
- MaxIdleConns: 100 -> 2000

Should improve throughput from ~15 feeds/sec to ~50-60 feeds/sec.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-05 12:11:58 -05:00

cmd

Add clean-descriptions migration tool

2026-02-04 23:43:58 -05:00

scripts

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.dockerignore

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.gitignore

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

.launch.sh

Simplify launch script

2026-02-04 23:49:21 -05:00

cdx.go

Remove category field from feeds

2026-02-04 20:37:26 -05:00

CLAUDE.md

Remove publishing code - now handled by publish service

2026-02-02 15:40:49 -05:00

crawler.go

Increase feed check parallelism for 1Gbps bandwidth

2026-02-05 12:11:58 -05:00

db.go

Remove unused enclosure_length from items table

2026-02-04 22:56:45 -05:00

docker-compose.yml

Simplify crawler: use CDX for feed discovery, remove unused loops

2026-02-03 22:53:23 -05:00

Dockerfile

Update shared references to commons

2026-02-02 15:19:48 -05:00

feed_check.go

Remove last_error_at from feeds table and queries

2026-02-04 20:53:52 -05:00

feed.go

Remove item_count column from feeds table - compute dynamically from items

2026-02-04 21:00:23 -05:00

html.go

Simplify crawler: use CDX for feed discovery, remove unused loops

2026-02-03 22:53:23 -05:00

item.go

Remove unused enclosure_length from items table

2026-02-04 22:56:45 -05:00

main.go

Add bulk import for CDX feeds

2026-02-04 10:42:52 -05:00

oauth.env.example

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

parser.go

Strip HTML and truncate descriptions on input

2026-02-04 22:59:21 -05:00

pds.env.example

Add PDS credentials env file for service auth

2026-01-26 19:38:36 -05:00

shortener.go

Trim shortener.go - keep only URL creation, remove click tracking

2026-02-02 13:28:10 -05:00

test-app

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

tld.go

Remove next_check_at column from feeds table

2026-02-04 16:07:04 -05:00

tlds.txt

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

util.go

Strip HTML and truncate descriptions on input

2026-02-04 22:59:21 -05:00