crawler

primal 253e04a749 Strip HTML and truncate descriptions on input

Clean descriptions when parsing feeds rather than at publish time.
Descriptions are now stored as plain text, max 300 chars.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-04 22:59:21 -05:00

cmd

Remove category field from feeds

2026-02-04 20:37:26 -05:00

scripts

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.dockerignore

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

.gitignore

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

.launch.sh

Use Unix timestamp for cache busting, remove version display

2026-02-01 18:01:04 -05:00

cdx.go

Remove category field from feeds

2026-02-04 20:37:26 -05:00

CLAUDE.md

Remove publishing code - now handled by publish service

2026-02-02 15:40:49 -05:00

crawler.go

Simplify crawler: use CDX for feed discovery, remove unused loops

2026-02-03 22:53:23 -05:00

db.go

Remove unused enclosure_length from items table

2026-02-04 22:56:45 -05:00

docker-compose.yml

Simplify crawler: use CDX for feed discovery, remove unused loops

2026-02-03 22:53:23 -05:00

Dockerfile

Update shared references to commons

2026-02-02 15:19:48 -05:00

feed_check.go

Remove last_error_at from feeds table and queries

2026-02-04 20:53:52 -05:00

feed.go

Remove item_count column from feeds table - compute dynamically from items

2026-02-04 21:00:23 -05:00

html.go

Simplify crawler: use CDX for feed discovery, remove unused loops

2026-02-03 22:53:23 -05:00

item.go

Remove unused enclosure_length from items table

2026-02-04 22:56:45 -05:00

main.go

Add bulk import for CDX feeds

2026-02-04 10:42:52 -05:00

oauth.env.example

Add AT Protocol OAuth 2.0 authentication for dashboard

2026-01-30 15:16:51 -05:00

parser.go

Strip HTML and truncate descriptions on input

2026-02-04 22:59:21 -05:00

pds.env.example

Add PDS credentials env file for service auth

2026-01-26 19:38:36 -05:00

shortener.go

Trim shortener.go - keep only URL creation, remove click tracking

2026-02-02 13:28:10 -05:00

test-app

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

tld.go

Remove next_check_at column from feeds table

2026-02-04 16:07:04 -05:00

tlds.txt

Restore working codebase with all methods

2026-02-01 19:08:53 -05:00

util.go

Strip HTML and truncate descriptions on input

2026-02-04 22:59:21 -05:00