primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
112 Commits 1 Branch 0 Tags
main
Commit Graph

10 Commits

Author SHA1 Message Date
primal
6eaa39f9db Remove publishing code - now handled by publish service 
Publishing functionality has been moved to the standalone publish service.
Removed:
- publisher.go, pds_auth.go, pds_records.go, image.go, handle.go
- StartPublishLoop and related functions from crawler.go
- Publish loop invocation from main.go

Updated CLAUDE.md to reflect the new architecture.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-02 15:40:49 -05:00
primal
8a9001c02c Restore working codebase with all methods 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:08:53 -05:00
primal
be595cb403 v100 2026-01-30 22:35:08 -05:00
primal
8192bce301 Add AT Protocol OAuth 2.0 authentication for dashboard 
- Implement full OAuth 2.0 with PKCE using haileyok/atproto-oauth-golang
- Backend For Frontend (BFF) pattern: tokens stored server-side only
- AES-256-GCM encrypted session cookies
- Auto token refresh when near expiry
- Restrict access to allowed handles (1440.news, wehrv.bsky.social)
- Add genkey utility for generating OAuth configuration
- Generic error messages to prevent handle enumeration
- Server-side logging of failed login attempts for security monitoring

New files:
- oauth.go: OAuth client wrapper and DID/handle resolution
- oauth_session.go: Session management with encrypted cookies
- oauth_middleware.go: RequireAuth middleware for route protection
- oauth_handlers.go: Login, callback, logout, metadata endpoints
- cmd/genkey/main.go: Generate OAuth secrets and JWK keypair
- oauth.env.example: Configuration template

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 15:16:51 -05:00
primal
522233c4a2 Tune concurrency settings: 100 workers, 100 batch size, 100 buffer 
- Import batch size: 100 (was 100k)
- Domain check/crawl fetch size: 100 (was 1000)
- Feed check fetch size: 100 (was 1000)
- Worker count: 100 fixed (was NumCPU)
- Channel buffers: 100 (was 256)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 23:33:57 -05:00
primal
516848e529 Revise domain status flow: skip uses takedown, add drop for permanent deletion 
- Import default changed from 'hold' to 'pass' (auto-crawl)
- Skip now uses PDS takedown (hides posts but preserves data)
- Added 'drop' status for permanent deletion (requires skip first)
- Added TakedownAccount/RestoreAccount PDS functions
- Un-skip restores PDS accounts and reactivates feeds
- Dashboard shows 'drop' button only for skipped domains

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 23:18:17 -05:00
primal
3999e96f26 Dashboard UI overhaul: inline feed details, TLD filtering, status improvements 
- Feed details now expand inline instead of navigating to new page
- Add TLD section headers with domains sorted by TLD then name
- Add TLD filter button to show/hide domain sections by TLD
- Feed status behavior: pass creates account, hold crawls only, skip stops, drop cleans up
- Auto-follow new accounts from directory account (1440.news)
- Fix handle derivation (removed duplicate .1440.news suffix)
- Increase domain import batch size to 100k
- Various bug fixes for account creation and profile updates

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 20:51:05 -05:00
primal
f4afb29980 Migrate from SQLite to PostgreSQL 
- Replace modernc.org/sqlite with jackc/pgx/v5
- Update all SQL queries for PostgreSQL syntax ($1, $2 placeholders)
- Use snake_case column names throughout
- Replace SQLite FTS5 with PostgreSQL tsvector/tsquery full-text search
- Add connection pooling with pgxpool
- Support Docker secrets for database password
- Add trigger to normalize feed URLs (strip https://, http://, www.)
- Fix anchor feed detection regex to avoid false positives
- Connect app container to atproto network for PostgreSQL access
- Add version indicator to dashboard UI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-28 20:38:13 -05:00
primal
93ab1f8117 Update CLAUDE.md to reflect current multi-file architecture 
The codebase evolved from a single-file app to a multi-file structure
with SQLite persistence, dashboard, and concurrent processing loops.
Updated documentation to accurately describe current architecture.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-26 13:47:19 -05:00
primal
f4cae127cc Add feed crawler with documentation 
- main.go: RSS/Atom feed crawler using Common Crawl data
- CLAUDE.md: Project documentation for Claude Code
- .gitignore: Ignore binary and go.* files
- Feed output now written to feed/ directory

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-22 15:15:30 -05:00