primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
100 Commits 1 Branch 0 Tags
86d669e08e06d3faceba06d7a24a4460d15bcbd1
Commit Graph

14 Commits

Author SHA1 Message Date
primal
7ec4207173 Migrate to normalized FK schema (domain_host, domain_tld) 
Replace source_host column with proper FK to domains table using
composite key (domain_host, domain_tld). This enables JOIN queries
instead of string concatenation for domain lookups.

Changes:
- Update Feed struct: SourceHost/TLD → DomainHost/DomainTLD
- Update all SQL queries to use domain_host/domain_tld columns
- Add column aliases (as source_host) for API backwards compatibility
- Update trigram index from source_host to domain_host
- Add getDomainHost() helper for extracting host from domain

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 22:36:25 -05:00
primal
81146fd572 Fix domain search when pattern looks like domain 
When searching for "npr.org" and viewing the .org TLD, use the host part
("npr") for matching instead of the full pattern ("npr.org").

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:19:21 -05:00
primal
7011b126fe Fix tld_enum comparison - cast to text instead of LOWER() 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:13:21 -05:00
primal
c6ec482d1f Add exact domain matching for domain-like search queries 
When searching for patterns like "npr.org", the search now also matches
the exact domain (host=npr, tld=org) in addition to the existing text
search across domain names, feed URLs, titles, and descriptions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:00:50 -05:00
primal
be595cb403 v100 2026-01-30 22:35:08 -05:00
primal
f49fc2f0ad v59: simplify to single feeds view with search 2026-01-30 17:16:14 -05:00
primal
c5ad66ee81 v55: fix item_count to query actual DB count 2026-01-30 17:05:07 -05:00
primal
442e010672 v52: simplify feed row: status, count, path, title inline 2026-01-30 16:49:38 -05:00
primal
c374260e11 v38: d:feeds only shows feeds with items 2026-01-30 16:04:38 -05:00
primal
a5fe2962c3 v35: Add git commit/push to deploy script 2026-01-30 15:58:49 -05:00
primal
8192bce301 Add AT Protocol OAuth 2.0 authentication for dashboard 
- Implement full OAuth 2.0 with PKCE using haileyok/atproto-oauth-golang
- Backend For Frontend (BFF) pattern: tokens stored server-side only
- AES-256-GCM encrypted session cookies
- Auto token refresh when near expiry
- Restrict access to allowed handles (1440.news, wehrv.bsky.social)
- Add genkey utility for generating OAuth configuration
- Generic error messages to prevent handle enumeration
- Server-side logging of failed login attempts for security monitoring

New files:
- oauth.go: OAuth client wrapper and DID/handle resolution
- oauth_session.go: Session management with encrypted cookies
- oauth_middleware.go: RequireAuth middleware for route protection
- oauth_handlers.go: Login, callback, logout, metadata endpoints
- cmd/genkey/main.go: Generate OAuth secrets and JWK keypair
- oauth.env.example: Configuration template

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 15:16:51 -05:00
primal
516848e529 Revise domain status flow: skip uses takedown, add drop for permanent deletion 
- Import default changed from 'hold' to 'pass' (auto-crawl)
- Skip now uses PDS takedown (hides posts but preserves data)
- Added 'drop' status for permanent deletion (requires skip first)
- Added TakedownAccount/RestoreAccount PDS functions
- Un-skip restores PDS accounts and reactivates feeds
- Dashboard shows 'drop' button only for skipped domains

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 23:18:17 -05:00
primal
43916c8042 Exclude skip status domains from default API listing 
When no status filter is provided, the domains API now excludes
domains with 'skip' status (including bare TLDs) by default.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 22:34:24 -05:00
primal
1066f42189 Refactor large Go files into focused modules 
Split dashboard.go (3,528 lines) into:
- routes.go: HTTP route registration
- api_domains.go: Domain API handlers
- api_feeds.go: Feed API handlers
- api_publish.go: Publishing API handlers
- api_search.go: Search API handlers
- templates.go: HTML templates
- dashboard.go: Stats functions only (235 lines)

Split publisher.go (1,502 lines) into:
- pds_auth.go: Authentication and account management
- pds_records.go: Record operations (upload, update, delete)
- handle.go: Handle derivation from feed URLs
- image.go: Image processing and favicon fetching
- publisher.go: Core types and PublishItem (439 lines)

Split feed.go (1,137 lines) into:
- item.go: Item struct and DB operations
- feed_check.go: Feed checking and processing
- feed.go: Feed struct and DB operations (565 lines)

Also includes domain import batch size increase (1k -> 100k).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 22:25:02 -05:00