primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
91 Commits 1 Branch 0 Tags
f2978e7ab59c9363d63bddf135e31cfe85085dca
Commit Graph

91 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
primal
f2978e7ab5 Clean up debug logging 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:11:49 -05:00
primal
8a9001c02c Restore working codebase with all methods 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:08:53 -05:00
primal
211812363a Add TLD sync loop for IANA TLD updates 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:07:43 -05:00
primal
d41f9cc7c9 Fix blocking TLD sync loop - add missing go keyword 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:05:50 -05:00
primal
c6ec482d1f Add exact domain matching for domain-like search queries 
When searching for patterns like "npr.org", the search now also matches
the exact domain (host=npr, tld=org) in addition to the existing text
search across domain names, feed URLs, titles, and descriptions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:00:50 -05:00
primal
71d8ec0a39 Resize small cards to 115px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:04:01 -05:00
primal
03dcf1cedc Resize small cards to 110px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:02:28 -05:00
primal
a34a284d77 Use Unix timestamp for cache busting, remove version display 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:01:04 -05:00
primal
49c2370d84 Resize small cards to 100px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:00:00 -05:00
primal
02564bfde7 Fix CSS/JS cache busting - sync versions on launch 
.launch.sh now updates CSS and JS query params to match app version.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:58:55 -05:00
primal
3a28518366 Resize small cards to 80px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:56:55 -05:00
primal
c50ee3b03e Resize small cards to 100px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:52:53 -05:00
primal
f307e6c845 Add guards to skip migrations if already done 
Checks column types before running ALTER TYPE migrations to avoid
slow table scans on every restart. Also guards column renames.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:44:35 -05:00
primal
58bb560ae6 Resize small cards to 110px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:27:56 -05:00
primal
dd17889695 Rename rate cards: alive/min, crawl/min, check/min 
Shorter, cleaner labels for the dashboard rate cards.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:26:18 -05:00
primal
be595cb403 v100 2026-01-30 22:35:08 -05:00
primal
f49fc2f0ad v59: simplify to single feeds view with search 2026-01-30 17:16:14 -05:00
primal
9530c2ceab v58: remove all explicit font-sizes, reduce feed indentation 2026-01-30 17:11:31 -05:00
primal
3405e31f2c v57: remove font-size from stats to use default 2026-01-30 17:07:42 -05:00
primal
3147b4e48a v56: standardize font sizes to match domain name 2026-01-30 17:06:24 -05:00
primal
c5ad66ee81 v55: fix item_count to query actual DB count 2026-01-30 17:05:07 -05:00
primal
406f9397c2 v54: fix d:feeds to load items 2026-01-30 17:02:18 -05:00
primal
a3d8f4ea8e v53: add feed info and items panels with click toggles 2026-01-30 16:59:38 -05:00
primal
442e010672 v52: simplify feed row: status, count, path, title inline 2026-01-30 16:49:38 -05:00
primal
6c9702eebc v51: remove debug logging 2026-01-30 16:43:23 -05:00
primal
2289d73288 v50: add debug logging for spacer click 2026-01-30 16:41:56 -05:00
primal
51d05e18a1 v49: fix spacer click using event delegation 2026-01-30 16:39:20 -05:00
primal
57801d0946 v48: domain name links to site, spacer toggles feeds 2026-01-30 16:36:14 -05:00
primal
5b3330ba07 v47: Fix d:feeds auto-expand for hidden container 2026-01-30 16:31:32 -05:00
primal
97051f3967 v46: Click domain name to toggle feeds div 2026-01-30 16:29:05 -05:00
primal
cf34db1e6c v45: Auto-expand feed details in d:feeds mode 2026-01-30 16:23:03 -05:00
primal
f59e7dcbc3 v44: Left-justify TLD footer 2026-01-30 16:19:08 -05:00
primal
018f47449f v43: Add TLD footer with collapse button 2026-01-30 16:17:59 -05:00
primal
cbf16bfbc8 v42: Revert to persistent session cookie (24h) 2026-01-30 16:13:24 -05:00
primal
aef0826004 v41: Session cookie for browser-close logout 2026-01-30 16:12:33 -05:00
primal
e0602b0123 v40: Persist OAuth sessions to database 2026-01-30 16:09:46 -05:00
primal
31b7b61bb0 v39: Fix session cookie Secure flag for HTTP 2026-01-30 16:05:59 -05:00
primal
c374260e11 v38: d:feeds only shows feeds with items 2026-01-30 16:04:38 -05:00
primal
388e846f18 v37: Add right margin to language column 2026-01-30 16:01:47 -05:00
primal
2504927022 v36: Widen language column to 32px 2026-01-30 16:00:27 -05:00
primal
a5fe2962c3 v35: Add git commit/push to deploy script 2026-01-30 15:58:49 -05:00
primal
8192bce301 Add AT Protocol OAuth 2.0 authentication for dashboard 
- Implement full OAuth 2.0 with PKCE using haileyok/atproto-oauth-golang
- Backend For Frontend (BFF) pattern: tokens stored server-side only
- AES-256-GCM encrypted session cookies
- Auto token refresh when near expiry
- Restrict access to allowed handles (1440.news, wehrv.bsky.social)
- Add genkey utility for generating OAuth configuration
- Generic error messages to prevent handle enumeration
- Server-side logging of failed login attempts for security monitoring

New files:
- oauth.go: OAuth client wrapper and DID/handle resolution
- oauth_session.go: Session management with encrypted cookies
- oauth_middleware.go: RequireAuth middleware for route protection
- oauth_handlers.go: Login, callback, logout, metadata endpoints
- cmd/genkey/main.go: Generate OAuth secrets and JWK keypair
- oauth.env.example: Configuration template

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 15:16:51 -05:00
primal
1a2f6c15a9 Remove screenshot 2026-01-30 14:11:26 -05:00
primal
655dbfdbac v26: Fix Safari button spacing with explicit min-width 
Safari was collapsing the invisible spacer button. Added min-width
and box-sizing to force consistent width across browsers.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 14:10:56 -05:00
primal
7bd9ee7c78 v21: Button spacing and deploy script 
- 1px spacing between status buttons
- Rounded corners on all buttons
- Invisible spacer for consistent row alignment
- Added scripts/deploy.sh for auto-incrementing version on deploy

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 13:39:12 -05:00
primal
36de78bc49 v19: Add consistent row alignment with fail button spacer 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 13:30:54 -05:00
primal
eb83ca3e5d Add partial indexes for domain check and crawl loops 
- idx_domains_to_check: status WHERE last_checked_at IS NULL
- idx_domains_to_crawl: status WHERE last_checked_at IS NOT NULL AND last_crawled_at IS NULL

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 13:25:04 -05:00
primal
522233c4a2 Tune concurrency settings: 100 workers, 100 batch size, 100 buffer 
- Import batch size: 100 (was 100k)
- Domain check/crawl fetch size: 100 (was 1000)
- Feed check fetch size: 100 (was 1000)
- Worker count: 100 fixed (was NumCPU)
- Channel buffers: 100 (was 256)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 23:33:57 -05:00
primal
516848e529 Revise domain status flow: skip uses takedown, add drop for permanent deletion 
- Import default changed from 'hold' to 'pass' (auto-crawl)
- Skip now uses PDS takedown (hides posts but preserves data)
- Added 'drop' status for permanent deletion (requires skip first)
- Added TakedownAccount/RestoreAccount PDS functions
- Un-skip restores PDS accounts and reactivates feeds
- Dashboard shows 'drop' button only for skipped domains

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 23:18:17 -05:00
primal
43916c8042 Exclude skip status domains from default API listing 
When no status filter is provided, the domains API now excludes
domains with 'skip' status (including bare TLDs) by default.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-29 22:34:24 -05:00