primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
99 Commits 1 Branch 0 Tags
265975c7c52c7bdb1376fbabf270a491009b2cdc
Commit Graph

99 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
primal
265975c7c5 Rename sessions table to oauth_sessions for consistency 
- Update schema to create oauth_sessions instead of sessions
- Add migration to rename existing sessions table
- Add token_expiry column for OAuth library compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-02 00:34:13 -05:00
primal
615aa6ef5d Fix TLD sync to use domain_tld column for feeds table 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 23:52:29 -05:00
primal
3f277ec165 Remove item ID column references - items now use composite PK (guid, feed_url) 
- Remove ID field from Item struct
- Remove ID field from SearchItem struct
- Update all SQL queries to not select id column
- Change MarkItemPublished to use feedURL/guid instead of id
- Update shortener to use item_guid instead of item_id
- Add migration to convert item_id to item_guid in short_urls table
- Update API endpoints to use feedUrl/guid instead of itemId

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 23:51:44 -05:00
primal
7ec4207173 Migrate to normalized FK schema (domain_host, domain_tld) 
Replace source_host column with proper FK to domains table using
composite key (domain_host, domain_tld). This enables JOIN queries
instead of string concatenation for domain lookups.

Changes:
- Update Feed struct: SourceHost/TLD → DomainHost/DomainTLD
- Update all SQL queries to use domain_host/domain_tld columns
- Add column aliases (as source_host) for API backwards compatibility
- Update trigram index from source_host to domain_host
- Add getDomainHost() helper for extracting host from domain

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 22:36:25 -05:00
primal
e7f6be2203 Add internal crawl endpoint without auth 
For triggering priority crawls from internal network.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:59:39 -05:00
primal
edf54ca212 Add graceful shutdown for goroutines 
- Add shutdownCh channel to signal goroutines to stop
- Check IsShuttingDown() in all main loops
- Wait 2 seconds for goroutines to finish before closing DB

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:23:57 -05:00
primal
81146fd572 Fix domain search when pattern looks like domain 
When searching for "npr.org" and viewing the .org TLD, use the host part
("npr") for matching instead of the full pattern ("npr.org").

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:19:21 -05:00
primal
7011b126fe Fix tld_enum comparison - cast to text instead of LOWER() 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:13:21 -05:00
primal
f2978e7ab5 Clean up debug logging 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:11:49 -05:00
primal
8a9001c02c Restore working codebase with all methods 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:08:53 -05:00
primal
211812363a Add TLD sync loop for IANA TLD updates 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:07:43 -05:00
primal
d41f9cc7c9 Fix blocking TLD sync loop - add missing go keyword 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:05:50 -05:00
primal
c6ec482d1f Add exact domain matching for domain-like search queries 
When searching for patterns like "npr.org", the search now also matches
the exact domain (host=npr, tld=org) in addition to the existing text
search across domain names, feed URLs, titles, and descriptions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:00:50 -05:00
primal
71d8ec0a39 Resize small cards to 115px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:04:01 -05:00
primal
03dcf1cedc Resize small cards to 110px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:02:28 -05:00
primal
a34a284d77 Use Unix timestamp for cache busting, remove version display 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:01:04 -05:00
primal
49c2370d84 Resize small cards to 100px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 18:00:00 -05:00
primal
02564bfde7 Fix CSS/JS cache busting - sync versions on launch 
.launch.sh now updates CSS and JS query params to match app version.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:58:55 -05:00
primal
3a28518366 Resize small cards to 80px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:56:55 -05:00
primal
c50ee3b03e Resize small cards to 100px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:52:53 -05:00
primal
f307e6c845 Add guards to skip migrations if already done 
Checks column types before running ALTER TYPE migrations to avoid
slow table scans on every restart. Also guards column renames.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:44:35 -05:00
primal
58bb560ae6 Resize small cards to 110px 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:27:56 -05:00
primal
dd17889695 Rename rate cards: alive/min, crawl/min, check/min 
Shorter, cleaner labels for the dashboard rate cards.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 17:26:18 -05:00
primal
be595cb403 v100 2026-01-30 22:35:08 -05:00
primal
f49fc2f0ad v59: simplify to single feeds view with search 2026-01-30 17:16:14 -05:00
primal
9530c2ceab v58: remove all explicit font-sizes, reduce feed indentation 2026-01-30 17:11:31 -05:00
primal
3405e31f2c v57: remove font-size from stats to use default 2026-01-30 17:07:42 -05:00
primal
3147b4e48a v56: standardize font sizes to match domain name 2026-01-30 17:06:24 -05:00
primal
c5ad66ee81 v55: fix item_count to query actual DB count 2026-01-30 17:05:07 -05:00
primal
406f9397c2 v54: fix d:feeds to load items 2026-01-30 17:02:18 -05:00
primal
a3d8f4ea8e v53: add feed info and items panels with click toggles 2026-01-30 16:59:38 -05:00
primal
442e010672 v52: simplify feed row: status, count, path, title inline 2026-01-30 16:49:38 -05:00
primal
6c9702eebc v51: remove debug logging 2026-01-30 16:43:23 -05:00
primal
2289d73288 v50: add debug logging for spacer click 2026-01-30 16:41:56 -05:00
primal
51d05e18a1 v49: fix spacer click using event delegation 2026-01-30 16:39:20 -05:00
primal
57801d0946 v48: domain name links to site, spacer toggles feeds 2026-01-30 16:36:14 -05:00
primal
5b3330ba07 v47: Fix d:feeds auto-expand for hidden container 2026-01-30 16:31:32 -05:00
primal
97051f3967 v46: Click domain name to toggle feeds div 2026-01-30 16:29:05 -05:00
primal
cf34db1e6c v45: Auto-expand feed details in d:feeds mode 2026-01-30 16:23:03 -05:00
primal
f59e7dcbc3 v44: Left-justify TLD footer 2026-01-30 16:19:08 -05:00
primal
018f47449f v43: Add TLD footer with collapse button 2026-01-30 16:17:59 -05:00
primal
cbf16bfbc8 v42: Revert to persistent session cookie (24h) 2026-01-30 16:13:24 -05:00
primal
aef0826004 v41: Session cookie for browser-close logout 2026-01-30 16:12:33 -05:00
primal
e0602b0123 v40: Persist OAuth sessions to database 2026-01-30 16:09:46 -05:00
primal
31b7b61bb0 v39: Fix session cookie Secure flag for HTTP 2026-01-30 16:05:59 -05:00
primal
c374260e11 v38: d:feeds only shows feeds with items 2026-01-30 16:04:38 -05:00
primal
388e846f18 v37: Add right margin to language column 2026-01-30 16:01:47 -05:00
primal
2504927022 v36: Widen language column to 32px 2026-01-30 16:00:27 -05:00
primal
a5fe2962c3 v35: Add git commit/push to deploy script 2026-01-30 15:58:49 -05:00
primal
8192bce301 Add AT Protocol OAuth 2.0 authentication for dashboard 
- Implement full OAuth 2.0 with PKCE using haileyok/atproto-oauth-golang
- Backend For Frontend (BFF) pattern: tokens stored server-side only
- AES-256-GCM encrypted session cookies
- Auto token refresh when near expiry
- Restrict access to allowed handles (1440.news, wehrv.bsky.social)
- Add genkey utility for generating OAuth configuration
- Generic error messages to prevent handle enumeration
- Server-side logging of failed login attempts for security monitoring

New files:
- oauth.go: OAuth client wrapper and DID/handle resolution
- oauth_session.go: Session management with encrypted cookies
- oauth_middleware.go: RequireAuth middleware for route protection
- oauth_handlers.go: Login, callback, logout, metadata endpoints
- cmd/genkey/main.go: Generate OAuth secrets and JWK keypair
- oauth.env.example: Configuration template

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 15:16:51 -05:00