primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 07621a7059 Switch back to infra-dns for DNS lookups main primal 2026-02-02 21:02:28 -05:00
  • e6761954c0 Use system DNS resolver instead of custom infra-dns primal 2026-02-02 20:55:57 -05:00
  • f2bb1e72d2 Split domain processing into separate check and crawl loops primal 2026-02-02 20:35:46 -05:00
  • 26de5d3753 Add status column to items table primal 2026-02-02 15:46:33 -05:00
  • 6eaa39f9db Remove publishing code - now handled by publish service primal 2026-02-02 15:40:49 -05:00
  • 7b50f5c008 Update shared references to commons primal 2026-02-02 15:19:48 -05:00
  • bd76ea1108 Trim shortener.go - keep only URL creation, remove click tracking primal 2026-02-02 13:28:10 -05:00
  • aea101a5e7 Update short URLs to use news.1440.news primal 2026-02-02 13:23:24 -05:00
  • ec53ad59db Phase 5: Remove dashboard code from crawler primal 2026-02-02 13:08:48 -05:00
  • fa82d8b765 Move plan to dedicated plans/ directory primal 2026-02-02 12:40:48 -05:00
  • 98bee87c05 Add dashboard separation plan primal 2026-02-02 12:39:08 -05:00
  • bce9369cb8 Fix OAuth session storage - add missing database columns primal 2026-02-02 00:44:19 -05:00
  • 86d669e08e Make oauth_sessions.access_token nullable primal 2026-02-02 00:35:53 -05:00
  • 265975c7c5 Rename sessions table to oauth_sessions for consistency primal 2026-02-02 00:34:13 -05:00
  • 615aa6ef5d Fix TLD sync to use domain_tld column for feeds table primal 2026-02-01 23:52:29 -05:00
  • 3f277ec165 Remove item ID column references - items now use composite PK (guid, feed_url) primal 2026-02-01 23:51:44 -05:00
  • 7ec4207173 Migrate to normalized FK schema (domain_host, domain_tld) primal 2026-02-01 22:36:25 -05:00
  • e7f6be2203 Add internal crawl endpoint without auth primal 2026-02-01 19:59:39 -05:00
  • edf54ca212 Add graceful shutdown for goroutines primal 2026-02-01 19:23:57 -05:00
  • 81146fd572 Fix domain search when pattern looks like domain primal 2026-02-01 19:19:21 -05:00
  • 7011b126fe Fix tld_enum comparison - cast to text instead of LOWER() primal 2026-02-01 19:13:21 -05:00
  • f2978e7ab5 Clean up debug logging primal 2026-02-01 19:11:49 -05:00
  • 8a9001c02c Restore working codebase with all methods primal 2026-02-01 19:08:53 -05:00
  • 211812363a Add TLD sync loop for IANA TLD updates primal 2026-02-01 19:07:43 -05:00
  • d41f9cc7c9 Fix blocking TLD sync loop - add missing go keyword primal 2026-02-01 19:05:50 -05:00
  • c6ec482d1f Add exact domain matching for domain-like search queries primal 2026-02-01 19:00:50 -05:00
  • 71d8ec0a39 Resize small cards to 115px primal 2026-02-01 18:04:01 -05:00
  • 03dcf1cedc Resize small cards to 110px primal 2026-02-01 18:02:28 -05:00
  • a34a284d77 Use Unix timestamp for cache busting, remove version display primal 2026-02-01 18:01:04 -05:00
  • 49c2370d84 Resize small cards to 100px primal 2026-02-01 18:00:00 -05:00
  • 02564bfde7 Fix CSS/JS cache busting - sync versions on launch primal 2026-02-01 17:58:55 -05:00
  • 3a28518366 Resize small cards to 80px primal 2026-02-01 17:56:55 -05:00
  • c50ee3b03e Resize small cards to 100px primal 2026-02-01 17:52:53 -05:00
  • f307e6c845 Add guards to skip migrations if already done primal 2026-02-01 17:44:35 -05:00
  • 58bb560ae6 Resize small cards to 110px primal 2026-02-01 17:27:56 -05:00
  • dd17889695 Rename rate cards: alive/min, crawl/min, check/min primal 2026-02-01 17:26:18 -05:00
  • be595cb403 v100 primal 2026-01-30 22:35:08 -05:00
  • f49fc2f0ad v59: simplify to single feeds view with search primal 2026-01-30 17:16:14 -05:00
  • 9530c2ceab v58: remove all explicit font-sizes, reduce feed indentation primal 2026-01-30 17:11:31 -05:00
  • 3405e31f2c v57: remove font-size from stats to use default primal 2026-01-30 17:07:42 -05:00
  • 3147b4e48a v56: standardize font sizes to match domain name primal 2026-01-30 17:06:24 -05:00
  • c5ad66ee81 v55: fix item_count to query actual DB count primal 2026-01-30 17:05:07 -05:00
  • 406f9397c2 v54: fix d:feeds to load items primal 2026-01-30 17:02:18 -05:00
  • a3d8f4ea8e v53: add feed info and items panels with click toggles primal 2026-01-30 16:59:38 -05:00
  • 442e010672 v52: simplify feed row: status, count, path, title inline primal 2026-01-30 16:49:38 -05:00
  • 6c9702eebc v51: remove debug logging primal 2026-01-30 16:43:23 -05:00
  • 2289d73288 v50: add debug logging for spacer click primal 2026-01-30 16:41:56 -05:00
  • 51d05e18a1 v49: fix spacer click using event delegation primal 2026-01-30 16:39:20 -05:00
  • 57801d0946 v48: domain name links to site, spacer toggles feeds primal 2026-01-30 16:36:14 -05:00
  • 5b3330ba07 v47: Fix d:feeds auto-expand for hidden container primal 2026-01-30 16:31:32 -05:00
  • 97051f3967 v46: Click domain name to toggle feeds div primal 2026-01-30 16:29:05 -05:00
  • cf34db1e6c v45: Auto-expand feed details in d:feeds mode primal 2026-01-30 16:23:03 -05:00
  • f59e7dcbc3 v44: Left-justify TLD footer primal 2026-01-30 16:19:08 -05:00
  • 018f47449f v43: Add TLD footer with collapse button primal 2026-01-30 16:17:59 -05:00
  • cbf16bfbc8 v42: Revert to persistent session cookie (24h) primal 2026-01-30 16:13:24 -05:00
  • aef0826004 v41: Session cookie for browser-close logout primal 2026-01-30 16:12:33 -05:00
  • e0602b0123 v40: Persist OAuth sessions to database primal 2026-01-30 16:09:46 -05:00
  • 31b7b61bb0 v39: Fix session cookie Secure flag for HTTP primal 2026-01-30 16:05:59 -05:00
  • c374260e11 v38: d:feeds only shows feeds with items primal 2026-01-30 16:04:38 -05:00
  • 388e846f18 v37: Add right margin to language column primal 2026-01-30 16:01:47 -05:00
  • 2504927022 v36: Widen language column to 32px primal 2026-01-30 16:00:27 -05:00
  • a5fe2962c3 v35: Add git commit/push to deploy script primal 2026-01-30 15:58:49 -05:00
  • 8192bce301 Add AT Protocol OAuth 2.0 authentication for dashboard primal 2026-01-30 15:16:51 -05:00
  • 1a2f6c15a9 Remove screenshot primal 2026-01-30 14:11:26 -05:00
  • 655dbfdbac v26: Fix Safari button spacing with explicit min-width primal 2026-01-30 14:10:56 -05:00
  • 7bd9ee7c78 v21: Button spacing and deploy script primal 2026-01-30 13:39:12 -05:00
  • 36de78bc49 v19: Add consistent row alignment with fail button spacer primal 2026-01-30 13:30:54 -05:00
  • eb83ca3e5d Add partial indexes for domain check and crawl loops primal 2026-01-30 13:25:04 -05:00
  • 522233c4a2 Tune concurrency settings: 100 workers, 100 batch size, 100 buffer primal 2026-01-29 23:33:57 -05:00
  • 516848e529 Revise domain status flow: skip uses takedown, add drop for permanent deletion primal 2026-01-29 23:18:17 -05:00
  • 43916c8042 Exclude skip status domains from default API listing primal 2026-01-29 22:34:24 -05:00
  • edce82f1af Skip bare TLDs during domain import primal 2026-01-29 22:28:09 -05:00
  • 1066f42189 Refactor large Go files into focused modules primal 2026-01-29 22:25:02 -05:00
  • 3999e96f26 Dashboard UI overhaul: inline feed details, TLD filtering, status improvements primal 2026-01-29 20:51:05 -05:00
  • 5908a8c03e Full cleanup when revoking feed publish status primal 2026-01-29 13:37:02 -05:00
  • a5af4e14c3 Simplify: auto-deny all domains starting with digits primal 2026-01-29 13:32:42 -05:00
  • 2386d551fc Auto-deny all-digit domains, whitelist 1440.news primal 2026-01-29 13:27:48 -05:00
  • 897ae66e81 Fix NULL handling for nullable integer columns in getFeed primal 2026-01-29 13:20:18 -05:00
  • ad78c1a4c0 Add JSON Feed support primal 2026-01-29 13:16:50 -05:00
  • 798f79bfe9 Auto-deny feeds that are not RSS or Atom type primal 2026-01-29 13:13:22 -05:00
  • f7535a277f Add sort toggle for domain list (A-Z vs feed count) primal 2026-01-29 13:09:31 -05:00
  • a8c73bb540 Add TLD stats display when viewing a TLD primal 2026-01-29 13:04:06 -05:00
  • 8e4993d3c5 Optimize stats and TLD queries for performance primal 2026-01-29 13:00:13 -05:00
  • f780c493c2 Enable all TLDs for import and auto-deny spam domains primal 2026-01-29 12:49:52 -05:00
  • 254b751799 Add rich text links, language filter, and domain deny feature primal 2026-01-29 12:36:58 -05:00
  • 283a221efd Use url.1440.news for shorter URLs (28 chars vs 32) primal 2026-01-28 22:52:45 -05:00
  • 1ab45033cd Use app.1440.news domain for short URLs primal 2026-01-28 22:49:04 -05:00
  • 94d64373ed Add URL shortener for link tracking and shorter posts primal 2026-01-28 22:43:42 -05:00
  • f4f80e91cc Add enclosure support for podcast/media items primal 2026-01-28 22:36:28 -05:00
  • 1609220a27 Limit handle subdomain to 18 chars (PDS restriction) primal 2026-01-28 22:22:59 -05:00
  • 1f092c87e9 Add automatic image resize for blobs exceeding size limit primal 2026-01-28 22:10:10 -05:00
  • 959abf06c0 Enable .com domain import from vertices.txt.gz primal 2026-01-28 21:59:14 -05:00
  • c54005b5ba Upgrade BBC images from 240px to 800px for better quality primal 2026-01-28 21:49:11 -05:00
  • 5975df6771 Use PubDate for TID/rkey generation for consistent ordering primal 2026-01-28 21:32:07 -05:00
  • bce6c93242 Use original publication date for post createdAt primal 2026-01-28 21:29:44 -05:00
  • a1f02cd0bc Fix image embeds and rkey collisions primal 2026-01-28 21:24:35 -05:00
  • 4e4e8c939a Add favicon as profile picture for feed accounts primal 2026-01-28 21:05:50 -05:00
  • 9a43b69b4b Add profile refresh on startup to backfill feed URLs primal 2026-01-28 21:03:10 -05:00
  • 9ecf0f700d Add feed URL to profile description primal 2026-01-28 20:56:09 -05:00
  • 39714858e5 Fix subdomain length limit to match AT Protocol spec primal 2026-01-28 20:53:42 -05:00