primal/crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enable .com domain import from vertices.txt.gz

Browse Source 
Filter imported domains to only .com TLD for now.
Re-enabled the import loop that was disabled for testing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
primal
2026-01-28 21:59:14 -05:00
parent c54005b5ba
commit 959abf06c0
2 changed files with 8 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
main.go
+3 -3
View File
@@ -32,10 +32,10 @@ func main() {
// Start all loops independently
fmt.Println("Starting import, crawl, check, and stats loops...")
// Import loop (background) - DISABLED for testing, using manual domains
// go crawler.ImportDomainsInBackground("vertices.txt.gz")
// Import loop (background) - imports .com domains from vertices.txt.gz
go crawler.ImportDomainsInBackground("vertices.txt.gz")
// Add only ycombinator domains for testing
// Add test domains (in addition to imported domains)
go crawler.ImportTestDomains([]string{
"news.ycombinator.com",
"ycombinator.com",