AI Crawler Priority Explorer

This project is a simple interactive demo that simulates how an AI crawler can prioritize newly discovered webpages for crawling.

What it does

The tool takes:

• a directed web graph (URL -> outlinks),
• precomputed PageRank scores,
• a value of k,

and returns the top k URLs to crawl using the scoring rule:

Priority Score = 0.7 * PageRank + 0.3 * Quality Score

Quality heuristic

• 1.0 for .edu or .gov domains
• 0.5 for other domains

Purpose

This demonstrates how PageRank can be used in modern AI crawling strategies to prioritize authoritative and high-quality pages for training data collection.