Skip to content

cli

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

cli

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
src
src
 
 
tests
tests
 
 
CHANGELOG.md
CHANGELOG.md
 
 
README.md
README.md
 
 
package.json
package.json
 
 
tsup.config.ts
tsup.config.ts
 
 
vitest.config.ts
vitest.config.ts
 
 

README.md

@crawlith/cli 🚀

Deterministic crawl intelligence engine for serious SEO analysis.

@crawlith/cli is the command-line interface for Crawlith, a modular SEO crawling engine built for depth, accuracy, and professional analysis. It goes beyond simple "broken link checking" by calculating PageRank, HITS scores, detecting content clusters, and analyzing transport-layer security.

✨ Features

  • 🌐 Deep Crawling: Multi-threaded, robots.txt compliant crawler with depth and page limits.
  • 🧠 Graph Intelligence: Calculates PageRank, Hub/Authority scores (HITS), and maps internal link structures.
  • 🧩 Content Clustering: Detects duplicate or near-duplicate content clusters using advanced hashing.
  • 🔒 Security Probing: Deep inspection of SSL/TLS configurations and HTTP transport layer.
  • 📈 Advanced Scoring: Unified SEO signals for on-page structure, thin content, and crawl efficiency.
  • 🖥️ Dashboard Integration: Instantly launch a local web UI to visualize complex crawl data.

📦 Installation

To use Crawlith globally on your system:

npm install -g @crawlith/cli
# or
pnpm add -g @crawlith/cli

Or run it instantly without installation using npx:

npx crawlith --help

🛠 Usage

1. Crawl a Website

Build a full link graph and SEO metrics for a domain.

crawlith crawl https://example.com --limit 1000 --depth 10

2. Analyze a Single Page

Perfect for quick on-page SEO audits and content structure checks.

crawlith page https://example.com/blog/seo-guide

3. Start the UI Dashboard

Visualize your crawl snapshots in a beautiful, interactive interface.

crawlith ui

4. Probe Security

Inspect transport-layer headers, SSL/TLS status, and HTTP/2 support.

crawlith probe https://example.com

5. List Tracked Sites

View all sites currently stored in your local intelligence database.

crawlith sites

📊 Why Crawlith CLI?

Traditional crawlers give you a flat list of errors. Crawlith treats your website as a graph, allowing you to identify:

  • Orphan Pages: Automatic detection of pages that have no internal inbound links.
  • Authority Sinks: Identify pages that capture authority but fail to distribute it effectively.
  • Link Roles: Understand which pages act as "Hubs" (navigational) vs "Authorities" (content-rich).
  • Deterministic Analysis: Reproducible crawls that ensure your metrics are consistent over time.

⌨️ Command Options

Option Description
-l, --limit <n> Maximum number of pages to crawl
-d, --depth <n> Maximum click depth from the start URL
--export [type] Export results to json, markdown, csv, or html
--ignore-robots Bypass robots.txt directives (use responsibly)
--sitemap Explicitly use a sitemap for discovery

🛡 License

Apache License 2.0 © Crawlith