DHT Crawler

v1.0.3 — A magnet spider that discovers BitTorrent info from the DHT network. Built with Rust and ratatui. No database — discovered torrents are shown in the TUI only.

Features

• DHT spider: Runs a Mainline DHT node in server mode (via a patched mainline fork). Receives announcements from the network: incoming get_peers and announce_peer requests are captured and their info hashes are added for metadata lookup. Also runs active discovery (get_peers(random_id)) as a fallback. Magnets are fed to librqbit to fetch metadata.
• TUI: Title "DHT Crawler", status bar shows connection status (dot), and a Torrents table with columns: id, hash, name, size, date.
• Navigation: Arrow Up/Down to move selection; Enter opens a download menu for the selected torrent.
• Download menu: Shows torrent details (Name, Hash, Length, Pieces, Files, Creation Date, Comment) when available. Creation date and comment are parsed from full torrent metadata when present. Actions: Copy magnet URL (to clipboard via arboard), Save magnet to file, or Open in BitTorrent (opens in default client, e.g. Transmission).
• Enriched magnets: When copying or saving, the magnet URI includes optional dn= (display name) and xl= (exact length) so other clients get name and size before fetching metadata.

Build and run

cargo build --release
./target/release/dht_crawler

To see panics or errors (e.g. if discovery stops and you want to check for a crash), run from a terminal so stderr is visible:

./target/release/dht_crawler

Or cargo run --release from the project directory. Any panic or eprintln! output will appear in that terminal.

Configuration

On first run, DHT Crawler creates ~/.config/dht_crawler/ and ~/.config/dht_crawler/config.ini if they don't exist. The config file is INI-style:

[dht_crawler]
passive = true
active = true
max_concurrent_magnets = 500
find_node_interval_secs = 30
find_node_burst_size = 5

• passive — Capture info hashes from incoming DHT requests (get_peers / announce_peer). Set to false to disable.
• active — Run active discovery (get_peers(random_id) loop). Set to false to disable.
• max_concurrent_magnets — Max number of metadata fetches in flight at once (10–2000, default 500). Lower = less bandwidth and memory; higher = more concurrent lookups and faster discovery when many hashes are available.
• find_node_interval_secs — How often to run find_node to refresh/expand the DHT routing table, in seconds (5–600, default 30). Only used when active discovery is on.
• find_node_burst_size — Number of find_node calls per refresh (1–20, default 5). Each call uses a different random target ID, so a burst discovers nodes across more of the ID space and grows the routing table faster. Only used when active discovery is on.

You can run in passive-only mode (no active probing), active-only mode (no forwarding of incoming requests), or both (default).

Cache: The discovered torrent list is saved to ~/.config/dht_crawler/cache.json every 30 seconds and when you quit (q). On the next start, that list is loaded so you keep your previous discoveries. Cached hashes are not re-fetched for metadata.

Requirements

• Rust 1.70+
• A network that allows DHT (some networks block BitTorrent)
• Optional: Open the DHT port (see below) so the node can receive incoming announcements; otherwise only active discovery is used.

Bandwidth: DHT Crawler does not download torrent content (no files). It does download metadata (name, size, file list, etc.) for each discovered hash so it can show details in the TUI. With many hashes and long runs, metadata plus DHT traffic can add up. You can lower this by setting max_concurrent_magnets in config.ini (default 500).

Why did discovery stop at a low number? Two common causes: (1) Stuck slots — we only show torrents whose metadata was fetched. Many hashes never get metadata (dead torrents, no seeders). We treat those as failed and discard them after 5 minutes, and when at the concurrency cap we evict the oldest pending lookup so we always accept new magnets from the DHT (prioritise crawling). (2) Low passive traffic — if your DHT port isn't open or you're not in other nodes' routing tables, you only get the initial burst of passive traffic, then rely on active discovery (~10 random probes/sec), which finds new hashes slowly. Opening the DHT port (see below) helps. Other DHT nodes may treat you as a "bad" or slow node if we're slow to respond (e.g. while busy in active discovery), which can reduce incoming requests; running in passive-only mode with the port open can make the node more responsive.

Getting more passive traffic

Passive discovery is when other clients send get_peers / announce_peer to your node; you get their info hashes without probing. To get more of it:

1. Open the DHT port on your router and firewall so your node is reachable from the internet. The DHT typically uses UDP 6881 (or another port if your mainline build binds elsewhere). Forward that UDP port to the machine running DHT Crawler. If the port is closed, you only get traffic from nodes you contact first (e.g. bootstrap), so passive traffic stays low.
2. Run for a while — as you respond to DHT requests, other nodes put you in their routing tables. The longer you're reachable, the more often you'll be chosen for get_peers/announce_peer and the more hashes you'll see.

You can check if your port is open with a port-checker site (UDP) or another device on a different network.

How the DHT works (in this app)

We do not keep re-requesting nodes from the bootstrap list. Bootstrap nodes are used once at startup: we contact them to get an initial set of nodes and build a routing table. After that, all traffic (get_peers, find_node) uses that table, and responses from any node can add more nodes to the table. So we discover more of the network over time from normal traffic.

In addition, we periodically run a find_node burst (interval find_node_interval_secs, default 30s; find_node_burst_size calls per burst, default 5) so the DHT layer refreshes and expands the routing table. Each call uses a different random target ID, so we discover nodes across more of the network. That way we don't rely only on the initial bootstrap or on get_peers responses.

Keys

• ↑ / ↓: Select previous/next torrent (or move selection in download menu)
• Enter: Open download menu for selected torrent; in menu, run the selected action
• 1: (in menu) Copy magnet URL to clipboard
• 2: (in menu) Save magnet to file (current directory, .magnet extension)
• 3: (in menu) Open in BitTorrent (default client)
• Esc: (in menu) Close menu
• q: Quit