Skip to content

feat: remove dht #466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
deedy5 merged 1 commit into from
May 17, 2026
+4 −2,552
Merged

feat: remove dht #466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 2 additions & 18 deletions .github/workflows/python-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,29 +27,13 @@ jobs:
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install system dependencies (macOS)
if: matrix.os == 'macos-latest'
run: |
brew install gmp
# Add homebrew paths to compiler search paths
if [ "$(uname -m)" = "arm64" ]; then
echo "CFLAGS=-I/opt/homebrew/include $CFLAGS" >> $GITHUB_ENV
echo "LDFLAGS=-L/opt/homebrew/lib $LDFLAGS" >> $GITHUB_ENV
else
echo "CFLAGS=-I/usr/local/include $CFLAGS" >> $GITHUB_ENV
echo "LDFLAGS=-L/usr/local/lib $LDFLAGS" >> $GITHUB_ENV
fi


- name: Install dependencies
shell: bash
run: |
python -m pip install -U pip
if [ "${{ matrix.os }}" = "windows-latest" ]; then
# DHT dependencies require GMP which is difficult to install on Windows
python -m pip install .[dev,api,mcp]
else
python -m pip install .[dev,api,mcp,dht]
fi
python -m pip install .[dev,api,mcp]
- name: Ruff
run: |
ruff check .
Expand Down
104 changes: 0 additions & 104 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ A metasearch library that aggregates results from diverse web search services.
* [Install](#install)
* [CLI version](#cli-version)
* [API Server](#api-server)
* [DHT Network (BETA)](#dht-network-beta)
* [MCP Server](#mcp-server)
* [Engines](#engines)
* [DDGS class](#ddgs-class)
Expand Down Expand Up @@ -83,109 +82,6 @@ chmod +x start_api.sh
[Go To TOP](#TOP)
___

## DHT Network (BETA)

DDGS includes an optional peer-to-peer distributed cache network. Search results are shared anonymously between users, drastically reducing rate limits and latency for everyone.

This is the only metasearch engine that can actually get faster the more people use it.

---

### Why this exists

Every DDGS user currently hits search engines independently. Everyone gets rate limited individually. Everyone waits 1-2 seconds for exactly the same results.

The DHT network fixes this:
- ✅ **90% faster** repeated queries (50ms instead of 1-2s)
- ✅ **No rate limits** for common queries
- ✅ **Zero infrastructure cost** - runs entirely on user machines
- ✅ **No central server** - cannot be blocked or shut down
- ✅ **Anonymous** - no one sees your queries, no logs

### How it works

When running:
1. Your node automatically discovers other DDGS users
2. All existing API endpoints automatically check the network first
3. If found, returns results immediately
4. If not, searches normally and shares the result anonymously
5. Every new user makes the network faster for everyone else

---

### Installation

```bash
# Install base DHT package
pip install -U ddgs[dht]

# Install required dependencies (works on Linux and macOS)
pip install coincurve@git+https://github.com/ofek/coincurve.git@7829b29c08ebb1cc80386a1cdaf8c2243c4ef5c5
pip install libp2p@git+https://github.com/libp2p/py-libp2p.git@0e88584c89377086883c6f5b26cd1a8052399be7

# macOS only: First install gmp
brew install gmp

# Windows: DHT is not supported. Use base package only.
```

When installed, DHT:
- Adds persistent local result cache
- Adds `/dht/*` endpoints to the API server automatically
- Starts full distributed network node when you run `ddgs api`
- Works fully transparently - all existing search methods automatically use cache

> **Platform Support**: DHT works on Linux and macOS. Windows is not currently supported due to libp2p dependencies.

### Running

1. Using terminal
```bash
ddgs api -d
```
2. Using DDGS initialization
```python
from ddgs import DDGS

ddgs = DDGS(api_url="http://localhost:4479", spawn_api=True)
```

You do **not** need to change any existing code. Any Python script using `DDGS().text()`, `images()`, etc. will automatically use both the local cache and the distributed DHT network.

DHT will automatically start in the background and begin participating in the network.

### DHT API Endpoints

| Endpoint | Method | Description |
|---|---|---|
| `/dht/cache` | GET | Get cached results |
| `/dht/cache` | POST | Store results to cache |
| `/dht/cache` | DELETE | Invalidate cached results |
| `/dht/status` | GET | DHT service status and metrics |
| `/dht/peers` | GET | List connected peers |
| `/dht/peers/detailed` | GET | Detailed peer information |
| `/dht/map` | GET | Network graph view |
| `/dht/metrics` | GET | Prometheus format metrics |

---

### Beta Status

This is working beta software. All functionality is complete, but network performance will improve as more users join:

| Network Size | Performance |
|--------------|-------------|
| < 50 nodes | Marginal - expect timeouts |
| 50-200 nodes | Good - >95% success rate |
| >200 nodes | Excellent - near perfect |

Results use eventual consistency and may take 1-5 minutes to propagate.

Please report any issues here: https://github.com/deedy5/ddgs/issues

[Go To TOP](#TOP)
___

## MCP Server

- **Install**
Expand Down
11 changes: 1 addition & 10 deletions ddgs/api_server/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""DDGS API server.

This module provides the FastAPI application for the DDGS REST API
and the distributed network cache service.
This module provides the FastAPI application for the DDGS REST API.
"""

__all__: list[str] = []
Expand All @@ -13,11 +12,3 @@
except ImportError:
# API dependencies not installed
pass

try:
from ddgs.api_server.dht_service import get_dht_service

__all__ += ["get_dht_service"]
except ImportError:
# DHT dependencies not installed
pass
181 changes: 0 additions & 181 deletions ddgs/api_server/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import Response
from pydantic import BaseModel, Field

from ddgs import DDGS
Expand Down Expand Up @@ -450,186 +449,6 @@ async def extract_content_get(url: str, fmt: str = "text_markdown") -> dict[str,
raise HTTPException(status_code=500, detail=f"Extraction failed: {e!s}") from e


# Optional DHT Endpoints - only added if DHT dependencies are installed
try:
from pydantic import BaseModel, Field

class CacheRequest(BaseModel):
"""Request model for cache operations."""

query: str = Field(..., description="Search query")
results: list[dict[str, Any]] = Field(..., description="Search results to cache")
category: str = Field("text", description="Search category")

class DhtStatusResponse(BaseModel):
"""Response model for DHT status."""

running: bool
connected: bool
peer_id: str | None
port: int | None
cache_size: int
cache_count: int
peer_count: int
routing_table_size: int
metrics: dict[str, Any]

@app.get("/dht/cache", response_model=SearchResponse)
async def get_cached(query: str, category: str = "text") -> SearchResponse:
"""Get cached results from DHT."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
results = dht.get_cached(query, category)
if results is None:
raise HTTPException(status_code=404, detail="Not found in cache")
return SearchResponse(results=results)

@app.post("/dht/cache", status_code=201)
async def cache_results(request: CacheRequest) -> dict[str, str]:
"""Cache results to DHT."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
dht.cache(request.query, request.results, request.category)
return {"status": "ok"}

@app.delete("/dht/cache", status_code=204)
async def invalidate_cache(query: str, category: str = "text") -> None:
"""Invalidate cached results."""
from ddgs.api_server import get_dht_service # noqa: PLC0415
from ddgs.dht.types import compute_query_hash # noqa: PLC0415

dht = get_dht_service()
query_hash = compute_query_hash(query, category)
if dht._cache:
dht._cache.delete(query_hash)

@app.get("/dht/status", response_model=DhtStatusResponse)
async def dht_status() -> DhtStatusResponse:
"""Get DHT service status."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
status = dht.get_status()
return DhtStatusResponse(**status)

@app.get("/dht/peers", response_model=list[str])
async def dht_peers() -> list[str]:
"""Get list of connected DHT peers."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
return dht.get_peers()

@app.get("/dht/peers/detailed", response_model=list[dict[str, Any]])
async def dht_peers_detailed() -> list[dict[str, Any]]:
"""Get detailed information about all connected peers."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
return dht._dht.get_neighbors() if dht._dht else []

@app.get("/dht/map")
async def dht_map() -> dict[str, Any]:
"""Get local DHT network map view as graph structure.

Each node only sees its own routing table neighborhood.
No global crawling, no gossip, zero additional network traffic.
"""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
if not dht._dht:
return {"nodes": [], "edges": [], "total_peers_estimated": 0, "local_view_size": 0, "bucket_depth": 0}

peers = dht._dht.get_neighbors()

# Build graph
nodes = []
edges = []

# Add self node
nodes.append(
{
"id": dht._dht.peer_id,
"type": "self",
"connections": len(peers),
}
)

# Add neighbor nodes
for peer in peers:
nodes.append(
{
"id": peer["peer_id"],
"type": "peer",
"xor_distance": peer["xor_distance"],
"latency_ms": peer["latency_ms"],
}
)
edges.append(
{
"source": dht._dht.peer_id,
"target": peer["peer_id"],
"latency_ms": peer["latency_ms"],
}
)

# Network size estimation using Kademlia math
# For 256 bit ID space: estimated_size = 2^(average_bucket_depth)
bucket_depth = next((i for i, count in enumerate(dht._dht.kbucket_distribution) if count == 0), 255)
estimated_size = 2**bucket_depth

return {
"nodes": nodes,
"edges": edges,
"total_peers_estimated": estimated_size,
"local_view_size": len(nodes),
"bucket_depth": bucket_depth,
}

@app.get("/dht/metrics")
async def dht_metrics() -> Response:
"""Get DHT metrics in Prometheus format."""
from ddgs.api_server import get_dht_service # noqa: PLC0415

dht = get_dht_service()
status = dht.get_status()

metrics = [
"# HELP ddgs_dht_running Whether DHT service is running",
"# TYPE ddgs_dht_running gauge",
f"ddgs_dht_running {1 if status['running'] else 0}",
"# HELP ddgs_dht_connected_peers Number of currently connected peers",
"# TYPE ddgs_dht_connected_peers gauge",
f"ddgs_dht_connected_peers {status.get('peer_count', 0)}",
"# HELP ddgs_dht_cache_entries Number of entries in local cache",
"# TYPE ddgs_dht_cache_entries gauge",
f"ddgs_dht_cache_entries {status['cache_count']}",
"# HELP ddgs_dht_cache_size_bytes Size of local cache in bytes",
"# TYPE ddgs_dht_cache_size_bytes gauge",
f"ddgs_dht_cache_size_bytes {status['cache_size']}",
"# HELP ddgs_dht_query_success_rate DHT query success rate (0-1)",
"# TYPE ddgs_dht_query_success_rate gauge",
f"ddgs_dht_query_success_rate {status['metrics'].get('query_success_rate', 1.0)}",
"# HELP ddgs_dht_average_query_latency_ms Average DHT query latency in milliseconds",
"# TYPE ddgs_dht_average_query_latency_ms gauge",
f"ddgs_dht_average_query_latency_ms {status['metrics'].get('average_query_latency_ms', 0.0)}",
"# HELP ddgs_dht_routing_table_size Number of entries in DHT routing table",
"# TYPE ddgs_dht_routing_table_size gauge",
f"ddgs_dht_routing_table_size {status.get('routing_table_size', 0)}",
]

return Response("\n".join(metrics), media_type="text/plain")

logger.info("DHT endpoints enabled - distributed cache available")

except ImportError:
# DHT dependencies not installed - silently skip adding endpoints
pass


if __name__ == "__main__":
import uvicorn

Expand Down
Loading
Loading