RustyPin - DCPP Wire Protocol

RustyPin is a Rust implementation of the Distributed Content Preservation Protocol (DCPP) wire protocol. DCPP enables decentralized, collective preservation of digital content across distributed networks.

The protocol's key insight is that every participant has data they care about preserving. By identifying "collections" of related content and requiring participants to store complete collections (not just their individual items), DCPP transform self-interest into collective benefit.

Overview

This library provides:

Wire Protocol - Full envelope message framing with CRC32C integrity checks
Message Types - All DCPP message types (HELLO, ANNOUNCE, MANIFEST, etc.)
Cryptographic Signing - Ed25519 signatures with canonical CBOR encoding
CID Verification - IPFS-compatible CIDv1 content addressing
Storage Backend - File-system storage with content verification
Rate Limiting - Configurable rate limiters for API protection
State Machine - Protocol state management for nodes and collections

Optional features add:

libp2p Networking - Real P2P with Kademlia DHT and GossipSub
BitTorrent - BEP 52 compliant hybrid v1+v2 torrents for data transfer
Metrics - Prometheus metrics endpoint
Ingest API - HTTP API for content ingestion
Encryption - AES-256-GCM collection encryption (at-rest/ingest/wire scopes)

Collection Encryption

Collection encryption is opt-in and scoped. Default scope is ingest with AES-256-GCM. At-rest encryption is configurable. Key material defaults to environment or secret-file loading.

Feature flags:

encryption enables AES-256-GCM helpers and storage integration.
aws-kms adds AWS KMS decrypt support for encrypted data keys.

Key sources (base64-encoded 32-byte key):

Environment variable: DCPP_ENCRYPTION_KEY_B64
Secret file path: DCPP_ENCRYPTION_KEY_FILE
AWS KMS: provide a base64 ciphertext blob and optional region

Daemon flags (requires --features encryption):

--encryption enables encryption.
--encryption-scope at-rest|ingest|wire selects scope.
--encryption-at-rest or --no-encryption-at-rest toggles storage encryption.
--encryption-key-env VAR uses a custom env var.
--encryption-key-file PATH uses a key file.
--encryption-kms-key B64 with optional --encryption-kms-region REGION (requires --features aws-kms). Scope behavior:
ingest encrypts content before storage and manifest operations; stored data is encrypted only if at-rest is enabled.
at-rest encrypts stored content only.
wire is reserved for future transport-level encryption.

Example (env key + ingest scope, at-rest on):

export DCPP_ENCRYPTION_KEY_B64="$(openssl rand -base64 32)"
cargo run --features "full encryption" --bin dcpp_daemon -- \
  --encryption --encryption-scope ingest --encryption-at-rest -c test:collection

DCPP Architecture Diagrams

This document provides a visual overview of the module layout, feature flag boundaries, and the message-handling data flow.

Module Dependencies + Feature Boundaries

flowchart TD
    subgraph Core["Core Modules (always on)"]
        Types["types"]
        Error["error"]
        Framing["framing"]
        Messages["messages"]
        Signing["signing"]
        Manifest["manifest / manifest_verify"]
        Sharding["sharding"]
        Peer["peer / connection"]
        Storage["storage"]
        CID["cid_verify"]
        State["state_machine"]
        Validation["validation"]
        RateLimit["rate_limit"]
    end

    subgraph Libp2pFeature["Feature: libp2p-host"]
        Libp2pReal["libp2p_real"]
        DHT["dht"]
    end

    subgraph BitTorrentFeature["Feature: bittorrent"]
        BitTorrent["bittorrent / bittorrent_real"]
    end

    subgraph IngestFeature["Feature: ingest-api"]
        Ingest["ingest (http_server, ingest_worker)"]
    end

    subgraph MetricsFeature["Feature: metrics"]
        Metrics["metrics"]
    end

    subgraph EncryptionFeature["Feature: encryption"]
        Encryption["encryption"]
    end

    subgraph StructuredLoggingFeature["Feature: structured-logging"]
        StructuredLogging["tracing + tracing-subscriber"]
    end

    subgraph TestUtilsFeature["Feature: test-utils"]
        Libp2pHost["libp2p_host (test-only)"]
        E2E["e2e_tests"]
        Load["load_tests"]
    end

    Messages --> Signing
    Messages --> Manifest
    Manifest --> CID
    Storage --> CID
    State --> Messages
    State --> Peer
    State --> Storage
    State --> Manifest

    Libp2pReal --> Framing
    Libp2pReal --> Messages
    Libp2pReal --> DHT

    BitTorrent --> Storage
    BitTorrent --> Manifest

    Ingest --> Storage
    Ingest --> Manifest
    Ingest --> CID

    Metrics --> State
    Metrics --> Storage

    Storage --> Encryption

    StructuredLogging --> State
    StructuredLogging --> Libp2pReal
    StructuredLogging --> Ingest

Message Handling Data Flow

flowchart LR
    Net["Network I/O (libp2p_real or TCP)"] --> FrameIn["Profile1 decode (framing)"]
    FrameIn --> MsgDecode["Message::from_cbor (messages)"]
    MsgDecode --> Verify["Signature + validation"]
    Verify --> StateUpdate["state_machine + peer"]
    StateUpdate --> StorageOps["storage + manifest updates"]
    StorageOps --> Response["Message::to_cbor + Profile1 encode"]
    Response --> NetOut["Network I/O"]

    Verify --> Audit["security_audit (tests)"]
    StateUpdate --> DHTOps["dht (optional)"]
    StorageOps --> BitTorrent["bittorrent_real (optional)"]

Rust Node Components (System Level)

flowchart TD
    Daemon["dcpp_daemon (bin)"]
    Config["Config and CLI"]
    StateMachine["State Machine"]
    Libp2p["libp2p Host"]
    DHT["Kademlia DHT"]
    PubSub["GossipSub"]
    RR["Request Response"]
    Messages["Wire Messages"]
    Signing["Signing and Verification"]
    Storage["Storage"]
    Manifest["Manifest and Validation"]
    BitTorrent["BitTorrent Backend"]
    Ingest["Ingest API"]
    Metrics["Metrics and Logs"]

    Daemon --> Config
    Daemon --> StateMachine
    Daemon --> Libp2p
    Daemon --> BitTorrent
    Daemon --> Storage
    Daemon --> Ingest
    Daemon --> Metrics

    Libp2p --> DHT
    Libp2p --> PubSub
    Libp2p --> RR
    RR --> Messages
    Messages --> Signing
    Messages --> Manifest
    Manifest --> Storage
    BitTorrent --> Storage

Swarm Visual

flowchart TD
    subgraph Swarm["libp2p Swarm"]
        Transport["Transport<br/>TCP and QUIC"]
        Noise["Noise Encryption"]
        Mux["Yamux Multiplexing"]
        Behaviour["DCPPBehaviour"]
        Kademlia["Kademlia DHT"]
        GossipSub["GossipSub PubSub"]
        Identify["Identify"]
        DCPP["DCPP Request Response"]
    end

    Transport --> Noise --> Mux --> Behaviour
    Behaviour --> Kademlia
    Behaviour --> GossipSub
    Behaviour --> Identify
    Behaviour --> DCPP

    KademliaBenefit["Benefit: decentralized peer discovery and provider lookup"]
    GossipSubBenefit["Benefit: fast collection-wide announcements"]
    IdentifyBenefit["Benefit: peer metadata and address exchange"]
    DCPPBenefit["Benefit: direct request-response control messages"]

    Kademlia -.-> KademliaBenefit
    GossipSub -.-> GossipSubBenefit
    Identify -.-> IdentifyBenefit
    DCPP -.-> DCPPBenefit

    TransportBenefit["Benefit: resilient connectivity across diverse networks"]
    NoiseBenefit["Benefit: authenticated encryption for all streams"]
    MuxBenefit["Benefit: multiple logical streams over one connection"]

    Transport -.-> TransportBenefit
    Noise -.-> NoiseBenefit
    Mux -.-> MuxBenefit

Installation

Add to your Cargo.toml:

[dependencies]
dcpp_wire_protocol = { git = "https://github.com/quirk0o/dcpp-wire-protocol" }

Or with specific features:

[dependencies]
dcpp_wire_protocol = { git = "https://github.com/quirk0o/dcpp-wire-protocol", features = ["full"] }

Quick Start

Creating and Signing Messages

use dcpp_wire_protocol::prelude::*;

// Generate a keypair for signing
let keypair = KeyPair::generate();

// Create a HELLO message
let hello = HelloMessage::new(
    keypair.peer_id(),
    vec![Capability::Guardian, Capability::Seeder],
    vec!["eth:0xBC4CA0EdC24".to_string()],
);

// Serialize to CBOR
let message = Message::Hello(hello);
let payload = message.to_cbor().unwrap();

// Frame with Profile1 (production framing)
let framed = Profile1::encode(MessageType::Hello, &payload).unwrap();

Verifying Content with CIDs

use dcpp_wire_protocol::cid_verify::{compute_cid, verify_cid, Multicodec};

// Compute a CID for content
let content = b"Hello, DCPP!";
let cid = compute_cid(content, Multicodec::Raw);

// Verify content matches CID
assert!(verify_cid(&cid, content).unwrap());

Using the Storage Backend

use dcpp_wire_protocol::storage::{FileSystemStorage, StorageBackend};

// Create storage backend
let storage = FileSystemStorage::new("/path/to/data").unwrap();

// Store content with CID verification
let cid = storage.store_verified("my-collection", None, b"content").unwrap();

// Retrieve and verify
let data = storage.retrieve_verified("my-collection", &cid).unwrap();

Rate Limiting

use dcpp_wire_protocol::rate_limit::{RateLimiter, RateLimitConfig};

// Create rate limiter: 100 requests per 60 seconds
let limiter = RateLimiter::new(RateLimitConfig::new(100, 60));

// Check if request is allowed
if limiter.check("client-ip").is_allowed() {
    // Process request
}

Feature Flags

Feature	Description	Use Case
`default`	Core protocol only	Minimal footprint, message parsing
`libp2p-host`	P2P networking with libp2p	Full network participation
`bittorrent`	BitTorrent data plane	Large file transfers
`bootstrap-discovery`	DNS/HTTP bootstrap	Node discovery
`ingest-api`	HTTP content ingestion API	Adding content to nodes
`metrics`	Prometheus metrics	Monitoring and alerting
`structured-logging`	JSON log format	Production logging
`full`	All production features	Complete functionality

Building with Features

# Core only (default)
cargo build

# With P2P networking
cargo build --features libp2p-host

# With BitTorrent support
cargo build --features bittorrent

# All features
cargo build --features full

Module Overview

Module	Description
`framing`	Wire protocol framing (Profile1 for production)
`messages`	DCPP message types and serialization
`signing`	Ed25519 signing and verification
`types`	Protocol constants, enums, and error codes
`storage`	Content storage backends
`cid_verify`	CIDv1 computation and verification
`manifest`	Collection manifests and items
`sharding`	Shard assignment algorithms
`peer`	Peer management and ranking
`state_machine`	Protocol state transitions
`rate_limit`	Request rate limiting
`validation`	Message validation utilities

Protocol Messages

DCPP defines these message types:

Type	Code	Description
HELLO	0x0001	Peer introduction and capability exchange
ANNOUNCE	0x0002	Collection availability announcement
GET_MANIFEST	0x0003	Request collection manifest
MANIFEST	0x0004	Collection manifest response
GET_PEERS	0x0005	Request peers for collection
PEERS	0x0006	Peer list response
HEALTH_PROBE	0x0007	Verify peer has content
HEALTH_RESPONSE	0x0008	Health probe response
GOODBYE	0x0009	Graceful disconnect
ERROR	0x00FF	Error response

Architecture

                    ┌─────────────────────────────────────────┐
                    │            Application Layer            │
                    │  (Your code using dcpp_wire_protocol)   │
                    └─────────────────┬───────────────────────┘
                                      │
                    ┌─────────────────▼───────────────────────┐
                    │           Protocol Layer                │
                    │  messages, signing, validation          │
                    └─────────────────┬───────────────────────┘
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         │                            │                            │
┌────────▼────────┐         ┌────────▼────────┐         ┌────────▼────────┐
│   Framing       │         │   Storage       │         │   Network       │
│   Profile1      │         │   FileSystem    │         │   libp2p        │
│   CRC32C        │         │   CID verify    │         │   BitTorrent    │
└─────────────────┘         └─────────────────┘         └─────────────────┘

Examples

See the src/bin/ directory for example applications:

dcpp_daemon - TCP daemon for testing
dcpp_client - TCP client for testing
dcpp_cli - Command-line interface for content management
generate_test_vectors - Generate test vectors for interoperability

Testing

# Run all tests
cargo test

# Run with all features
cargo test --features full

# Run specific module tests
cargo test framing::tests
cargo test messages::tests
cargo test signing::tests

# Run benchmarks
cargo bench --features benchmarks

Fuzzing

Fuzz targets are provided for security testing:

cd fuzz
cargo +nightly fuzz run fuzz_cbor_message
cargo +nightly fuzz run fuzz_profile1_decode
cargo +nightly fuzz run fuzz_manifest
cargo +nightly fuzz run fuzz_path_sanitization

Security

This library includes security hardening:

Path traversal prevention - All filesystem paths are validated
Input sanitization - Collection IDs and CIDs are sanitized
Rate limiting - Configurable request limits
Signature verification - Ed25519 with canonical CBOR
CID verification - Content integrity via IPFS-compatible CIDs
Fuzz tested - All parsing code is fuzz tested

Report security issues to: [security contact]

Platform Support

Tested on:

Linux (x86_64, aarch64)
macOS (x86_64, aarch64)
Windows (x86_64)

Requirements

Rust 1.70 or higher
OpenSSL development headers (for libp2p/bittorrent features)

Platform-Specific Setup

Ubuntu/Debian:

sudo apt-get install pkg-config libssl-dev protobuf-compiler

macOS:

brew install openssl protobuf
export OPENSSL_DIR=$(brew --prefix openssl)

Documentation

Contributing

See CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see LICENSE for details.

Acknowledgments

DCPP builds on these excellent projects:

libp2p - Modular P2P networking
IPFS - Content-addressed storage concepts
BitTorrent - Proven data transfer protocol

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
.github		.github
benches		benches
docs		docs
examples		examples
fuzz		fuzz
src		src
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile.rust		Dockerfile.rust
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.example.yml		docker-compose.example.yml
rustfmt.toml		rustfmt.toml

Folders and files

Latest commit

History

Repository files navigation

RustyPin - DCPP Wire Protocol

Overview

Collection Encryption

DCPP Architecture Diagrams

Module Dependencies + Feature Boundaries

Message Handling Data Flow

Rust Node Components (System Level)

Swarm Visual

Installation

Quick Start

Creating and Signing Messages

Verifying Content with CIDs

Using the Storage Backend

Rate Limiting

Feature Flags

Building with Features

Module Overview

Protocol Messages

Architecture

Examples

Testing

Fuzzing

Security

Platform Support

Requirements

Platform-Specific Setup

Documentation

Contributing

License

Acknowledgments

About

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages