🐦 Flock

Get Started • Quick Start • Architecture • API Reference • Configuration

Overview

Flock is a fully-typed Python framework for building systems that run across multiple machines. It bundles the components a distributed system normally needs — Raft consensus, leader election, cluster membership, a message bus, scheduling, and security — into one importable package, so you wire up services with dependency injection instead of standing up separate infrastructure.

It's built around 43 focused subsystems (flock.consensus, flock.scheduler, flock.security, flock.workflow, and others), each independently testable and typed under mypy --strict.


Language	Python 3.11+
Core dependency footprint	`pydantic`, `structlog`, `msgpack`
Consensus model	Raft (leader election, log replication, snapshots)
Transport	Pluggable — TCP built in
Install	`pip install flock-p2p`

What's New in Flock v1.2.0

Flock v1.2.0 introduces Performance & Observability, bringing a complete performance monitoring, profiling, and regression detection suite designed to analyze, monitor, and optimize your distributed clusters passively:

⚡ Performance Foundation (Speed Measurement): Easily measure how fast your code runs. Use timing decorators (@time_execution) to track execution times and run tests (BenchmarkEngine) to see how much throughput your servers can handle.
🔍 Runtime Profiling (Resource Analysis): See exactly where your app is spending its time and memory. It identifies slow parts of your code (hotspots) and tracks memory allocation deltas without slowing down your production environment.
📉 Regression Detection (Slowdown Prevention): Save a benchmark as a "baseline" when your app is running perfectly. Flock will automatically compare future runs against this baseline to warn you if a new code update makes the app slower.
🎯 Performance Optimization (Smart Suggestions): Get automated recommendations on what to fix. The engine scans your performance data and highlights critical bottlenecks ranked by priority (Critical, High, Medium, Low).
📊 Performance Monitoring (Real-time Health): View a live health status (Healthy, Degraded, Critical) of your cluster. It watches system metrics in real time and raises warnings if things slow down.
📋 Performance Reporting (Release Scorecards): Generate a simple performance scorecard (grades A to F) and readiness certificate to confirm if your app is ready and safe to release to users.

The Story Behind Flock

💡 A single computer is no longer enough.

Whether it's traffic outgrowing one server, data too big for one machine, or uptime that can't depend on a single point of failure — the fix is always the same: run your software across multiple computers working together as one.

That's distributed computing. It quietly powers Netflix, Google, WhatsApp, and Uber. But here's what most tutorials leave out:

"Building a distributed system from scratch is one of the hardest things in software engineering."

🩹 Fault Tolerance

_{What happens when a server crashes mid-op?}

👑 Leader Election

_{Who's in charge, with no single source of truth?}

🔄 Consistency

_{How do 5 machines agree on the same data?}

🔒 Security

_{How do you stop traffic interception?}

📊 Observability

_{How do you catch failures before users do?}

These problems have occupied computer scientists for decades. Google and Amazon threw years of engineering effort at solving them internally — most teams don't have that runway.

That's why Flock exists — one pip install for infrastructure that used to take enterprise teams years to build.

📖 Read the full story

Flock was born from a simple frustration: developers shouldn't have to spend months building infrastructure before they can start building their actual product — especially when the Python ecosystem had no single, cohesive package that solved this.

The tools that solve consensus, replication, security, and observability have always existed — but scattered across research papers, expensive enterprise platforms, or locked inside the infrastructure teams of large tech companies. Nothing brought them together for the average Python developer.

Flock closes that gap: open-source, fully typed, and built to give any developer the same caliber of distributed-systems infrastructure that used to be exclusive to the biggest engineering teams in the world.

What Problem Does Flock Solve?

Flock provides a complete distributed systems stack in Python — mapping familiar real-world problems to concrete solutions:

Real-World Problem	Computer Science Term	Flock Solution
Who is in charge?	Leader Election	`ConsensusService` (Raft algorithm)
Stay online when a server dies	Fault Tolerance	Automatic failover + replication
All servers agree on the same data	Consensus	Raft log replication + state machine
Prevent data loss on crashes	Durability	Write-Ahead Log (WAL) + snapshots
Find other servers automatically	Service Discovery	`DiscoveryService`
Secure communication	mTLS / Zero-Trust	`SecurityService`
Know when things go wrong	Observability	Metrics, tracing, alerts, dashboards
Scale up when traffic spikes	Autoscaling	AI-powered `OrchestratorService`
Run jobs across many servers	Distributed Scheduling	`SchedulerService` + `PlacementEngine`
Recover from disasters	Disaster Recovery	`RecoveryService` with backups + PITR

🍳 Prefer an analogy? The Restaurant Kitchen Problem

Imagine running a restaurant. On a quiet Tuesday, one chef handles everything. On a Friday night with 200 customers, that one chef becomes a bottleneck — you need multiple chefs working together, which raises new problems:

Who's the head chef, and who takes over if they go home sick? → leader election
A chef drops a dish mid-prep — who picks it up where it was left off? → fault tolerance
How do two chefs avoid cooking the same order twice? → consensus
How do you stop the dishwasher from accessing the safe? → security & RBAC
How do you know which chef is overloaded, in real time? → observability

Flock solves all of these — not for kitchens, but for any system that needs to run across multiple servers.

Who Is Flock For?

Flock scales from a student reading source code for the first time to an enterprise team running hundreds of clusters.

🎓 Students & Learners

👨‍💻 Developers & Side Projects

🏢 Startups & Small Teams

🏭 Enterprise

🔬 Researchers & Data Scientists

🎓 Students & Learners

See a real, working implementation of Raft consensus, leader election, distributed state machines, and service meshes — not just theory. Every subsystem is documented, strictly typed, and built to textbook standards, so the source code doubles as a reference implementation.

pip install flock-p2p
python -c "from flock.consensus import ConsensusService; help(ConsensusService)"

👨‍💻 Individual Developers & Side Projects

Skip weeks of reading consensus papers and debugging race conditions — install Flock and have a working distributed foundation in an afternoon.

Use cases: personal projects outgrowing a single VPS · home lab clusters (Raspberry Pi farms) · distributed data pipelines · multiplayer game server backends · distributed web scrapers

🏢 Startups & Small Engineering Teams

A team of 2–5 engineers doesn't need to become distributed-systems experts to run production infrastructure. Consensus, replication, security, and observability are already solved.

Use cases: multi-region SaaS deployment · distributed task queues with guaranteed execution · real-time multi-node data processing · highly available APIs with automatic failover

🏭 Enterprise & Large Organizations

Built-in support for multi-cloud federation, policy-as-code governance, compliance auditing, fleet management, RBAC, secrets vaulting, and intrusion detection — capabilities most companies otherwise build in-house over years.

Use cases: multi-cloud infrastructure across AWS/GCP/Azure · compliance auditing (HIPAA, SOC2) · fleet management across hundreds of nodes · disaster recovery with point-in-time restore · policy-enforced workload governance

🔬 Researchers & Data Scientists

Distribute an ML training job across multiple GPUs or machines, or build a data pipeline that processes terabytes of data — Flock provides the distributed execution substrate so you can focus on the algorithm, not the infrastructure.

How Does Flock Help You?

The Traditional Approach (Without Flock)

Building the same distributed foundation from scratch typically looks like this:

Timeline	What You're Building	The Hard Part
Month 1–2	A consensus algorithm (Raft or Paxos)	Debugging election edge cases, split-brain scenarios
Month 3	A membership registry	Heartbeats, node join/leave, network partitions
Month 4	A message bus	Protocol design, serialization, versioning, backward compatibility
Month 5–6	Security	TLS, authentication, authorization, secrets management
Month 7–8	Observability	Metrics collection, distributed tracing, alerting
Month 9+	Finally — your actual product	—

⏱️ ~9 months before writing a single line of business logic — for a skilled team.

The Flock Approach

pip install flock-p2p   # 30 seconds

# A few minutes to a working distributed system
from flock.consensus import ConsensusService
from flock.events.bus import EventBus
from flock.messaging.bus import MessageBus

consensus = ConsensusService("node-1", membership, message_bus, event_bus)
await consensus.start()

# You now have Raft consensus, leader election, log replication,
# fault tolerance, and event publishing. Start building your product.

⏱️ Time to a working distributed foundation: under an hour.

The Difference, By the Numbers

Metric	Build Yourself	With Flock
Time to first working cluster	Months	Minutes
Infrastructure code you maintain	Thousands of lines	Already written
Test coverage for edge cases	You write it all	663 tests included
Consensus algorithm correctness	Your responsibility	Implemented Raft, unit-tested
Security posture	Your responsibility	Built-in mTLS, RBAC, secrets vault
Ongoing maintenance	On you	Shared with the OSS community

✨ Features

CORE PLATFORM

⚡ Full Raft Consensus

_{Leader election, log replication, state machine, term management, log compaction}

🌐 Cluster Membership

_{Node discovery, heartbeat health monitoring, live membership registry}

🔄 Distributed State Machine

_{Consistent replicated state with snapshotting and WAL}

📡 Transport-Independent Messaging

_{Pluggable transport (TCP built-in); typed message routing}

🗂️ DataGrid

_{In-memory distributed KV store — replication, partitioning, failover}

EXECUTION & SCHEDULING

🧠 AI-Powered Orchestration

_{ML-based placement, predictive autoscaling, anomaly detection}

📋 Workflow Engine

_{DAG workflows with checkpointing, parallelism, failure recovery}

⏰ Advanced Scheduler

_{Cron, event-driven, deadline-aware scheduling}

🎯 Constraint-Aware Placement

_{CPU/memory/affinity/anti-affinity placement engine}

ENTERPRISE INFRASTRUCTURE

🔒 Zero-Trust Security _{mTLS, RBAC, certificate management, credential rotation, intrusion detection}	🛡️ Secrets Vault _{Encrypted storage, key rotation, compliance auditing}	📊 Full Observability _{Tracing, structured logging, metrics, alerts, profiling, dashboards}
🌍 Multi-Cloud Federation _{Cross-region federation, latency-aware routing, trust handshakes}	🏛️ Control Plane _{Fleet management, cluster enrollment, org governance}	📜 Policy-as-Code _{Declarative policy compiler, rule engine, compliance enforcement}
💾 Disaster Recovery _{Automated snapshots, incremental backups, PITR restore}	🛒 Plugin Marketplace _{Package registry, dependency resolution, sandboxed execution}	🚀 Deployment Automation _{Docker/K8s manifests, rolling updates, rollback}

DEVELOPER EXPERIENCE

✅ 635 Tests

_{Full regression coverage, 42 subsystems}

🔤 Fully Typed

_{mypy --strict clean, 390 source files}

📦 Single Install

_{pip install flock-p2p}

🖥️ TUI Dashboard

_{Keyboard-driven CLI dashboard (flock)}

🐍 Python 3.11+

_{Modern Python, no legacy baggage}

📦 Installation

pip install flock-p2p

🖥️ Interactive CLI Dashboard

Once installed, you can immediately launch the interactive TUI onboarding dashboard by running:

flock

This starts a keyboard-driven, in-place terminal dashboard that lets you run diagnostics, launch local cluster simulations, view examples, check version details, and access documentation.

Requirements

Dependency	Version	Purpose
Python	≥ 3.11	Runtime
`pydantic`	≥ 2.0.0	Data validation and models
`structlog`	≥ 23.1.0	Structured logging
`msgpack`	≥ 1.0.0	Binary serialization

Development Installation

For contributing or running the test suite locally:

1. Clone the repository

git clone https://github.com/Ashish6298/Flock.git

2. Enter the project directory

cd Flock

3. Install in editable mode with dev dependencies

pip install -e .[dev]

_{Dev extras include: pytest · pytest-asyncio · mypy · black · ruff}

🚀 Quick Start • Three examples. Each one runnable in under a minute.

🧱 Start a Cluster

_{Raft consensus, leader election}

🏛️ Join the Control Plane

_{Fleet enrollment & governance}

📋 Submit a Workflow

_{DAG-based task execution}

1️⃣ Start a Single-Node Cluster

_{Spin up Raft consensus, elect a leader, and commit your first log entry.}

Show code

import asyncio
from unittest.mock import MagicMock
from flock.events.bus import EventBus
from flock.messaging.bus import MessageBus
from flock.consensus import ConsensusService
from flock.cluster.registry import MembershipRegistry
from flock.cluster.models import NodeMember, ClusterMemberStatus

async def main() -> None:
    # Wire up core infrastructure
    event_bus = EventBus()
    transport = MagicMock()       # Replace with TCPTransport in production
    serializer = MagicMock()      # Replace with JsonSerializer in production
    message_bus = MessageBus(transport, serializer)

    # Build membership registry
    registry = MembershipRegistry()
    registry.register(NodeMember(
        node_id="node-1",
        host="127.0.0.1",
        port=9000,
        status=ClusterMemberStatus.ACTIVE,
    ))

    # Start Raft consensus
    consensus = ConsensusService(
        node_id="node-1",
        membership=registry,
        message_bus=message_bus,
        event_bus=event_bus,
    )
    await consensus.start()

    # Submit a command (only succeeds if this node is the leader)
    if consensus.is_leader():
        entry = await consensus.submit_command(b"hello-world")
        print(f"Committed log entry: {entry}")

    await consensus.stop()

asyncio.run(main())

2️⃣ Register a Cluster with the Control Plane

_{Enroll a cluster into a fleet with labels, versioning, and active feature tracking.}

Show code

import asyncio
from flock.events.bus import EventBus
from flock.messaging.bus import MessageBus
from flock.controlplane.service import ControlPlaneService
from flock.controlplane.models import EnrolledCluster
from unittest.mock import MagicMock

async def main() -> None:
    event_bus = EventBus()
    message_bus = MessageBus(MagicMock(), MagicMock())

    cp = ControlPlaneService(
        node_id="coordinator",
        message_bus=message_bus,
        event_bus=event_bus,
    )
    await cp.start()

    cluster = EnrolledCluster(
        cluster_id="cluster-east",
        fleet_id="fleet-prod",
        name="US East Compute",
        version="1.0.0",
        labels={"region": "us-east-1", "tier": "prod"},
        features_active=["Consensus", "Security", "Observability"],
        last_seen=0.0,
    )
    cp.coordinator.clusters.enroll_cluster(cluster)
    print(f"Cluster enrolled: {cluster.name}")

    await cp.stop()

asyncio.run(main())

3️⃣ Submit a Distributed Workflow

_{Define a DAG with dependencies and submit it for checkpointed execution.}

Show code

import asyncio
from flock.workflow.service import WorkflowService
from flock.workflow.models import WorkflowDefinition, WorkflowStep
from flock.events.bus import EventBus
from flock.messaging.bus import MessageBus
from flock.storage.backend import StorageBackend
from unittest.mock import MagicMock

async def main() -> None:
    storage = StorageBackend()
    event_bus = EventBus()
    message_bus = MessageBus(MagicMock(), MagicMock())

    workflow_svc = WorkflowService(
        node_id="node-1",
        storage_backend=storage,
        message_bus=message_bus,
        event_bus=event_bus,
    )
    await workflow_svc.start()

    # Define a two-step DAG workflow
    wf = WorkflowDefinition(
        workflow_id="wf-001",
        name="data-pipeline",
        steps=[
            WorkflowStep(step_id="ingest", name="Ingest Data", dependencies=[]),
            WorkflowStep(step_id="transform", name="Transform", dependencies=["ingest"]),
        ],
    )
    await workflow_svc.submit(wf)
    await workflow_svc.stop()

asyncio.run(main())

🏗️ Architecture

Flock is a distributed, event-driven platform built around a strong consistency core. The architecture connects application interfaces, coordination services, cluster nodes, and persistence layers through a unified event graph.

✨ Design Principles

A great architecture isn't defined by frameworks—it's defined by the principles that guide every design decision.

These core principles ensure the system remains modular, scalable, maintainable, and easy to evolve as new requirements emerge.

Transport Independence

Build once. Change the transport anytime.

Business logic communicates exclusively through the MessageBus, never with TCP, UDP, gRPC, WebSockets, or any other transport directly.

The transport layer becomes an implementation detail—not a dependency.

✅ Why it matters

Switch communication protocols without touching business logic
Use an in-memory transport for lightning-fast tests
Keep networking concerns isolated from the domain
Add new transports with minimal effort

💡 Guiding Principle Infrastructure should be replaceable without affecting domain logic.

Event-Driven Architecture

Components react to events—not to each other.

Every meaningful state transition publishes a strongly typed event through the EventBus.

Any number of subscribers can respond independently, allowing the system to grow without creating tight coupling.

🔄 How it works

Service A
    │
    ▼
Publishes Event
    │
    ▼
┌──────────────┬──────────────┬──────────────┐
│ Subscriber A │ Subscriber B │ Subscriber C │
└──────────────┴──────────────┴──────────────┘

✅ Why it matters

Loose coupling between components
Add new features without modifying existing services
Natural support for asynchronous workflows
Independent testing of event handlers

💡 Guiding Principle Publish events instead of invoking services directly.

Immutable Domain Models

State is created—not modified.

Every domain object is implemented using:

@dataclass(frozen=True)

Instead of changing existing objects, each state transition creates a new immutable instance representing the updated state.

🔄 Example

Order(status="Pending")
          │
          ▼
Order(status="Shipped")

The original object remains unchanged.

✅ Why it matters

Predictable behavior
Safer concurrent execution
Easier debugging
Fewer unintended side effects

💡 Guiding Principle Domain objects represent values—not mutable state.

Thread Safety

Concurrency is a design requirement—not an afterthought.

Whenever shared mutable state exists, access is synchronized using:

threading.RLock()

This guarantees correctness even under concurrent workloads.

✅ Why it matters

Prevent race conditions
Preserve consistent application state
Safe multi-threaded execution
Reliable behavior under load

💡 Guiding Principle Shared mutable state must always be synchronized.

SOLID & Dependency Inversion

Depend on abstractions—not implementations.

Services receive all dependencies through constructor injection, rather than creating infrastructure internally.

This clean separation keeps business logic independent from implementation details.

🔄 Dependency Flow

Business Service
       │
       ▼
Interface (Abstraction)
       │
       ▼
Concrete Implementation

✅ Why it matters

Highly testable services
Easily replace implementations
Clear separation of concerns
Long-term maintainability

💡 Guiding Principle Services should never construct their own infrastructure.

Principles at a Glance

Principle	Purpose
Transport Independence	Decouple communication from transport protocols.
Event-Driven Architecture	Enable collaboration through published events instead of direct calls.
Immutable Domain Models	Create predictable, side-effect-free state transitions.
Thread Safety	Ensure correctness under concurrent execution.
SOLID & Dependency Inversion	Keep services modular, testable, and infrastructure-agnostic.

How They Work Together

                +--------------------+
                |   Business Logic   |
                +---------+----------+
                          │
          depends only on abstractions
                          │
                          ▼
                 +----------------+
                 |  Message Bus   |
                 +-------+--------+
                         │
        publishes events │
                         ▼
                 +----------------+
                 |   Event Bus    |
                 +-------+--------+
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     Service A      Service B      Service C
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                Immutable Domain Models
                         │
                Thread-safe execution

The Bigger Picture

Each principle reinforces the others:

Transport Independence keeps infrastructure replaceable.
Events keep services loosely coupled.
Immutability makes state predictable.
Thread Safety ensures correctness under concurrency.
Dependency Inversion keeps the architecture clean and testable.

Together, they create a system that is:

Modular • Scalable • Resilient • Testable • Extensible • Future-Proof

💙 Good architecture isn't about making today's code work—it's about making tomorrow's changes effortless.

🧩 Flock Subsystems

Flock is composed of 42 focused subsystems.

Each subsystem has a single responsibility and can evolve independently while collaborating through shared interfaces and the event-driven architecture.

🏗️ Core Distributed Systems

The foundation of the cluster.

Subsystem	Module	Responsibility
🗳️ Consensus	`flock.consensus`	Raft consensus, leader election, log replication, replicated state machine
🏛️ Cluster	`flock.cluster`	Node membership, registry, health management
❤️ Heartbeat	`flock.heartbeat`	Failure detection and peer liveness monitoring
🔍 Discovery	`flock.discovery`	Service registration and peer discovery
🌐 Transport	`flock.transport`	Pluggable network transport (TCP and future transports)
📦 Protocol	`flock.protocol`	Typed protocol messages and `MessageType` definitions

💬 Communication Layer

Everything related to messaging and event distribution.

Subsystem	Module	Responsibility
📨 Messaging	`flock.messaging`	MessageBus, routing, middleware, handlers
⚡ Events	`flock.events`	EventBus for publish/subscribe communication
🌊 Streaming	`flock.streaming`	Distributed event streaming with backpressure

💾 Data & Storage

Persistent and distributed data management.

Subsystem	Module	Responsibility
🗄️ DataGrid	`flock.datagrid`	Distributed key-value storage with partitioning
💽 Storage	`flock.storage`	WAL, persistence, recovery
📸 Snapshot	`flock.snapshot`	Snapshotting, compaction, replication
🔄 State Machine	`flock.statemachine`	Generic replicated state machine
📑 Serialization	`flock.serialization`	JSON and MessagePack serialization

⚙️ Runtime & Scheduling

Task execution and workload management.

Subsystem	Module	Responsibility
🚀 Runtime	`flock.runtime`	Execution context and runtime environment
⏱️ Scheduler	`flock.scheduler`	Priority-based task scheduling
📅 Scheduling	`flock.scheduling`	Cron, triggers, deadlines, recurring jobs
📋 Results	`flock.results`	Result collection, serialization, registry

☁️ Distributed Compute

Cluster-wide execution and orchestration.

Subsystem	Module	Responsibility
🤖 Orchestrator	`flock.orchestrator`	AI-assisted workload orchestration and autoscaling
📍 Placement	`flock.placement`	Constraint-aware workload placement
🧠 Resources	`flock.resources`	CPU, memory, balancing, capacity planning
🔀 Workflow	`flock.workflow`	DAG workflow engine with checkpointing
⚡ Functions	`flock.functions`	Serverless function execution

🌐 Networking & Services

Application networking capabilities.

Subsystem	Module	Responsibility
🕸️ Mesh	`flock.mesh`	Service mesh, routing, circuit breaking, load balancing
🔍 Query	`flock.query`	Distributed query parser, planner, optimizer

📊 Operations & Monitoring

Visibility into the running cluster.

Subsystem	Module	Responsibility
📈 Observability	`flock.observability`	Metrics, tracing, profiling, alerts, logging
⚡ Performance	`flock.performance`	High-resolution timing, profiling, regression checks, and engineering reporting
📺 Dashboard	`flock.dashboard`	Real-time operational dashboard

🔐 Security & Governance

Securing and governing the platform.

Subsystem	Module	Responsibility
🛡️ Security	`flock.security`	Zero-Trust security, RBAC, vault, cryptography
📜 Policy	`flock.policy`	Policy-as-Code, compliance, rule engine
🎛️ Control Plane	`flock.controlplane`	Fleet management and cluster governance

🤖 Intelligence

Machine learning and predictive capabilities.

Subsystem	Module	Responsibility
🧠 AI	`flock.ai`	Prediction, anomaly detection, adaptive learning

🌍 Enterprise & Cloud

Large-scale deployment capabilities.

Subsystem	Module	Responsibility
🌐 Federation	`flock.federation`	Multi-region and multi-cloud federation
🚑 Recovery	`flock.recovery`	Disaster recovery, backup, point-in-time restore
🚀 Deployment	`flock.deployment`	Kubernetes and Docker deployment automation
📦 Release	`flock.release`	Release lifecycle and production readiness

🔌 Extensibility

Customize and extend Flock.

Subsystem	Module	Responsibility
🛒 Marketplace	`flock.marketplace`	Plugin registry, dependency resolution, sandboxing
🧩 Plugins	`flock.plugins`	Plugin loading, lifecycle, isolation
🔗 Interfaces	`flock.interfaces`	Shared interfaces, abstract base classes, protocols

🛠️ Developer Experience

Developer-facing tools.

Subsystem	Module	Responsibility
⚙️ Config	`flock.config`	Configuration management
🌐 API	`flock.api`	HTTP gateway and REST API
💻 CLI	`flock.cli`	Command-line interface

📦 Architecture Map

                              FLOCK
                                │
 ┌─────────────────────────────────────────────────────────────┐
 │                  Core Distributed Systems                   │
 └─────────────────────────────────────────────────────────────┘
            │
            ├── 💬 Communication
            ├── 💾 Data & Storage
            ├── ⚙️ Runtime & Scheduling
            ├── ☁️ Distributed Compute
            ├── 🌐 Networking
            ├── 📊 Observability
            ├── 🔐 Security & Governance
            ├── 🤖 AI
            ├── 🌍 Enterprise
            ├── 🔌 Extensibility
            └── 🛠️ Developer Experience

📈 At a Glance

Category	Modules
🏗️ Core Distributed Systems	6
💬 Communication	3
💾 Data & Storage	5
⚙️ Runtime & Scheduling	4
☁️ Distributed Compute	5
🌐 Networking	2
📊 Operations	3
🔐 Security & Governance	3
🤖 Intelligence	1
🌍 Enterprise	4
🔌 Extensibility	3
🛠️ Developer Experience	3

43 focused subsystems working together to provide a modular, scalable, distributed application platform.

📚 API Reference

The Flock API is asynchronous, strongly typed, and modular. Expand any section below to explore its API, methods, and examples.

Core Types — Immutable domain models that represent nodes, tasks, and shared entities used throughout the Flock ecosystem.

NodeInfo

from flock import NodeInfo

node = NodeInfo(
    node_id="node-1",
    host="10.0.0.1",
    port=9000,
    metadata={"zone": "us-east-1a"},
)

TaskSpec

from flock import TaskSpec

task = TaskSpec.create(
    "process_batch",
    dataset="sales_q4",
    limit=1000,
)

TaskStatus

class TaskStatus(str, Enum):
    PENDING = "PENDING"
    RUNNING = "RUNNING"
    COMPLETED = "COMPLETED"
    FAILED = "FAILED"
    CANCELLED = "CANCELLED"

ConsensusService — Implements the Raft consensus algorithm for leader election, replicated logs, and distributed state consistency.

Create a service

from flock.consensus import ConsensusService

service = ConsensusService(...)

Start

await service.start()
await service.stop()

Cluster State

service.is_leader()
service.current_term
service.leader_id

Submit Command

entry = await service.submit_command(
    b"payload"
)

Published Events

Event	Description
`consensus.leader.elected`	Leader elected
`consensus.term.changed`	Current term changed
`consensus.log.committed`	Log committed
`consensus.replication.failed`	Replication failure

SecurityService — Provides authentication, authorization, cryptographic operations, identity management, and secure secret storage.

Initialize

from flock.security.service import SecurityService

service = SecurityService(...)

Authenticate

service.authentication_engine.authenticate(
    token,
    peer_id,
)

Authorize

service.authorization_engine.authorize(
    identity,
    "write",
    "resource",
)

Secrets

service.secrets_manager.store(
    "db_password",
    b"secret",
)

secret = service.secrets_manager.retrieve(
    "db_password",
)

WorkflowService — Executes distributed DAG workflows with dependency management, checkpointing, and fault recovery.

Create Service

from flock.workflow.service import WorkflowService

service = WorkflowService(...)

Submit Workflow

await service.submit(workflow)

EventBus — Enables loosely coupled publish/subscribe communication between independent services and subsystems.

Subscribe

from flock.events.bus import EventBus

bus = EventBus()

bus.subscribe(
    "consensus.leader.elected",
    callback,
)

Publish

bus.publish(
    "consensus.leader.elected",
    payload,
)

MessageBus — Provides transport-independent, strongly typed messaging with routing, middleware, and asynchronous delivery.

Register Handler

from flock.messaging.bus import MessageBus
from flock.protocol.packet import MessageType

@bus.handler(MessageType.VOTE_REQUEST)
async def handle_vote(message):
    ...

Send Message

await bus.send(
    peer_id="node-2",
    message_type=MessageType.HEARTBEAT,
    payload=b"...",
)

API Design Principles

Every public API in Flock follows the same architectural philosophy:

Immutable Models — Domain objects are value types that never mutate.
Async First — Network and I/O operations use async / await.
Event Driven — Components collaborate through published events.
Strongly Typed — APIs expose explicit types, enums, and contracts.
Transport Independent — Business logic is isolated from networking protocols.
Dependency Injection — Infrastructure is supplied externally for flexibility and testability.

The API is intentionally consistent across every subsystem, making it easy to learn, extend, and scale distributed applications with Flock.

⚙️ Configuration

Flock follows an explicit configuration model. Services are configured entirely through constructor arguments and dependency injection—there are no global configuration files, hidden state, or singleton objects.

This approach keeps every subsystem modular, testable, and easy to replace while making dependencies explicit.

✨ Configuration Philosophy

🧩 Dependency Injection — Services receive their dependencies through constructors.
🚫 No Global State — No global configuration objects or singletons.
🔒 Explicit Dependencies — Every required component is visible in the constructor.
🧪 Test Friendly — Infrastructure can easily be replaced with mocks or in-memory implementations.
🔄 Composable — Mix and match implementations without changing business logic.

🏗️ Typical Service Wiring — Initialize infrastructure, construct services, start them, and gracefully shut them down.

Create Infrastructure

event_bus = EventBus()
message_bus = MessageBus(
    transport,
    serializer,
)

Create Services

membership = MembershipRegistry()

consensus = ConsensusService(
    "node-1",
    membership,
    message_bus,
    event_bus,
)

security = SecurityService(
    "node-1",
    secret_key,
    identity,
    message_bus,
    event_bus,
)

workflow = WorkflowService(
    "node-1",
    storage,
    message_bus,
    event_bus,
)

Start Services

await consensus.start()
await security.start()
await workflow.start()

Application Runs

# Your application logic...

Graceful Shutdown

await workflow.stop()
await security.stop()
await consensus.stop()

⚙️ Configuration Parameters — Common constructor options available across core Flock services.

Service	Parameter	Default	Description
`ConsensusService`	`heartbeat_interval`	`0.15s`	Frequency of leader heartbeats sent to followers.
`ConsensusService`	`election_timeout_min`	`0.30s`	Minimum randomized election timeout before starting a new election.
`ConsensusService`	`election_timeout_max`	`0.60s`	Maximum randomized election timeout.
`SecurityService`	`secret_key`	Required	32-byte encryption key used for cryptographic operations.
`FederationService`	`region`	Required	Geographic region identifier for multi-region deployments.

💡 Best Practices — Recommended patterns for configuring production applications.

Recommended Guidelines

✅ Create shared infrastructure (EventBus, MessageBus) once and reuse it.
✅ Inject dependencies instead of creating them inside services.
✅ Start services in dependency order.
✅ Shut down services gracefully in reverse order.
✅ Keep configuration close to application startup.
✅ Use immutable configuration values whenever possible.

📌 Design Principles

Every Flock application follows the same startup lifecycle:

Create Infrastructure
        │
        ▼
Construct Services
        │
        ▼
Inject Dependencies
        │
        ▼
Start Services
        │
        ▼
Application Running
        │
        ▼
Graceful Shutdown

By making dependencies explicit rather than hidden, Flock applications remain predictable, maintainable, and straightforward to test at any scale.

🚀 Examples

Learn Flock by example. These practical snippets demonstrate common workflows—from creating your first cluster to federation, policy enforcement, and disaster recovery.

Whether you're just getting started or exploring advanced capabilities, the examples below provide a solid foundation for building distributed applications with Flock.

📚 Available Examples

Example	Description
🚀 Getting Started	Create your first Flock cluster and verify your installation.
🌍 Multi-Node Federation	Connect multiple clusters across regions into a unified federation.
📜 Policy-as-Code	Define and enforce runtime policies using the policy engine.
💾 Disaster Recovery	Create snapshots and restore cluster state after failures.

🚀 Getting Started — Build and run your first Flock application in just a few commands.

Clone the repository

git clone https://github.com/Ashish6298/Flock.git
cd Flock
pip install -e .

Run the example

python examples/getting_started.py

Expected Output

Successfully initialized Flock Cluster: US East compute under organization registries.

💡 Tip: This example verifies that your installation is working correctly and demonstrates the basic application startup lifecycle.

🌍 Multi-Node Federation — Connect multiple clusters into a single federated deployment spanning regions or cloud providers.

from flock.federation.service import FederationService
from flock.federation.models import FederatedCluster

federation = FederationService(
    node_id="gateway-node",
    message_bus=message_bus,
    event_bus=event_bus,
)

await federation.start()

remote = FederatedCluster(
    cluster_id="cluster-west",
    region="us-west-2",
    endpoint="https://west.internal:9443",
    trust_level="verified",
)

federation.registry.register(remote)

Federation enables geographically distributed clusters to communicate securely while maintaining independent operation.

📜 Policy-as-Code — Define security and operational policies that are evaluated automatically at runtime.

from flock.policy.service import PolicyService
from flock.policy.models import PolicyDefinition, PolicyRule

policy_svc = PolicyService(
    node_id="node-1",
    event_bus=event_bus,
)

policy = PolicyDefinition(
    policy_id="deny-root",
    name="Deny Root Privilege Tasks",
    rules=[
        PolicyRule(
            rule_id="r1",
            condition="task.user == 'root'",
            action="DENY",
            severity="CRITICAL",
        )
    ],
)

policy_svc.engine.load_policy(policy)

Policies can enforce security, compliance, scheduling constraints, or custom business rules without changing application code.

💾 Disaster Recovery — Protect your cluster by creating snapshots and restoring state after failures.

from flock.recovery.service import RecoveryService

recovery = RecoveryService(
    node_id="node-1",
    storage=storage,
    event_bus=event_bus,
)

await recovery.start()

# Create a snapshot
snapshot_id = await recovery.create_snapshot(
    label="pre-migration-backup"
)

# Restore a snapshot
await recovery.restore_snapshot(snapshot_id)

Snapshots capture a consistent view of cluster state, enabling reliable backup, migration, and disaster recovery workflows.

💡 Explore More

The examples/ directory contains additional sample applications covering different Flock subsystems and integration patterns. Each example is designed to be self-contained, making it easy to experiment with specific features or use them as a starting point for your own projects.

The fastest way to learn Flock is to run the examples, modify them, and build from there.

🧪 Testing

Flock is built with testability as a first-class design goal. The complete test suite validates all 42 subsystems, ensuring reliability across distributed components, communication layers, storage systems, and core services.

📊 Test Coverage Overview

Metric	Result
🧩 Subsystems Tested	42
✅ Total Tests	629
⚡ Execution Time	~10 seconds
🔍 Type Checking	`mypy --strict`
📁 Source Files Checked	390

▶️ Running Tests — Execute the complete Flock validation suite and verify system correctness.

Run Full Test Suite

python -m pytest tests/ -v

This runs all tests across every subsystem.

Run a Specific Subsystem

python -m pytest tests/test_consensus_service.py -v

Useful when developing or debugging a particular component.

Generate Coverage Report

Install coverage support:

pip install pytest-cov

Run tests with coverage:

python -m pytest tests/ \
    --cov=src/flock \
    --cov-report=html

The generated HTML report provides a detailed view of:

Covered modules
Missing lines
Coverage percentage
Test distribution

Run Strict Type Checking

mypy --strict src/

Flock uses strict static typing to catch interface errors before runtime.

📈 Test Results — Latest validation results from the complete test and type-checking pipeline.

Pytest Results

629 passed in ~10s

Type Checking Results

Success: no issues found in 390 source files

🧩 Testing Philosophy — How Flock maintains reliability across a distributed architecture.

Flock testing focuses on:

Area	Purpose
🏗️ Subsystem Isolation	Each component can be tested independently.
🔄 Integration Testing	Ensures services communicate correctly.
⚡ Async Testing	Validates concurrent workflows and event handling.
📨 Messaging Tests	Verifies transport and routing behavior.
🗳️ Consensus Testing	Validates leader election and replication logic.
💾 Recovery Testing	Ensures snapshots and restoration work correctly.
🔒 Security Testing	Validates authentication and authorization flows.

🚦 Development Workflow

A typical development cycle:

        Write Code
            │
            ▼
      Run Unit Tests
            │
            ▼
    Run Type Checking
            │
            ▼
   Validate Integration
            │
            ▼
       Submit Change

Every subsystem in Flock is continuously validated to keep the platform stable as new capabilities are added.

🗂️ Project Structure

Flock is organized as a modular distributed platform with 42 focused subsystems under the flock package. Each subsystem owns a specific responsibility while communicating through shared interfaces, events, and dependency injection.

🌳 Repository Overview

Flock/
│
├── 📦 src/
│   └── 🐦 flock/                         # Main package (42 subsystems)
│       │
│       ├── __init__.py                   # Public API exports
│       │                                  # FlockError, NodeInfo, TaskSpec, TaskStatus
│       │
│       ├── 🗳️ consensus/                 # Raft consensus engine
│       │                                  # Leader election, log replication,
│       │                                  # state machine management
│       │
│       ├── 🏢 cluster/                   # Cluster membership
│       │                                  # Node registry and cluster models
│       │
│       ├── 🔐 security/                  # Zero-Trust security framework
│       │                                  # Authentication, RBAC, cryptography
│       │
│       ├── 🔄 workflow/                  # Distributed DAG workflow engine
│       │
│       ├── 📊 observability/             # Monitoring infrastructure
│       │                                  # Metrics, tracing, profiling, alerts
│       │
│       ├── 🌍 federation/                # Multi-cloud federation layer
│       │                                  # Cross-region cluster communication
│       │
│       ├── 🎛️ controlplane/              # Fleet management and governance
│       │
│       ├── 📜 policy/                    # Policy-as-Code engine
│       │
│       ├── 💾 recovery/                  # Disaster recovery subsystem
│       │                                  # Snapshots and restoration
│       │
│       ├── 🧩 marketplace/               # Plugin registry and extensions
│       │
│       └── ...                           # 32 additional subsystems
│
├── 🧪 tests/                             # Automated test suite
│                                         # 211 test files, 629 tests
│
├── 🚀 examples/                          # Runnable examples and demos
│
├── 📚 docs/                              # Documentation and audit reports
│
├── ⚙️ .github/
│   └── workflows/
│       └── ci.yml                        # GitHub Actions CI pipeline
│
└── 📄 pyproject.toml                     # Package metadata,
                                          # dependencies, and tooling

🧩 Source Package Layout

🐦 src/flock — Core package containing all distributed system capabilities.

The flock package contains every subsystem required to run and extend the platform.

Public API

from flock import (
    FlockError,
    NodeInfo,
    TaskSpec,
    TaskStatus,
)

Core Subsystems

Module	Purpose
`consensus`	Raft consensus, leader election, replication
`cluster`	Membership and node management
`security`	Identity, authentication, authorization
`workflow`	DAG-based workflow execution
`observability`	Metrics, tracing, profiling
`federation`	Multi-region cluster federation
`controlplane`	Fleet management
`policy`	Policy evaluation and enforcement
`recovery`	Backup and restoration
`marketplace`	Plugin discovery and management

🧪 tests — Comprehensive validation suite covering every major subsystem.

The test suite validates:

✅ Core domain models
✅ Distributed communication
✅ Consensus behavior
✅ Event handling
✅ Storage and recovery
✅ Security workflows
✅ Service integrations

Current size:

211 test files
629 automated tests

🚀 examples — Runnable applications demonstrating Flock capabilities.

Examples provide practical entry points for:

Cluster initialization
Federation
Policy management
Workflow execution
Recovery operations

Each example is designed to be executed independently.

📚 docs — Documentation, architecture guides, and audit material.

Contains:

Architecture documentation
API references
Design principles
Operational guides
Audit reports

⚙️ CI/CD — Automated quality checks using GitHub Actions.

The CI pipeline validates:

🧪 Test execution
🔍 Type checking
📦 Package integrity
🚦 Build verification

Configuration:

.github/workflows/ci.yml

🏗️ Architecture Flow

                 Flock Platform
                       │
                       ▼
              ┌─────────────────┐
              │   src/flock     │
              └────────┬────────┘
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
   Core Services   Infrastructure   Extensions
        │              │              │
        ▼              ▼              ▼
 Consensus       Messaging       Plugins
 Workflow        Storage         Marketplace
 Security        Transport       Federation

The repository structure mirrors the architecture itself: small, focused subsystems connected through clear interfaces, making Flock easy to understand, test, and extend.

🚦 CI/CD Pipeline

Flock uses an automated GitHub Actions CI pipeline to maintain code quality, reliability, and consistency across all changes.

Every push and pull request targeting main automatically triggers the validation workflow.

🔄 Pipeline Overview

Developer Push / Pull Request
              │
              ▼
      GitHub Actions Runner
              │
              ▼
      Install Environment
              │
              ▼
       Type Validation
              │
              ▼
       Test Execution
              │
              ▼
        Merge Approved ✅

⚙️ CI Workflow Steps — Automated checks executed for every code change.

1️⃣ Setup Python Environment

- Set up Python 3.11

The pipeline runs against the supported Python runtime to ensure consistent behavior.

2️⃣ Install Dependencies

pip install -r requirements

Installs all required dependencies, including:

📦 msgpack — binary serialization support
🧪 pytest — testing framework
⚡ pytest-asyncio — async test support

3️⃣ Install Flock Package

pip install -e .

Installs Flock in editable mode so the CI environment tests the actual source package.

4️⃣ Strict Type Checking

mypy --strict src/

Flock uses zero-tolerance type checking to detect:

Missing type annotations
Invalid interfaces
Unsafe type usage
Contract violations

5️⃣ Full Regression Test Suite

python -m pytest tests/ -v

Runs the complete automated test suite:

629 tests
42 subsystems

✅ Quality Gates — Conditions required before changes can be merged.

Every pull request must successfully complete:

Check	Requirement
🐍 Python Environment	Python 3.11 setup
📦 Dependencies	All packages installed successfully
🔍 Type Safety	`mypy --strict` passes
🧪 Tests	All 629 tests pass
🚦 Pipeline Status	No CI failures

🛡️ Why CI Matters

The CI pipeline protects Flock by ensuring:

🔒 New changes do not break existing behavior
🧩 Subsystems remain compatible
🧪 Distributed components stay validated
📈 Code quality remains consistent
🚀 Releases are safer and predictable

Every change entering the main branch is automatically verified against Flock's type safety and regression standards.

🤝 Contributing

Contributions help Flock grow. Whether you're fixing bugs, improving documentation, adding features, or expanding subsystems, every contribution is welcome.

Before submitting changes, please review the full contribution guidelines in CONTRIBUTING.md.

🚀 Quick Start

Follow these steps to set up your development environment and submit your first contribution.

🛠️ Development Workflow — Fork, develop, validate, and submit your changes.

1️⃣ Fork & Clone

Create your fork and clone it locally:

git clone https://github.com/YOUR_USERNAME/Flock.git
cd Flock

2️⃣ Install Development Dependencies

Install Flock with all development tools:

pip install -e .[dev]

This installs:

🧪 Testing tools
🔍 Type checking tools
🛠️ Development utilities

3️⃣ Make Your Changes

When developing:

Follow existing architecture patterns
Keep changes focused
Add tests for new functionality
Maintain backward compatibility

4️⃣ Run Validation

Before opening a pull request, run the complete validation pipeline:

mypy --strict src/

python -m pytest tests/ -v

Your changes must pass:

✅ Strict type checking
✅ Full regression suite
✅ Existing subsystem tests

5️⃣ Submit Pull Request

Submit your pull request against:

main

Ensure the PR description explains:

What changed
Why the change is needed
How it was tested

📏 Code Standards — Development rules that keep Flock consistent, reliable, and maintainable.

All contributions must follow these standards:

🔍 Type Safety

All code must pass:

mypy --strict src/

Requirements:

✅ Complete type annotations
✅ No unchecked typing shortcuts
🚫 No # type: ignore exceptions

🧪 Testing Requirements

Every new feature must include tests.

Change	Requirement
New feature	Add corresponding tests
Bug fix	Add regression test
API change	Update API tests
Subsystem change	Validate affected components

🧩 Architecture Patterns

Follow Flock's core design principles:

🔒 Use immutable dataclasses
🏛️ Use dependency injection
⚡ Prefer event-driven communication
🚚 Keep infrastructure replaceable
🧩 Avoid unnecessary coupling

Example:

@dataclass(frozen=True)
class DomainModel:
    value: str

📝 Logging

Use structlog for all library logging.

✅ Recommended:

logger.info(
    "task_started",
    task_id=task_id,
)

❌ Avoid:

print("Task started")

Library code should never use print() for logging.

🧭 Contribution Checklist

Before submitting a pull request:

Check
📦 Dependencies installed
🧪 Tests added/updated
🔍 `mypy --strict` passes
🚫 No `type: ignore` added
🔒 Immutable models followed
🏛️ Dependency injection maintained
📝 Logging uses `structlog`

Great contributions are not only about adding code—they preserve the architecture, quality, and principles that make Flock scalable.

🔐 Security

Security is a core part of Flock's architecture. Vulnerability reports are handled through a responsible disclosure process to ensure issues can be investigated and resolved safely before public discussion.

🛡️ Reporting Security Issues

If you discover a potential security vulnerability, please do not open a public issue.

Instead, report it privately by following the process described in:

📄 SECURITY.md

🚨 Responsible Disclosure — How security reports are handled safely and efficiently.

When reporting a vulnerability, please include:

🔍 Description — Clear explanation of the issue
🧩 Affected Component — Subsystem, module, or service involved
📝 Reproduction Steps — Minimal steps to reproduce the behavior
💥 Impact Assessment — Potential security impact
🛠️ Suggested Fix — If you have recommendations

Providing detailed information helps the maintainers investigate and resolve the issue quickly.

🔒 Security Areas — Components covered by Flock's security model.

Security considerations include:

Area	Description
🔑 Authentication	Identity verification between nodes and services
🛡️ Authorization	Access control and permission enforcement
🔐 Cryptography	Secure communication and secret handling
🌐 Federation Security	Trust management between clusters
📜 Policy Enforcement	Runtime security and compliance rules
💾 Data Protection	Secure storage and recovery mechanisms

⚠️ Please Avoid

To protect users and maintainers:

❌ Do not publicly disclose unpatched vulnerabilities
❌ Do not include sensitive data in reports
❌ Do not test against systems without authorization
❌ Do not publish exploit details before a fix is available

Responsible disclosure helps keep the Flock ecosystem secure while giving maintainers the opportunity to protect all users.

📄 License

Flock is released under the MIT License, allowing developers to freely use, modify, distribute, and build upon the project while preserving the original license terms.

⚖️ MIT License

Flock is open source software licensed under the MIT License.

The full license text is available here:

📄 LICENSE

✨ What This Means — The permissions and responsibilities provided by the MIT License.

The MIT License allows you to:

Permission	Description
✅ Use	Use Flock for personal, commercial, or internal projects
✅ Modify	Change and adapt the source code
✅ Distribute	Share original or modified versions
✅ Integrate	Include Flock in larger applications

📌 License Requirements — Conditions that apply when using Flock.

When redistributing Flock or derivative works:

Include the original copyright notice
Include the MIT License text
Preserve license attribution

🌱 Open Source Commitment

Flock is built with the goal of being:

🧩 Accessible to developers and organizations
🔄 Easy to extend and integrate
🌍 Community-driven and transparent
🚀 Free to use across different environments

The MIT License keeps Flock flexible while ensuring proper attribution to the project and its contributors.

📚 Citation

If you use Flock in research, academic work, technical publications, or production systems, please cite the project to acknowledge the work and help others discover the platform.

📝 BibTeX Citation

Use the following BibTeX entry:

@software{flock2026,
  title   = {Flock: Enterprise-Grade Federated Distributed Computing Platform},
  author  = {Ashish6298},
  year    = {2026},
  url     = {https://github.com/Ashish6298/Flock},
  version = {1.2.0}
}

📖 Citation Guidelines — When and why to cite Flock.

Please cite Flock when you:

🔬 Use Flock as part of research or experiments
📄 Reference Flock in technical papers or articles
🏢 Deploy Flock in production systems
🧩 Build extensions or integrations based on Flock
📚 Include Flock in educational material

🌐 Repository

Source code and project information:

https://github.com/Ashish6298/Flock

Citing Flock helps recognize the project and supports continued development of the platform.

Enterprise-Grade Federated Distributed Computing Platform

Built with ❤️ for the distributed systems community

Flock provides a modular foundation for building resilient distributed systems with consensus, federation, workflows, security, observability, and scalable execution.

🌟 Explore Flock

🚀 Get Started	🏗️ Architecture	🤝 Contribute
Install & run examples	Explore 42 subsystems	Join development
Learn the API	Understand design principles	Submit improvements

🧩 Built Around

🗳️ Consensus     ⚡ Event Driven     🔐 Security
🌍 Federation    📊 Observability    🔄 Workflows
💾 Recovery      📨 Messaging        🧠 Intelligence

🌍 Community

Have ideas, improvements, or questions?

💬 Join discussions · 🐛 Report issues · 🚀 Share your projects built with Flock

📚 Resources

📦 PyPI • 💻 Source Code • 🐛 Bug Reports • 💬 Community Discussions

Built with ❤️ by the Flock community

_{Open source · Distributed by design · MIT Licensed}

Name		Name	Last commit message	Last commit date
Latest commit History 41 Commits
.github		.github
docs		docs
examples		examples
flock_p2p-1.1.1.dev0+g92a46d46d.d20260725		flock_p2p-1.1.1.dev0+g92a46d46d.d20260725
src/flock		src/flock
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CITATION.cff		CITATION.cff
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
ROADMAP.md		ROADMAP.md
SECURITY.md		SECURITY.md
flock-architecture.svg		flock-architecture.svg
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

🐦 Flock • A Distributed Computing Framework for Python

Overview

What's New in Flock v1.2.0

Table of Contents

The Story Behind Flock

💡 A single computer is no longer enough.

"Building a distributed system from scratch is one of the hardest things in software engineering."

What Problem Does Flock Solve?

Who Is Flock For?

How Does Flock Help You?

The Traditional Approach (Without Flock)

The Flock Approach

The Difference, By the Numbers

✨ Features

CORE PLATFORM

⚡ Full Raft Consensus

🌐 Cluster Membership

🔄 Distributed State Machine

📡 Transport-Independent Messaging

🗂️ DataGrid

EXECUTION & SCHEDULING

🧠 AI-Powered Orchestration

📋 Workflow Engine

⏰ Advanced Scheduler

🎯 Constraint-Aware Placement

ENTERPRISE INFRASTRUCTURE

🔒 Zero-Trust Security

🛡️ Secrets Vault

📊 Full Observability

🌍 Multi-Cloud Federation

🏛️ Control Plane

📜 Policy-as-Code

💾 Disaster Recovery

🛒 Plugin Marketplace

🚀 Deployment Automation

DEVELOPER EXPERIENCE

✅ 635 Tests

🔤 Fully Typed

📦 Single Install

🖥️ TUI Dashboard

🐍 Python 3.11+

📦 Installation

🖥️ Interactive CLI Dashboard

Requirements

Development Installation

🚀 Quick Start • Three examples. Each one runnable in under a minute.

🧱 Start a Cluster

🏛️ Join the Control Plane

📋 Submit a Workflow

1️⃣ Start a Single-Node Cluster

2️⃣ Register a Cluster with the Control Plane

3️⃣ Submit a Distributed Workflow

🏗️ Architecture

✨ Design Principles

Transport Independence

✅ Why it matters

Event-Driven Architecture

🔄 How it works

✅ Why it matters

Immutable Domain Models

🔄 Example

✅ Why it matters

Thread Safety

✅ Why it matters

SOLID & Dependency Inversion

🔄 Dependency Flow

✅ Why it matters

Principles at a Glance

How They Work Together

The Bigger Picture

🧩 Flock Subsystems

🏗️ Core Distributed Systems

💬 Communication Layer

💾 Data & Storage

⚙️ Runtime & Scheduling

☁️ Distributed Compute