sonic-firewall

Containerized stateful firewall for SONiC leaf switches, providing zone-based east-west microsegmentation over VXLAN EVPN fabrics.

Overview

sonic-firewall implements a two-tier forwarding model for inter-zone traffic:

• Slow path (CPU): New connections are evaluated by nftables/conntrack inside the docker-firewall container. Policy decisions (permit/deny) are made based on zone-to-zone rules.
• Fast path (ASIC): Established flows are offloaded to the switch ASIC as SAI ACL entries, forwarding at line rate without CPU involvement.

This architecture is modeled on SONiC's existing NAT subsystem (natsyncd + natorch), which already demonstrates the conntrack-to-Redis-to-SAI pattern.

 Management Host                          Leaf Switch (docker-firewall)
+------------------+                     +-----------------------------+
| fwcontroller     |  <── HTTP poll ──   | fwcontrollerd               |
| (Flask REST API) |  ── policy JSON ─>  | (writes CONFIG_DB)          |
| (SQLite storage) |                     |   ↓                         |
+------------------+                     | fwzonemgrd  fwpolicyd       |
      ^                                  | fwsyncd     fworch          |
      | HTTP                             | fwlogd                      |
+------------------+                     +-----------------------------+
| fwctl CLI        |                                     |
+------------------+                                     v
                                             AclOrch --> SAI --> ASIC

Each leaf switch runs the full per-switch pipeline inside docker-firewall:

  CONFIG_DB         +-------------------------------------------------+
  FW_ZONE --------->|  fwzonemgrd    fwpolicyd    fwsyncd    fworch   |
  FW_POLICY ------->|  (zones)       (nftables)   (conntrack) (ACL)   |
  FW_GLOBAL ------->|       |            |             |         |    |
                    |       v            v             v         v    |
                    |  STATE_DB     nftables      APP_DB     APP_DB   |
                    |  zone state   kernel rules  FW_FLOW    ACL_RULE |
                    |                                                  |
                    |  fwlogd (deny counter polling → log + STATE_DB)  |
                    +-------------------------------------------------+

Components

Daemons

Daemon	Language	Description
fwzonemgrd	Python	Zone manager. Reads FW_ZONE from CONFIG_DB, resolves the Zone→VNI→VLAN→IP chain using VXLAN and NEIGH tables, writes runtime zone membership to STATE_FW_ZONE_TABLE.
fwpolicyd	Python	Policy engine. Reads FW_POLICY from CONFIG_DB, compiles zone-to-zone rules into nftables rulesets with conntrack integration. Writes baseline deny ACLs to APP_DB.
fwsyncd	C++ / Python	Flow sync daemon. Monitors kernel conntrack events via NfNetlink, identifies inter-zone flows by querying zone membership, writes offload-eligible flows to APP_FW_FLOW_TABLE. Supports warm restart with conntrack dump + reconciliation.
fworch	C++ / Python	Firewall orchestrator. Consumes flow entries from APP_DB, writes ACL rules (forward + reverse per flow) to APP_ACL_RULE_TABLE for AclOrch. Manages TCAM capacity with LRU eviction, hitbit-based aging, circuit breaker anti-churn, /24 subnet aggregation, and warm restart flow restoration.
fwlogd	Python	Deny log daemon. Polls nftables deny counters every 5 seconds, tracks deltas per policy, writes structured JSON to /var/log/fw-denies.jsonl and a ring buffer (last 1000 entries) to STATE_FW_DENY_LOG_TABLE.
fwcontrollerd	Python	Leaf agent (opt-in). Polls the centralized controller for policy updates, writes zones/policies/global config to CONFIG_DB, triggering existing daemons. Caches last-known-good bundle for partition tolerance.

Centralized Management (Phase 5)

Component	Location	Description
fwcontroller	Management host	Flask REST API + Web UI with SQLite backend. Zone/policy/config CRUD, policy versioning with auto-snapshot, rollback, drift detection, staged rollout, telemetry aggregation. Web UI at /ui/ with HTMX live-refresh.
fwctl	Management host	Click CLI for controller API. Zone/policy/config management, version/rollback operations, drift/remediation, rollout lifecycle, telemetry display.

The controller uses a pull model: each leaf agent polls every 10 seconds with its current config_hash (SHA-256 of sorted policy JSON). The controller returns 304 if unchanged, or the full policy bundle if the hash differs. This is simpler than push (no server on the switch, no NAT/firewall traversal issues) and the agent controls its own poll rate.

Startup Order

Managed by supervisord with dependent_startup:

rsyslogd --> start.sh --> fwzonemgrd --> fwpolicyd --> fwsyncd --> fworch
                                                              --> restore_fw_state
                                                              --> fwlogd
                                                              --> fwcontrollerd (opt-in)

fwcontrollerd has autostart=false — existing standalone deployments are unaffected. Set FW_CONTROLLER_URL environment variable to enable.

CLI Commands

Configuration (config firewall ...):

config firewall enable
config firewall disable
config firewall zone add <name> --vni <vni_list> [--description <desc>]
config firewall zone del <name>
config firewall policy add <name> --from <zone> --to <zone> --action permit|deny
                          [--protocol tcp|udp|icmp] [--dst-port <ports>]
                          [--src-port <ports>] [--priority <N>] [--log]
                          [--description <desc>]
config firewall policy del <name>

Display (show firewall ...):

show firewall zones
show firewall policy [--zone <name>]
show firewall flows [--zone <name>] [--top N]
show firewall counters [--zone <name>]
show firewall tcam-usage
show firewall deny-log [--limit N]

Clear (clear firewall ...):

clear firewall counters
clear firewall flows
clear firewall deny-log

Controller CLI (fwctl)

Manages the centralized controller from the management host. Uses FW_CONTROLLER_URL env var or --controller flag (default: http://localhost:8643).

Zone/Policy/Config Management:

fwctl zone list                              # List all zones
fwctl zone add <name> --vni <vni_list>       # Create zone
fwctl zone del <name>                        # Delete zone
fwctl zone show <name>                       # Show zone details
fwctl policy list                            # List all policies
fwctl policy add <name> --from <z> --to <z> --action permit|deny
                        [--protocol tcp|udp|icmp] [--dst-port <ports>]
                        [--src-port <ports>] [--priority N] [--log]
fwctl policy del <name>                      # Delete policy
fwctl policy show <name>                     # Show policy details
fwctl config show                            # Show global config
fwctl config set <key> <value>               # Update global config

Operations:

fwctl status                                 # Controller health check
fwctl validate                               # Validate referential integrity
fwctl leaves                                 # List registered leaf switches
fwctl telemetry                              # Fabric-wide telemetry summary

Versioning & Rollback:

fwctl version list                           # List policy versions
fwctl version show <version>                 # Show version details
fwctl rollback <version>                     # Restore to version + force-sync all leaves

Drift Detection & Remediation:

fwctl drift                                  # Show drift status per leaf
fwctl remediate <leaf_id>                    # Force single leaf to re-sync
fwctl remediate --all                        # Force all leaves to re-sync

Staged Rollout:

fwctl rollout list                           # List rollouts
fwctl rollout create [--description <desc>]  # Create rollout (pending)
fwctl rollout show <id>                      # Show rollout details
fwctl rollout stage <id>                     # Begin staging (leaves download but don't apply)
fwctl rollout activate <id>                  # Activate (leaves apply staged bundle)
fwctl rollout abort <id>                     # Abort rollout

Web UI

The controller includes a server-rendered Web UI accessible at http://<host>:8643/ui/. Built with Flask + Jinja2, HTMX for partial page updates, Alpine.js for client-side interactions, and Pico CSS for classless styling. No Node.js, no npm, no build step — all JS/CSS is vendored as static files.

Page	URL	Description
Dashboard	/ui/	Fabric overview: status, zone/policy/leaf counts, active rollout, recent versions
Zones	/ui/zones	Zone CRUD with inline add form and delete confirmation
Policies	/ui/policies	Policy CRUD with zone dropdowns, action colors, priority ordering
Leaves	/ui/leaves	Leaf status table with auto-refresh (5s polling), per-leaf remediate
Config	/ui/config	Global config form with watermark validation
Versions	/ui/versions	Version history with per-version rollback
Rollouts	/ui/rollouts	Rollout lifecycle: create, stage, activate, abort
Drift	/ui/drift	Drift detection summary with per-leaf remediate (10s polling)

Controller REST API

The controller exposes a REST API on port 8643 (default). All request/response bodies are JSON.

Method	Path	Description
GET	/api/v1/health	Health check: status, zone/policy counts, config_hash
GET	/api/v1/zones	List all zones
POST	/api/v1/zones	Create zone (name, vni_list, description)
GET	/api/v1/zones/<name>	Get zone details
PUT	/api/v1/zones/<name>	Update zone fields
DELETE	/api/v1/zones/<name>	Delete zone (409 if referenced by policy)
GET	/api/v1/policies	List all policies
POST	/api/v1/policies	Create policy (name, src_zone, dst_zone, action, ...)
GET	/api/v1/policies/<name>	Get policy details
PUT	/api/v1/policies/<name>	Update policy fields
DELETE	/api/v1/policies/<name>	Delete policy
GET	/api/v1/config	Get global config
PUT	/api/v1/config	Update global config
GET	/api/v1/leaves	List registered leaf switches
GET	/api/v1/leaves/<leaf_id>	Get leaf details
GET	/api/v1/policy/current	Agent pull endpoint (query: leaf_id, config_hash, flow_count, offloaded_count, hostname). Returns 304 if up-to-date, 200 with full bundle otherwise.
GET	/api/v1/validate	Validate referential integrity
GET	/api/v1/versions	List policy versions (query: limit)
GET	/api/v1/versions/<id>	Get version metadata
GET	/api/v1/versions/<id>/bundle	Get version's full policy bundle
POST	/api/v1/rollback/<version>	Restore version + force-sync all leaves
GET	/api/v1/drift	Drift status per leaf (in_sync/drifted/stale)
POST	/api/v1/remediate/<leaf_id>	Force single leaf to re-sync
POST	/api/v1/remediate	Force all leaves to re-sync
GET	/api/v1/rollouts	List rollouts
POST	/api/v1/rollouts	Create rollout (description)
GET	/api/v1/rollouts/<id>	Get rollout details
POST	/api/v1/rollouts/<id>/stage	Begin staging
POST	/api/v1/rollouts/<id>/activate	Activate (apply staged bundle)
POST	/api/v1/rollouts/<id>/abort	Abort rollout
GET	/api/v1/telemetry/summary	Fabric-wide telemetry summary

Policy bundle format (returned by /api/v1/policy/current):

{
  "zones": {"web": {"vni_list": "1000,1001", "description": "Web tier"}, ...},
  "policies": {"web_to_db": {"src_zone": "web", "dst_zone": "database", "action": "permit", ...}, ...},
  "global_config": {"enabled": "true", "max_flows": "8192", ...},
  "config_hash": "a1b2c3d4e5f6...",
  "timestamp": 1700000000.0,
  "action": "stage",
  "rollout_id": 1
}

The action and rollout_id fields are only present during a staged rollout. When action is "stage", the leaf agent downloads but does not apply the bundle until the rollout is activated.

Redis Schema

All table names are defined in src/schema/fw_schema.h (C++) and src/schema/fw_schema.py (Python).

CONFIG_DB (persistent configuration):

Table	Key	Fields
FW_ZONE	zone_name	vni_list, description
FW_POLICY	policy_name	src_zone, dst_zone, action, protocol, src_port, dst_port, priority, log, description
FW_GLOBAL	config	enabled, default_action, tcp_established_timeout, udp_timeout, icmp_timeout, tcam_high_watermark, tcam_low_watermark, max_flows

APP_DB (inter-process communication):

Table	Key	Fields
FW_FLOW_TABLE	proto:src_ip:src_port:dst_ip:dst_port	action, src_zone, dst_zone, priority, tcp_state, packet_count, byte_count, offloaded
FW_POLICY_ACL_TABLE	policy_name	ACL rule fields for baseline deny/permit

STATE_DB (runtime state):

Table	Key	Fields
FW_ZONE_TABLE	zone_name	vni_list, member_ips, vlan_list, description
FW_FLOW_STATE_TABLE	proto:src_ip:src_port:dst_ip:dst_port	state, hitbit_ts, offload_ts
FW_GLOBAL_STATE_TABLE	state	total_flows, offloaded_flows, tcam_used, tcam_total, tcam_usage_percent, total_evictions, circuit_breaker_active
FW_RESTORE_TABLE	Flags	warm_restart, restored
FW_DENY_LOG	seq_num	timestamp, policy, deny_packets, deny_bytes
FW_CONTROLLER_TABLE	leaf_id	sync_status, config_hash, last_sync, consecutive_failures, controller_url

COUNTERS_DB (telemetry):

Table	Key	Fields
COUNTERS_FW_FLOW	proto:src_ip:src_port:dst_ip:dst_port	packets, bytes, last_hit
COUNTERS_FW_ZONE	src_zone:dst_zone	permitted_packets, permitted_bytes, denied_packets, denied_bytes, offloaded_flows, cpu_flows
COUNTERS_FW_GLOBAL	global	total_new_connections, total_offloads, total_evictions, total_denied, tcam_high_watermark_hits

YANG Model

The YANG data model is at yang/sonic-firewall.yang and defines:

• Type definitions: vni-id, zone-name, policy-name, fw-action, ip-protocol
• CONFIG containers: FW_GLOBAL, FW_ZONE, FW_POLICY (with leafref validation for zone references)
• STATE containers: FW_ZONE_TABLE, FW_GLOBAL_STATE_TABLE

CoPP Configuration

copp/copp_fw_config.json defines the Control Plane Policing trap group for punting unmatched inter-zone traffic to the CPU:

• Trap group queue4_group3: sr_tcm mode, CIR=10,000 PPS, red_action=DROP
• Trap fw_intercept: catches traffic matching the inter-zone punt ACL rule

Repository Structure

sonic-firewall/
  src/
    schema/
      fw_schema.h              # Redis table name constants (C++)
      fw_schema.py             # Python schema (constants, defaults, flags)
    fwsyncd/
      fwsync.h                 # FwSync class definition (C++)
      fwsync.cpp               # Conntrack-to-Redis bridge (C++)
      fwsyncd.cpp              # Main entry point (C++)
      fwsyncd_py.py            # Python prototype: ConntrackListener + FwSync + WarmRestartAssist
      copp_setup.py            # CoPP nftables rate-limiting
      save_fw_state.py         # Warm restart shutdown state save
    fworch/
      fworch.h                 # FwOrch class definition (C++)
      fworch.cpp               # Flow-to-ACL orchestrator (C++)
      fworchmain.cpp           # Main entry point (C++)
      fworch_py.py             # Python prototype: FwOrch + LRU + HitbitPoller + CircuitBreaker + Aggregation
    fwpolicyd/
      fwpolicyd.py             # Policy engine + nftables manager
    fwzonemgrd/
      fwzonemgrd.py            # Zone manager daemon
    fwlogd/
      fwlogd.py                # Deny log daemon: DenyLogPoller + parse_deny_counters
    fwcontroller/
      __init__.py
      models.py                # PolicyStore (SQLite), Zone/Policy/LeafInfo/PolicyVersion/Rollout dataclasses
      compiler.py              # PolicyCompiler: validate + compile to CONFIG_DB format
      fwcontroller.py          # Flask REST API: all CRUD + versioning + rollout + drift + telemetry
      ui.py                    # Web UI Blueprint: HTMX + Alpine.js server-rendered pages
      static/                  # Vendored JS/CSS: htmx.min.js, alpine.min.js, pico.min.css, app.css
      templates/               # Jinja2 templates: base.html + 8 page templates + 7 partials
    fwcontrollerd/
      __init__.py
      fwcontrollerd.py         # LeafAgent: polls controller, applies bundles, caches for partition tolerance
  docker/
    Dockerfile.j2              # Container build template (SONiC build system)
    Dockerfile.lab             # Standalone lab image (Python prototypes, no C++)
    supervisord.conf           # Process management with dependent startup (SONiC)
    supervisord.conf.lab       # Lab supervisor config (priority-based, no dependent_startup)
    start.sh                   # Container init (conntrack, nftables setup)
    critical_processes         # SONiC process health monitoring list
    manifest.json              # Container manifest (privileged, warm restart)
  cli/
    config/plugins/firewall.py # config firewall commands (per-switch)
    config/plugins/fwctl.py    # fwctl CLI commands (controller management)
    show/plugins/show_firewall.py  # show firewall commands (zones, policy, flows, counters, tcam, deny-log)
    clear/plugins/clear_firewall.py # clear firewall commands (counters, flows, deny-log)
  scripts/
    restore_fw_state.py        # Warm restart state restoration
    e2e_test_vs.sh             # End-to-end test for SONiC VS
    e2e_test_redis.py          # End-to-end integration test with real Redis
    benchmark_pipeline.py      # Performance benchmark harness
    e2e_test_controller.py     # E2E controller lifecycle test (Phase 5a+5b)
    build_lab.sh               # Build docker-firewall lab image (linux/amd64)
    deploy_lab.sh              # Deploy lab image on SONiC switch (root)
    validate_lab.sh            # Post-deployment validation (daemons, Redis, ACL, conntrack)
  yang/
    sonic-firewall.yang        # YANG data model
  copp/
    copp_fw_config.json        # CoPP trap configuration
  tests/
    conftest.py                # Mock SONiC modules + test fixtures
    redis_adapter.py           # RedisTable wrapper for live E2E testing
    test_cli.py                # CLI tests — 30 tests
    test_fwpolicyd.py          # Policy engine tests — 23 tests
    test_fwzonemgrd.py         # Zone manager tests — 16 tests
    test_fwsyncd.py            # Flow sync tests — 44 tests
    test_fworch.py             # Orchestrator tests — 56 tests
    test_fwlogd.py             # Deny log tests — 12 tests
    test_save_fw_state.py      # Warm restart tests — 8 tests
    test_schema.py             # Schema convention tests — 7 tests
    test_fwcontroller.py       # Controller tests — 121 tests (store, compiler, API, versioning, rollback, drift, rollout, telemetry)
    test_fwcontrollerd.py      # Leaf agent tests — 32 tests (poll, apply, staging, caching)
    test_fwctl_cli.py          # Controller CLI tests — 31 tests
    test_fwcontroller_ui.py    # Web UI tests — 32 tests (pages, zone/policy/config CRUD, rollout, drift, leaves)
  Makefile                     # Build targets for C++ daemons + pytest
  setup.py                     # Python package definition

Total: ~17,800 LOC across 78 files.

Building

Prerequisites

For C++ daemons (requires SONiC build environment):

• sonic-swss-common (libswsscommon, headers)
• libnl-3, libnl-genl-3, libnl-nf-3, libnl-route-3
• libnetfilter_conntrack
• hiredis

For Python components (testable standalone):

• Python 3.9+
• click, tabulate, natsort
• flask (for controller only — not needed on leaf switches)

For E2E integration testing:

• Redis server on localhost
• redis-py (pip install redis)

Build C++ Daemons

make              # Build fwsyncd and fworch
make fwsyncd      # Build flow sync daemon only
make fworch       # Build orchestrator only
make install      # Install all components

Run Tests

python3 -m venv .venv
source .venv/bin/activate
pip install pytest click tabulate natsort flask
make test         # or: pytest tests/ -v

All Python components are fully testable outside a SONiC environment. The test infrastructure (tests/conftest.py) injects mock SONiC modules (swsscommon, utilities_common, sonic_py_common) into sys.modules so CLI plugins and daemon code can be imported and tested with mock Redis connectors.

Current test results: 412/412 passing.

Test File	Tests	Coverage
test_cli.py	30	Config enable/disable, zone CRUD (VNI validation, duplicate detection, in-use protection), policy CRUD (zone existence, same-zone rejection, logging), show commands (zones, policy with filter, flows, counters, TCAM, deny-log), clear commands (counters, flows, deny-log)
test_fwpolicyd.py	23	NftablesManager (init, idempotency, permit/deny rules, port matching, multi-port, flush, default deny, conntrack zones, failure handling), FwPolicyEngine (global config with defaults, policy loading, nftables compilation with priority ordering, baseline ACLs deny-only, dynamic policy add/delete, zone IP loading)
test_fwzonemgrd.py	16	Zone processing (basic, multiple), VNI reverse mapping, global config (enabled/disabled/missing), STATE_DB updates (with IPs, empty VLANs), config change handling (add/delete), NEIGH_TABLE parsing, VXLAN mapping loading, end-to-end zone resolution chain
test_fwsyncd.py	44	Conntrack text/XML parsing, flow key format, offload scoring, zone resolution (inter/intra/unknown/caching), offload eligibility (TCP/UDP), flow add/remove/process, warm restart assist (stale/same/new/delete/flush/reconcile), conntrack dump (flags, missing command, timeout)
test_fworch.py	56	ACL table creation, flow offload (2 rules, fields, deny/permit), STATE_DB/COUNTERS_DB writes, flow delete cleanup, LRU eviction, TCAM capacity/pressure, hitbit aging (TCP/UDP/active/STATE_DB), counter polling, circuit breaker (open/close/cooldown/counters/window), subnet aggregation (/24 mask, threshold, proto filter, marks), warm restart restore (offloaded/cpu/TCAM limit/LRU order/missing/empty), performance sanity (>1000 ops/sec)
test_fwlogd.py	12	parse_deny_counters (standard/multi/empty/no-deny), DenyLogPoller (baseline, delta tracking, JSONL writing, STATE_DB ring buffer, eviction, no-state-table), integration (stop event, threaded)
test_save_fw_state.py	8	Conntrack save (success/not-found/error), nftables save (success/not-found/error), warm restart flag, main orchestration
test_schema.py	7	Redis table naming conventions, CONFIG_DB entry structure, APP_DB flow key format, STATE_DB zone table, COUNTERS_DB key format
test_fwcontroller.py	121	PolicyStore CRUD (zone/policy/global/leaf), referential integrity, config hash determinism, PolicyCompiler (compile/validate/to_config_db), REST API (all endpoints), policy versioning (auto-snapshot, list/get/bundle), rollback (restore + force-sync), drift detection (in_sync/drifted/stale, remediation), staged rollout (create/stage/activate/abort, duplicate rejection), telemetry aggregation

| test_fwcontrollerd.py | 32 | LeafAgent poll (200/304/error), apply writes zones/policies/global, delete stale entries, failure counter, status reporting, URL building, stop event, backoff, idempotent apply, graceful degradation without swsscommon, staged bundle handling, partition tolerance (cache write/read/apply on startup, missing cache) | | test_fwctl_cli.py | 31 | Zone CRUD commands, policy CRUD commands, config show/set, leaves list, status, validate, env var URL, unreachable error, version list/show, rollback, drift, remediate leaf/all, rollout list/create/stage/activate/abort, telemetry | | test_fwcontroller_ui.py | 32 | Full-page routes (dashboard, zones, policies, leaves, config, versions, rollouts, drift), zone CRUD (create, missing name, invalid VNI, duplicate, delete, delete with ref), policy CRUD (create, bad zone, delete), config update (success, bad watermarks), version rollback (success, not found), rollout lifecycle (create, stage, activate, abort), drift (partial with data, remediate), leaf management (empty table, with data, remediate one, remediate all), root redirect |

E2E Integration Testing

The e2e_test_redis.py script runs 8 end-to-end test scenarios against a real Redis instance, validating the full conntrack→fwsyncd→fworch→ACL pipeline without requiring SONiC VS:

# Requires redis-server running on localhost
pip install redis
python3 scripts/e2e_test_redis.py

Results: 36/36 checks passing (validated on Hetzner server 135.181.61.27).

Controller E2E Test

The e2e_test_controller.py script tests the full controller lifecycle including Phase 5b hardening features — no external dependencies required:

python3 scripts/e2e_test_controller.py

Covers 20 steps: zone/policy creation → validation → agent poll (200) → CONFIG_DB apply → poll (304) → policy update → re-sync → deletion → leaf registration → health check → versioning → rollback → drift detection → remediation → staged rollout (create/stage/activate/abort) → partition tolerance (cache/restore) → telemetry.

Performance Benchmarks

The benchmark_pipeline.py harness measures throughput of each pipeline stage using mock tables (no Redis I/O dependency):

python3 scripts/benchmark_pipeline.py

Results (Hetzner, AMD EPYC):

Stage	Ops/sec
Conntrack line parsing	294,374
FwSync process_event (zone lookup)	268,103
FwOrch process_flow_event (offload)	68,975
FwOrch age_idle_flows (eviction)	347,438
Circuit breaker (closed, offload)	66,652
Circuit breaker (open, CPU-only)	155,390
Subnet aggregation	444,725

The bottleneck is FwOrch offload at ~69K ops/sec (each offload writes 2 ACL rules + STATE_DB + COUNTERS_DB). This is well above the ~50K new connections/sec target identified in the risk analysis.

Lab Deployment (Hardware Testing)

A standalone lab Docker image is available for testing on real SONiC switches without the full SONiC build system. It uses Python prototypes (no C++ binaries) and bind-mounts swsscommon from the host switch.

# On your dev machine: build the image
bash scripts/build_lab.sh
# Produces docker-firewall-lab.tar.gz (~55MB)

# Transfer to the switch
scp docker-firewall-lab.tar.gz admin@<switch-ip>:~/

# On the switch: deploy
sudo bash deploy_lab.sh
# Loads Docker image, installs CLI plugins, starts container with:
#   --privileged --network=host --pid=host
#   Bind-mounts: Redis socket, swsscommon, warmboot dir, /etc/sonic, log dir

# Validate deployment
sudo bash validate_lab.sh
# Checks: daemon status, CONFIG_DB/STATE_DB/APP_DB tables, SONiC ACL,
#          conntrack, nftables, container logs, traffic pipeline

The lab image runs all Python daemons (fwzonemgrd, fwpolicyd, fwsyncd, fworch, fwlogd) under supervisord with priority-based startup ordering. The controller agent (fwcontrollerd) is opt-in — set FW_CONTROLLER_URL to enable.

What Requires a SONiC Environment

• C++ compilation: fwsyncd and fworch depend on sonic-swss-common headers and netfilter libraries
• VS integration tests: Full stack on docker-syncd-vs (zone config → policy → nftables → conntrack → flow offload → ACL verification)
• Hardware validation: Broadcom, Mellanox, or Cisco 8000 platforms for ASIC offload testing
• gNMI streaming telemetry: Requires SONiC telemetry container infrastructure

Configuration Example

# Enable the firewall
config firewall enable

# Define zones
config firewall zone add web --vni 1000,1001 --description "Web tier"
config firewall zone add database --vni 2000 --description "Database tier"
config firewall zone add management --vni 3000 --description "Management"

# Define policies
config firewall policy add web_to_db \
    --from web --to database \
    --action permit --protocol tcp --dst-port 3306,5432 \
    --priority 100 --log

config firewall policy add mgmt_to_all \
    --from management --to web \
    --action permit --priority 50

config firewall policy add db_to_web \
    --from database --to web \
    --action deny --priority 200

# View configuration
show firewall zones
show firewall policy
show firewall tcam-usage
show firewall flows --top 10
show firewall deny-log --limit 20

Controller Deployment Example

# ---- Management Host ----
# Start the controller
python3 -m fwcontroller.fwcontroller --port 8643 --db fw_controller.db

# Create zones and policies via fwctl
export FW_CONTROLLER_URL=http://controller-host:8643
fwctl zone add web --vni 1000,1001 --description "Web tier"
fwctl zone add database --vni 2000 --description "Database tier"
fwctl policy add web_to_db --from web --to database \
    --action permit --protocol tcp --dst-port 3306,5432 --priority 100 --log
fwctl config set enabled true

# Validate and check status
fwctl validate
fwctl status

# ---- Leaf Switches ----
# Set env var in docker-firewall container and enable the agent
FW_CONTROLLER_URL=http://controller-host:8643

# Agent polls automatically, applying policies to CONFIG_DB

# ---- Operations ----
# Check which leaves are in sync
fwctl drift

# Force a drifted leaf to re-sync
fwctl remediate leaf-01

# Staged rollout (safe multi-leaf deploy)
fwctl rollout create --description "Add new zone"
fwctl rollout stage 1          # Leaves download but don't apply
fwctl leaves                   # Verify all leaves staged
fwctl rollout activate 1       # All leaves apply simultaneously

# Rollback to a previous version
fwctl version list             # Find the version to restore
fwctl rollback 3               # Restore version 3 + force-sync all leaves

# Fabric telemetry
fwctl telemetry

Design Decisions

Why nftables + conntrack (not VPP, Suricata, etc.)

• nftables/conntrack is mature (15+ years), already in the SONiC kernel, and proven by the NAT subsystem
• VPP would be 10-20x faster per CPU core but requires architectural changes to coexist with an ASIC-based syncd — not justified for MVP
• Suricata is overkill for L3/L4 stateful inspection and its memory footprint (1-4GB) is too heavy for switch CPUs with 4-8GB total

MVP: AclOrch Indirection vs. Direct SAI

The MVP writes flow entries to APP_ACL_RULE_TABLE, letting SONiC's existing AclOrch handle SAI programming. This reuses the entire SAI programming pipeline, CrmOrch TCAM tracking, and counter infrastructure. For production workloads with high flow churn (>1000 adds/deletes per second), a direct SAI path bypassing AclOrch is planned.

TCAM Management

Switch TCAM is limited (2K-16K ACL entries) versus potentially 500K+ conntrack flows. The design uses:

• Watermark-based admission: Stop offloading new flows when TCAM exceeds the high watermark (default 80%)
• LRU eviction: Remove least-recently-hit flows when TCAM pressure is detected, using SAI hitbit polling
• Circuit breaker: Stops all offload activity when churn rate exceeds threshold, preventing TCAM thrashing
• /24 subnet aggregation: Consolidates multiple per-host flows into aggregate /24 rules under TCAM pressure
• Graceful degradation: Flows that can't be offloaded remain on the CPU slow path via nftables

Deny Logging

Deny events are captured by polling nftables counters (not packet-level logging) to minimize CPU overhead. Events are written as structured JSON Lines for SIEM compatibility and stored in a STATE_DB ring buffer (1000 entries max) for CLI access.

Controller Architecture

• Pull model (not push) — leaf agent is just an HTTP client; no server on the switch. Simpler, fewer firewall/NAT issues, agent controls its own poll rate.
• SQLite (not Redis) for controller — runs on management host, not a switch. Single-file DB, ACID, WAL mode for concurrent reads. No server process.
• urllib.request (not requests) for agent — zero new dependencies inside docker-firewall.
• Content-addressable config_hash (not version counter) — SHA-256 of sorted JSON detects manual CONFIG_DB drift on next poll. Best-effort convergence to controller state.
• Delete-then-write for policy application — ensures leaf CONFIG_DB converges even after manual edits. Order: delete stale policies → delete stale zones → write zones → write policies (respects referential constraints).
• Auto-versioning — every mutation (zone/policy/config CRUD) auto-snapshots the full policy state. Enables point-in-time rollback.
• Staged rollout — two-phase deployment: during staging, agents download the bundle but don't apply. On activation, all agents apply simultaneously. Supports abort at any point.
• Partition tolerance — agent caches last-known-good bundle to disk. On startup, if controller is unreachable, applies cached bundle so the leaf continues enforcing policy after reboot during a network partition.
• autostart=false for fwcontrollerd — opt-in controller mode; existing standalone deployments unaffected.

Phased Roadmap

Phase	Status	Description
Phase 0: Foundation	Complete	Repository scaffold, Docker container shell, schema, stub daemons, test infrastructure
Phase 1: Zone Policy (Software-Only)	Complete	Zone manager, policy engine, nftables slow-path enforcement, CLI
Phase 2: ASIC Offload	Complete	fwsyncd conntrack monitoring, fworch ACL programming, CoPP punt rule, two-tier forwarding
Phase 3: TCAM Management + Warm Restart	Complete	LRU eviction with hitbit aging, circuit breaker anti-churn, /24 subnet aggregation, warm restart state save/restore/reconcile
Phase 4: Deny Logging + Testing + Benchmarks	Complete	fwlogd deny counter daemon, live Redis E2E integration test, performance benchmark harness, deny-log CLI commands. Validated on Hetzner Linux server.
Phase 5a: Controller MVP	Complete	Centralized controller (Flask REST API + SQLite), leaf agent (fwcontrollerd), policy compiler, fwctl CLI, multi-leaf management via pull model. 183 new tests.
Phase 5b: Controller Hardening	Complete	Policy versioning with auto-snapshot, rollback to any version, drift detection + auto-remediation, staged rollout (two-phase), partition tolerance (agent-side bundle caching), fabric-wide telemetry aggregation.
Phase 5c: Controller Web UI	Complete	Server-rendered Web UI (Flask + Jinja2 + HTMX + Alpine.js + Pico CSS). Dashboard, zone/policy CRUD, leaf status with auto-refresh, global config, version history + rollback, staged rollout lifecycle, drift detection + remediation. No build step, no npm. 32 tests.
Phase 6: Lab Deployment	In Progress	Standalone Docker image using Python prototypes (no C++ build). Lab-specific Dockerfile, supervisord config, build/deploy/validate scripts. Designed for SONiC switches with bind-mounted swsscommon.

Remaining Work

Near-Term (Lab Validation)

• Hardware E2E testing: Run the full pipeline on a physical SONiC switch (Cisco 8101-32FH-O or Broadcom platform) with the lab Docker image. Validate zone configuration, policy enforcement, conntrack flow detection, ACL offload, and deny logging end-to-end.
• VS integration testing: Complete scripts/e2e_test_vs.sh on SONiC Virtual Switch (docker-syncd-vs) to validate the full stack without hardware.
• Lab image validation: Execute scripts/validate_lab.sh on a deployed switch and resolve any daemon startup or Redis integration issues.

Medium-Term (Production Hardening)

• C++ daemon compilation: Build fwsyncd and fworch C++ binaries in the SONiC build environment. The Python prototypes are functional but the C++ implementations are needed for production performance (NfNetlink conntrack events, direct SAI ACL programming).
• Direct SAI path for fworch: Migrate from AclOrch indirection (APP_ACL_RULE_TABLE) to direct sai_acl_api calls for high-churn flow offload (>1000 adds/deletes per second).
• Performance benchmarking on hardware: Measure actual new-connection rate (target: 50K-200K conn/sec), TCAM utilization under sustained traffic, and CoPP effectiveness under SYN flood on target switch platforms.
• TCAM scale validation: Confirm LRU eviction, circuit breaker, and /24 subnet aggregation behavior under real TCAM pressure (2K-16K ACL entries).
• Conntrack event reliability: Tune NfNetlink socket buffer sizes and validate periodic full-table reconciliation under high event rates.
• Controller HA: Active/standby controller deployment for production resilience.
• Load testing: Validate controller with 40+ simulated leaves, concurrent policy updates, and staged rollouts.

Long-Term (Ecosystem Integration)

• gNMI streaming telemetry: Integrate with SONiC telemetry container for flow decisions, zone counters, and deny log export via gNMI.
• SONiC build system integration: Add docker-firewall to sonic-buildimage (Dockerfile.j2, rules file, platform templates).
• sonic-swss-common schema PR: Upstream the FW_* table name constants to schema.h.
• YANG model integration: Register sonic-firewall.yang with SONiC YANG model infrastructure for config validation.
• Asymmetric routing support: Controller-coordinated conntrack state sync between leaf switches (currently requires symmetric IRB).
• Kubernetes NetworkPolicy integration: Namespace→VNI→zone mapping for CNI policy offload to switch hardware.
• L7 deep packet inspection: Evaluate Suricata or VPP integration for application-layer inspection (Plan B from conceptual plans).
• Multi-vendor hardware validation: Test on Broadcom (Memory series), Mellanox (Spectrum), and Cisco 8000 (Silicon One) platforms.
• Regression testing: Ensure no impact on existing SONiC features (BGP, VXLAN, NAT, ACL, QoS, LLDP, SNMP).

Security

See SECURITY.md for authentication, TLS, CSRF protection, Redis access controls, and production deployment hardening guidelines.

License

Apache-2.0