Skip to content

testing-guide.md

Latest commit

 

History

History
439 lines (328 loc) · 16.9 KB

testing-guide.md

File metadata and controls

439 lines (328 loc) · 16.9 KB
Raw

Dora Testing Guide

This guide covers how to run, write, and troubleshoot tests across the Dora workspace.

Looking for which tests cover which capability? See testing-capabilities.md (#1633) — the capability-oriented source-of-truth companion to this guide.

Prerequisites

  • Rust toolchain — MSRV is the workspace rust-version in Cargo.toml (currently 1.88.0)
  • Python 3 with numpy and pyarrow installed (pip install numpy pyarrow) — required for Python smoke tests

Quick Start (5-minute validation)

Run these three commands to validate that the workspace is healthy:

# 1. Format check (~5s)
cargo fmt --all -- --check

# 2. Lint (~60s first run, cached after)
cargo clippy --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python \
  -- -D warnings

# 3. Unit + integration tests (~90s first run)
cargo test --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

All three must pass before opening a PR. Python packages are excluded because they require maturin.

Test Tiers

Tier What it covers Command Speed
Format Code style cargo fmt --all -- --check ~5s
Lint Warnings, correctness cargo clippy --all ... ~60s
Unit Individual functions cargo test --all ... ~90s
CLI Command parsing, validation cargo test -p dora-cli ~5s
Integration Node I/O via env vars cargo test --test example-tests ~30s
Smoke Full CLI lifecycle cargo test --test example-smoke -- --test-threads=1 ~3min
E2E Multi-dataflow scenarios cargo test --test ws-cli-e2e -- --ignored --test-threads=1 ~2min
Fault tolerance Restart policies, timeouts cargo test --test fault-tolerance-e2e ~45s
Typos Spelling Install typos-cli, then typos ~2s

Tier Details

Unit Tests

Unit tests live alongside the code they test using #[cfg(test)] modules. Key crates with tests:

Crate Test count What's tested
dora-arrow-convert ~26 Round-trip Arrow type conversions
dora-cli ~96 Command parsing, value parsers, log grep/filtering, JSON parsing, WebSocket client, cluster config
dora-coordinator ~24 WS control/daemon plane, health check, concurrent requests, artifact store, rate limiter, error sanitization
dora-coordinator-store ~10 In-memory and redb CRUD, schema versioning, persistence
dora-core ~8 Dataflow descriptor validation
dora-daemon ~2 Shlex argument parsing
dora-node-api ~10 Input tracking, service/action helpers (ID generation, send_service_request/response)
dora-log-utils ~11 Log parsing utilities
dora-message ~36 Common types, WS protocol, node/data IDs, metadata, auth tokens
ros2-bridge ~30 ROS2 message/service/action parsing

Run a single crate's tests:

cargo test -p dora-cli
cargo test -p dora-core
cargo test -p dora-arrow-convert

CLI Tests

CLI tests verify command parsing, argument validation, and value parsers without running any commands. They live in #[cfg(test)] modules inside the CLI crate.

What's tested:

  • Clap schema validation (Args::command().debug_assert())
  • Parsing of every subcommand (run, up, down, start, stop, list, logs, build, graph, new, status, inspect top, topic list/hz/echo, node list)
  • Rejection of unknown subcommands
  • --help and --version exit codes
  • Value parsers: parse_store_spec (coordinator store backend), parse_window (topic hz window)
  • Utility functions: parse_version_from_pip_show

How to run:

cargo test -p dora-cli

How to add new tests:

When adding a new CLI subcommand or value parser, add a corresponding test in the #[cfg(test)] module of the same file. For subcommand parsing, add a parse_ok call in binaries/cli/src/command/mod.rs. For value parsers, add tests in the file that defines the parser function.

Integration Tests (Node I/O)

File: tests/example-tests.rs

These tests run compiled node executables with pre-recorded inputs and compare outputs against expected baselines. No coordinator or daemon is needed.

cargo test --test example-tests

How it works:

  1. Builds and runs a node crate (e.g., rust-dataflow-example-node)
  2. Sets DORA_TEST_WITH_INPUTS to a JSON file with timed events
  3. Sets DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 for deterministic output
  4. Compares JSONL output against tests/sample-inputs/expected-outputs-*.jsonl

Sample input/output files live in tests/sample-inputs/.

Smoke Tests

File: tests/example-smoke.rs

Two execution modes are tested for each applicable example:

  • Networked (dora up + dora start --detach + poll + dora stop + dora down): exercises the full coordinator/daemon WS control plane.
  • Local (dora run --stop-after): runs everything in-process, testing the single-process dataflow path.
# Must run single-threaded (shared coordinator port)
cargo test -p dora-examples --test example-smoke -- --test-threads=1

# Run only networked or local tests
cargo test -p dora-examples --test example-smoke smoke_rust -- --test-threads=1
cargo test -p dora-examples --test example-smoke smoke_local -- --test-threads=1

A bash script is also available for quick local validation:

./scripts/smoke-all.sh              # all examples
./scripts/smoke-all.sh --rust-only  # Rust examples only
./scripts/smoke-all.sh --python-only # Python examples only

The suite currently exercises 45 smoke scenarios split across:

  • Rust dataflows and benchmark examples
  • Python dataflows in both networked and local modes
  • module expansion via a runnable module dataflow
  • service and action examples in both modes
  • streaming, typed-dataflow, and log aggregation examples
  • cross-language Rust↔Python examples in both modes
  • deterministic validated-pipeline checks
  • queue/timeout regressions, including the Rust queue-latest receiver

Representative coverage includes:

Category Examples
Rust/networked rust-dataflow, rust-dataflow_dynamic, rust-dataflow-url, benchmark
Python/networked python-dataflow, python-async, python-echo, python-drain, python-log, python-logging, python-multiple-arrays, python-concurrent-rw, python-recv-async
Local-mode Python examples above plus service-example, action-example, module-dataflow, streaming-example, typed-dataflow, log-aggregator, validated-pipeline, queue regressions
Cross-language cross-language/rust-to-python.yml, cross-language/python-to-rust.yml
Feature-focused service-example, action-example, streaming-example, validated-pipeline

Examples requiring special dependencies (webcam, CUDA, ROS2, C/C++ toolchain, multi-machine deploy) are not included in smoke tests.

E2E Tests (WebSocket CLI)

File: tests/ws-cli-e2e.rs

Two groups:

Non-ignored (fast): Start an in-process coordinator and test WsSession directly:

cargo test --test ws-cli-e2e
  • cli_list_empty -- empty dataflow listing
  • cli_status_no_daemon -- daemon connectivity check
  • cli_stop_nonexistent -- error for missing dataflows
  • cli_multiple_requests_same_session -- session reuse

Ignored (full stack): Use dora up with real nodes:

cargo test --test ws-cli-e2e -- --ignored --test-threads=1
  • e2e_start_list_stop -- start, list, stop lifecycle
  • e2e_sequential_dataflows -- two dataflows in sequence

Fault Tolerance Tests

File: tests/fault-tolerance-e2e.rs

These test restart policies and input timeouts using Daemon::run_dataflow directly (no CLI needed).

cargo test --test fault-tolerance-e2e

Tests:

  • restart_recovers_from_failure -- node with restart_policy: on-failure survives panics (15s)
  • max_restarts_limit_reached -- node exhausts max_restarts: 2 budget (15s)
  • input_timeout_delivers_input_closed_to_downstream -- input_timeout: 0.5s fires after a silent upstream, delivering InputClosed to the observer (5s)

Dataflow YAMLs for these tests live in tests/dataflows/.

Coordinator Integration Tests

Files: binaries/coordinator/tests/ws_control_tests.rs, binaries/coordinator/tests/ws_daemon_tests.rs

These start an in-process coordinator and test the WebSocket control/daemon planes.

cargo test -p dora-coordinator

Topics covered: health check, list/stop/destroy requests, invalid JSON/params, concurrent requests, ping/pong, daemon registration, disconnect cleanup, error sanitization (no internal chain leaks), artifact store cleanup on drop.

CI Pipeline

Two workflows split by cadence (#1716):

  • .github/workflows/ci.yml — runs on every PR and push to main. Linux-only. Blocks merge. Target ~30-45 min critical path.
  • .github/workflows/nightly.yml — daily 06:40 UTC cron + manual dispatch. Cross-platform. Does NOT block PRs; auto-files nightly-regression issue on failure. ~3-4 hours wall-clock.

PR CI (ci.yml) — fast Linux-only gate

Job Runner What runs
fmt ubuntu-latest cargo fmt --all -- --check
clippy ubuntu-latest cargo clippy --all ... -- -D warnings
test ubuntu-latest cargo check, cargo build, cargo test --all ... (excluding Python crates and dora-examples) plus fast CLI smoke/semantic checks
e2e ubuntu-latest ws-cli-e2e and fault-tolerance-e2e
contract-tests ubuntu-latest tests/example-smoke.rs::contract_* behavior contracts
bench ubuntu-latest criterion benchmark regression check
typos ubuntu-latest crate-ci/typos@master
audit ubuntu-latest cargo-audit + cargo-deny via make qa-audit
unwrap-budget ubuntu-latest .unwrap() / .expect() budget check
check-license ubuntu-latest license validation

Nightly CI (nightly.yml) — cross-platform breadth

Job Runner What runs
smoke-suite ubuntu-latest Full tests/example-smoke.rs (52 tests, all Python examples)
test-cross-platform macOS + Windows Same steps as PR test, on the non-Linux matrix
examples ubuntu/macOS/windows Rust, Rust Git, Multiple Daemons, C, C++, C++ Arrow, CMake
cli-tests ubuntu/macOS/windows dora new template projects, Python dynamic node, queue latency, error-event
bench-example ubuntu/macOS/windows cargo run --example benchmark --release
cross-check matrix (8 targets) compile-only across x86_64/aarch64 linux-gnu + musl + pc-windows-gnu + apple-darwin
ros2-bridge ubuntu-22.04 ROS2 Humble bridge tests + examples
msrv ubuntu-latest cargo-hack check --rust-version
log-sinks / service-action / streaming / record-replay ubuntu-latest Integration smoke against individual dataflow surfaces
cluster-smoke / topic-and-top-smoke / cpu-affinity-smoke / redb-backend-smoke / daemon-reconnect-smoke / state-reconstruction-smoke ubuntu-latest Specialized lifecycle / regression smokes
file-issue-on-failure ubuntu-latest Auto-files issue on any failure

The full smoke suite in tests/example-smoke.rs runs in nightly (smoke-suite), not in PR CI. Developers who need cross-platform or broader coverage before merge run make qa-test / make qa-examples / make qa-nightly locally — see qa-runbook.md.

Writing New Tests

Unit tests

Add a #[cfg(test)] module in the same file as the code under test:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parses_valid_input() {
        let result = parse("valid");
        assert_eq!(result, expected);
    }
}

Integration tests for nodes

Use the integration testing framework in dora-node-api. Three approaches:

1. setup_integration_testing (recommended)

Call before the node's main function to inject inputs and capture outputs:

#[test]
fn test_main_function() -> eyre::Result<()> {
    let events = vec![
        TimedIncomingEvent {
            time_offset_secs: 0.01,
            event: IncomingEvent::Input {
                id: "tick".into(),
                metadata: None,
                data: None,
            },
        },
        TimedIncomingEvent {
            time_offset_secs: 0.055,
            event: IncomingEvent::Stop,
        },
    ];
    let inputs = TestingInput::Input(
        IntegrationTestInput::new("node_id".parse().unwrap(), events),
    );
    let (tx, rx) = flume::unbounded();
    let outputs = TestingOutput::ToChannel(tx);
    let options = TestingOptions { skip_output_time_offsets: true };

    integration_testing::setup_integration_testing(inputs, outputs, options);
    crate::main()?;

    let outputs = rx.try_iter().collect::<Vec<_>>();
    assert_eq!(outputs, expected_outputs);
    Ok(())
}

2. Environment variable mode

Test the compiled executable directly, closest to production behavior:

DORA_TEST_WITH_INPUTS=path/to/inputs.json \
DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 \
DORA_TEST_WRITE_OUTPUTS_TO=/tmp/out.jsonl \
cargo run -p my-node

3. DoraNode::init_testing

For testing node logic without going through main:

let (node, events) = DoraNode::init_testing(inputs, outputs, Default::default())?;

Generating test input files

Record real dataflow events by setting DORA_WRITE_EVENTS_TO:

DORA_WRITE_EVENTS_TO=/tmp/recorded-events dora run examples/rust-dataflow/dataflow.yml

This writes inputs-{node_id}.json files that can be used directly with DORA_TEST_WITH_INPUTS.

Workspace-level integration tests

Add new test files in the tests/ directory. For tests that need the full CLI stack, follow the patterns in tests/example-smoke.rs:

Networked pattern (exercises coordinator + daemon):

  1. Build nodes with Once guards (avoid rebuilding per test)
  2. Clean up stale processes with dora down
  3. Start cluster with dora up
  4. Run dataflow with dora start --detach
  5. Poll dora list --json for completion
  6. Clean up with dora stop --all and dora down

Local pattern (single-process, in-process coordinator):

  1. Build CLI with Once guard
  2. Run dora run <yaml> --stop-after <duration>
  3. Assert exit code is success

Conventions

  • Use assert2::assert! for better error messages (available as dev-dependency)
  • Use tempfile::NamedTempFile for temporary output files
  • E2E tests that need exclusive port access should be #[ignore] and run with --test-threads=1
  • Async tests use #[tokio::test(flavor = "multi_thread")]
  • Fault tolerance test dataflows go in tests/dataflows/
  • Sample input/output baselines go in tests/sample-inputs/

Troubleshooting

cargo test fails to compile Python packages

Always exclude Python packages:

cargo test --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

Smoke/E2E tests fail with "address already in use"

A stale coordinator or daemon is still running. Clean up:

dora down
# or kill processes manually:
pkill -f dora-coordinator
pkill -f dora-daemon

Smoke tests hang or timeout

  • Increase the timeout in the test if your machine is slow (look for Duration::from_secs(...))
  • Check that example nodes build successfully: 
    cargo build -p rust-dataflow-example-node -p rust-dataflow-example-status-node \
  -p rust-dataflow-example-sink -p rust-dataflow-example-sink-dynamic
cargo build -p log-sink-file -p log-sink-alert -p log-sink-tcp
cargo build --release -p benchmark-example-node -p benchmark-example-sink
  • For Python smoke tests, ensure pyarrow and numpy are installed

E2E tests fail when run in parallel

Smoke and ignored E2E tests must run single-threaded:

cargo test --test example-smoke -- --test-threads=1
cargo test --test ws-cli-e2e -- --ignored --test-threads=1

Integration test output doesn't match expected

  1. Check that DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 is set (time offsets vary per machine)
  2. Regenerate baselines if the node's behavior intentionally changed: 
    DORA_TEST_WITH_INPUTS=tests/sample-inputs/inputs-rust-node.json \
DORA_TEST_NO_OUTPUT_TIME_OFFSET=1 \
DORA_TEST_WRITE_OUTPUTS_TO=tests/sample-inputs/expected-outputs-rust-node.jsonl \
cargo run -p rust-dataflow-example-node

Typos check fails

The typos config is in _typos.toml. To add a false-positive exclusion:

[default.extend-identifiers]
MyCustomIdent = "MyCustomIdent"

Tests pass locally but fail in CI

  • CI runs on Ubuntu; check for platform-specific assumptions (paths, process signals)
  • CI uses rust-cache so dependency versions may differ from your local lockfile
  • Ensure cargo fmt --all -- --check passes (CI enforces this)