Skip to content

Add --cleanup CLI flag with idempotent deploy#156

Open
szachovy wants to merge 12 commits into
masterfrom
fix/49-cli-cleanup-flags
Open

Add --cleanup CLI flag with idempotent deploy#156
szachovy wants to merge 12 commits into
masterfrom
fix/49-cli-cleanup-flags

Conversation

@szachovy

@szachovy szachovy commented Apr 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Closes #49

Changes

CLI

  • Added --cleanup flag to superset-cluster shell script
  • Passes deploy or cleanup action as 6th argument to Python

Source (src/initialize.py)

  • Idempotent deploy: Each start_* method checks container health before deploying. If healthy → skip; if any other state → remove and redeploy from scratch
  • Credential recovery: _recover_existing_credentials() reads CA key/cert, root password, superset password, and secret key from healthy nodes. Enables partial redeploy with consistent credentials
  • Cleanup: cleanup() runs teardown_node() per node, gracefully handling unreachable nodes
  • Teardown: teardown_node() uses guarded commands (if/then/fi, xargs -r) joined by &&
  • Compose cleanup: mysql-mgmt redeploy removes initcontainer and compose volume; superset redeploy leaves swarm to clean secrets/networks

Source (src/crypto.py)

  • Added serialization() static method for PEM string → RSAPrivateKey/Certificate conversion

Source (services/mysql-mgmt/docker_compose.yml)

  • Initcontainer scans all 3 MySQL nodes to find an existing InnoDB cluster (not just primary)
  • If cluster exists: adds missing MySQL nodes via addInstance() with incremental recovery
  • If no cluster: creates from scratch (existing behavior)
  • Uses discovered cluster node for router bootstrap and user management

Terraform (tests/setup/main.tf)

  • Removed always_run = timestamp() triggers from all resources
  • Made SSH key generation, SSH config entries, and group vars creation idempotent
  • terraform apply now only recreates missing containers; running nodes are untouched

Tests (tests/testsuite/)

10-stage test flow:

  1. Sanity — infrastructure readiness
  2. System — initial cluster deployment
  3. Pre-disaster functional — normal operation verification
  4. Disaster — stop mgmt primary (node-0) and mysql primary (node-2)
  5. Post-disaster functional — failover verification
  6. Recovery — remove stopped containers, terraform apply recreates them with full provisioning
  7. ** partial redeploy (3 healthy nodes skipped, fresh mysql + mgmt joined ecosystem)Redeploy**
  8. Post-redeploy functional — verify full cluster operates normally after recovery
  9. Cleanup — run --cleanup to teardown all nodes
  10. Cleanup verification — verify no containers, volumes, dirs, .pyc files, or VIP remain

Other

  • .pyc files moved from /opt to /opt/superset-cluster
  • CHANGELOG.md updated

Add --cleanup CLI flag with deploy/cleanup actions
6b1d5da 
Replace blanket '|| true' error suppression in teardown with proper
conditional logic: if/then guards for swarm leave, directory removal,
and VIP cleanup; xargs -r for container, volume, and network removal.
Exit codes are checked via _run_ssh_command() and real failures raise
RuntimeError.

Deploy action (default) always tears down existing state first, making
it idempotent and safe for both fresh installs and redeployments.

Add redeploy and cleanup test stages to the test suite:
- redeploy.yml: restarts stopped nodes from disaster tests, runs a
  fresh deploy, and verifies the cluster is fully functional again.
- cleanup.yml: runs --cleanup and verifies no containers, volumes,
  deployment directories, .pyc files, or VIP addresses remain.

Closes #49

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@szachovy szachovy force-pushed the fix/49-cli-cleanup-flags branch 5 times, most recently from efd3cb3 to 3fee28a Compare April 8, 2026 21:09
@szachovy szachovy changed the title Add --cleanup and --redeploy CLI flags Add --cleanup CLI flag with idempotent deploy Apr 8, 2026
@szachovy szachovy force-pushed the fix/49-cli-cleanup-flags branch 2 times, most recently from 9c80160 to 4f4c363 Compare April 8, 2026 21:39
Add --cleanup CLI flag with idempotent deploy
478acfe 
Make deploy idempotent by checking component health before acting:
- start_mysql_servers() skips nodes where mysql is already healthy
- start_mysql_mgmt() skips nodes where mysql-mgmt is already healthy
- start_superset() skips nodes where superset service already exists
- credentials() handles idempotent directory creation

Cleanup via --cleanup tears down all containers, volumes, networks,
routes, deployed files, and VIP addresses. Teardown commands use
conditional logic (if/then guards, xargs -r) instead of || true,
with all commands per node executed in a single SSH call and exit
status checked.

Test suite extended with redeploy and cleanup stages:
- redeploy.yml: restarts disaster-stopped nodes, runs cleanup then
  fresh deploy, verifies cluster fully functional
- cleanup.yml: runs --cleanup, verifies no containers, volumes,
  directories, .pyc files, or VIP addresses remain

Closes #49

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@szachovy szachovy force-pushed the fix/49-cli-cleanup-flags branch from 4f4c363 to 478acfe Compare April 8, 2026 22:03
szachovy and others added 10 commits May 2, 2026 22:04
Merge remote-tracking branch 'origin/master' into fix/49-cli-cleanup-…
1ebf631 
…flags

# Conflicts:
#	CHANGELOG.md
#	src/remote.py
Fix teardown command chaining and remove redundant error handling
7b7d307 
- Replace && chaining in teardown_node() with independent command
  execution so VIP and directory cleanup are not skipped when an
  intermediate command fails
- Remove redundant try/except IOError around create_directory() since
  it is already idempotent via mkdir -p

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove cleanup test stages from test flow
d60d27d 
Simplify test suite to 7 stages: sanity, deploy, functional_pre,
disaster, recover, redeploy, functional_post. The test focus is on
verifying idempotent redeploy recovers a partially failed cluster,
not on full cleanup verification.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix mysql-mgmt cluster detection and healthcheck timeout
6565f99 
- Add retry logic (3 attempts with 5s delay) to InnoDB cluster detection
  in the initcontainer. Fixes race condition where the second mgmt node
  cannot find the cluster just created by the first mgmt node.

- Use explicit cluster name in dba.getCluster('superset') for more
  reliable detection vs the parameterless dba.getCluster() which depends
  on instance metadata availability.

- Increase mysql-mgmt healthcheck_retries from 3 to 24 in container.py
  to give Docker enough time to report the maincontainer as healthy
  (the maincontainer sleeps 25s before starting mysqlrouter, and Docker's
  own healthcheck needs additional time to pass).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix Docker Compose variable interpolation, mypy errors, and add clean…
1387619 
…up test

- Escape shell-only variables (node, attempt, CLUSTER_NODE, CLUSTER_STATUS)
  with $$ in docker_compose.yml to prevent Docker Compose from interpolating
  them as empty strings
- Fix mypy type errors in crypto.py and initialize.py by adding assert
  isinstance checks for RSAPrivateKey deserialization
- Add cleanup test stage at the end of deploy.yml that runs --cleanup flag
  and verifies all containers, volumes, directories, and VIPs are removed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix SSH user in recovery test and revert healthcheck retries
59181bd 
- Fix recover.yml SSH command to connect as 'superset' user (matching
  the authorized_keys in the container image and the pattern used in
  sanity.yml)
- Revert mysql-mgmt healthcheck_retries to original value of 3 since
  the Docker Compose variable interpolation fix ensures containers
  start correctly without needing extended polling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix post-redeploy functional test assertion
925ed2a 
The post-redeploy functional test incorrectly used after_disaster=True
which expects degraded cluster status (OK_NO_TOLERANCE_PARTIAL). After
the redeploy step fully restores the cluster, the correct assertion is
after_disaster=False which validates the cluster is back to OK status
with Single-Primary topology.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add retries to post-redeploy superset functional test
0cd12b4 
After a full cluster redeploy, Superset may report a query as successful
before results are fully propagated to Redis. Add retries (3 attempts,
20s delay) to the superset post-disaster functional test, matching the
retry pattern already used by the mgmt post-disaster test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run post-redeploy functional tests on freshly deployed node-0
8380993 
After disaster recovery and redeploy, node-1's Superset has stale
Celery/Redis state from surviving the disaster without restart. Node-0
was freshly redeployed with clean state and properly validates that the
redeploy produced a working cluster. This matches the pre-disaster test
which also runs on node-0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add post-disaster functional tests between disaster and recovery
34f74ff 
Run failover verification on surviving node (node-1) after disaster
simulation but before node recovery. Tests verify:
- MySQL InnoDB cluster entered OK_NO_TOLERANCE_PARTIAL state
- A secondary node was promoted to new MySQL primary
- Superset remains accessible via the VIP through surviving nodes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

CI-1: Add --cleanup / --teardown / --redeploy CLI flags

1 participant

@szachovy