Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 63 additions & 26 deletions deploy_tee/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ for its [InitConfig](https://github.com/SeismicSystems/enclave/blob/seismic/crat
never provisions; it consumes a descriptor of an already-running node.
VHD upload to the Azure image registry is a seismic-images concern
(`make push-azure-*`), not this tool.
- **For `seismic-tee-bootstrap genesis`:** the `genesis` binary (built from
- **For `seismic-tee-bootstrap genesis-ceremony`:** the `genesis` binary (built from
summit) on PATH, plus a summit genesis template passed via
`--summit-template`.
- **For `seismic-tee-bootstrap up`** (cohort provisioning): the `pulumi`
Expand Down Expand Up @@ -67,7 +67,7 @@ only consumes one to reach an already-running node.
|---|---|---|---|
| `seismic-tee` | `configure` | any operator, on first boot | POST the node TOML to tdx-init. Reads the node address from `--node <descriptor.json>`. (Attestation verification is disabled pending attested-tls — see below.) |
| `seismic-tee-bootstrap` | `up` / `down` | Seismic, internal | Provision / tear down a cohort of TDX nodes, one independent Pulumi stack each (drives Pulumi via the Automation API). |
| `seismic-tee-bootstrap` | `genesis` | Seismic, once per network | Gather the cohort's summit pubkeys, build `genesis.toml`, POST it to each node. |
| `seismic-tee-bootstrap` | `genesis-ceremony` | Seismic, once per network | Gather the cohort's summit pubkeys, build `genesis.toml`, POST it to each node. One-shot: `send_genesis` only exists in the pre-genesis node API. |
| `seismic-tee-bootstrap` | `manifest` | Seismic | Assemble / validate the network manifest. |

Only `seismic-tee` is cloud-agnostic and **never wraps Pulumi**;
Expand Down Expand Up @@ -118,46 +118,76 @@ uv run seismic-tee configure --node /tmp/node.json \
`http://host:7878` URL (or a node descriptor JSON, which resolves to
`http://<public_ip>:7878`); repeatable. The `--manifest` is the network
identity doc you're given for the network you're joining; it pins the
`network_id`. You never run `seismic-tee-bootstrap` — that's
Seismic-internal network creation.
`network_id`. The reth genesis JSON is POSTed alongside it (from
`--reth-genesis`, defaulting to `reth-genesis.json` beside the manifest —
the artifact-set layout); tdx-init writes it for reth's `--chain`. You
never run `seismic-tee-bootstrap` — that's Seismic-internal network
creation.

### Creating a new network (genesis ceremony)

```bash
# 0. (Upstream) Build the image in seismic-images (CI or `make build`)
# and publish it + its measurements. This tool does not build.

# 1. Assemble the one shared network manifest (all nodes use the same bytes).
# There is no per-node TOML — the genesis node is founded via
# `seismic-tee-bootstrap configure`, joiners via `seismic-tee configure`
# (step 3). uv run seismic-tee-bootstrap manifest assemble ... -o network-manifest.json
# 1. Create the network directory: `init` scaffolds the three authored
# inputs (chain spec, summit template, measurements); you edit them; then
# `assemble --dir` derives the artifact set (manifest, filled template,
# promoted policy) into the same directory. Edits go to the *inputs* +
# re-assemble — the manifest itself is pure output, never hand-edited.
# Commit the directory for any network that matters: a fresh assemble
# mints a fresh genesis_nonce, so the same network_id can never be
# regenerated from the inputs — the directory is everything needed to
# (re)configure, join, or debug the network later. assemble needs
# `seismic-reth` on PATH for the offline genesis-hash gate.
uv run seismic-tee-bootstrap manifest init deploy_tee/networks/devnet-3 \
--reth-genesis ../seismic-reth/crates/seismic/chainspec/res/genesis/dev.json \
--measurements ../seismic-images/build/measurements.json \
--measurement-id seismic_2026-06-11.abc123.vhd
# → edit deploy_tee/networks/devnet-3/summit-template.toml (namespace, …)
uv run seismic-tee-bootstrap manifest assemble deploy_tee/networks/devnet-3

# 2. Provision the cohort — one independent Pulumi stack per node — and
# capture each node's descriptor. Shared settings inherit from the dev
# stack config (pulumi/seismic_node/Pulumi.dev.yaml); the prefix derives
# from it (→ dev-bootstrap-node), so each node's stack, resource group,
# VM, and DNS record are dev-bootstrap-node-<i>. Descriptors default to a
# gitignored descriptors/ dir next to the Pulumi project; --out-dir overrides.
uv run seismic-tee-bootstrap up --count 2 --out-dir /tmp
# → /tmp/dev-bootstrap-node-1.json, /tmp/dev-bootstrap-node-2.json
# VM, and DNS record are dev-bootstrap-node-<i>.
# --network ties the cohort to the network directory: it cross-checks the
# config's vhd_blob_url basename against the measurement policy's
# measurement_id records, refusing a stale image pin before any resource
# exists (name check only — attestation is the cryptographic gate), and
# writes the descriptors into <dir>/nodes/ (gitignored: infra state, not
# identity). Without --network they land in a shared gitignored
# descriptors/ dir next to the Pulumi project; --out-dir overrides either.
uv run seismic-tee-bootstrap up --count 2 --network deploy_tee/networks/devnet-3
# → deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-{1,2}.json

# 3. Configure each node against the shared manifest and POST. Node 1 founds
# the network (genesis, via the bootstrap CLI); node 2 joins (operator CLI),
# fetching root_key from node 1.
uv run seismic-tee-bootstrap configure --genesis /tmp/dev-bootstrap-node-1.json --manifest ./network-manifest.json
uv run seismic-tee configure --node /tmp/dev-bootstrap-node-2.json \
--peer /tmp/dev-bootstrap-node-1.json --manifest ./network-manifest.json
uv run seismic-tee-bootstrap configure \
--genesis deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-1.json \
--manifest deploy_tee/networks/devnet-3/network-manifest.json
uv run seismic-tee configure \
--node deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-2.json \
--peer deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-1.json \
--manifest deploy_tee/networks/devnet-3/network-manifest.json

# 4. Run the genesis ceremony once: builds genesis.toml from the cohort
# (needs the `genesis` binary on PATH) and fans it out to every summit.
uv run seismic-tee-bootstrap genesis --summit-template ./summit-genesis-template.toml \
--node /tmp/dev-bootstrap-node-1.json /tmp/dev-bootstrap-node-2.json
# One-shot: send_genesis only exists in the pre-genesis node API, so this
# can never be re-run against a live network. --summit-template defaults
# to the artifact-set copy beside --manifest (verified against the
# manifest's summit.genesis_template_hash); --node defaults to the
# descriptors in the nodes/ dir beside it.
uv run seismic-tee-bootstrap genesis-ceremony \
--manifest deploy_tee/networks/devnet-3/network-manifest.json
```

After step 3 each node is up; after step 4 they produce blocks. RPC is at
`https://<fqdn>/rpc` (the `fqdn` from each descriptor).

## The config: `--peer` (`[enclave]`), `--manifest`, `--email`
## The config: `--peer` (`[enclave]`), `--manifest`, `--reth-genesis`, `--email`

There is no per-node TOML. The config is assembled from flags + the descriptor,
so nothing is copy-pasted per node (and can't drift):
Expand All @@ -168,6 +198,9 @@ so nothing is copy-pasted per node (and can't drift):
- `[domain]` — `name` from the descriptor's `fqdn` (the cert domain), `email`
from `--email` (default `ops@seismic.systems`).
- `[network].manifest_base64` — from `--manifest`.
- `[network].reth_genesis_base64` — from `--reth-genesis` (default:
`reth-genesis.json` beside `--manifest`). Checked client-side against the
manifest's `eth.chain_id` before POSTing.

The combined schema is owned by `tdx-init` (`#[serde(deny_unknown_fields)]`);
this is the body a joining node POSTs:
Expand All @@ -181,8 +214,9 @@ peers = ["http://az-1.seismicdev.net:7878"] # from --peer
name = "az-1.seismicdev.net"
email = "ops@seismic.systems"

[network] # injected: --manifest; tdx-init 400s if absent
manifest_base64 = "..."
[network] # injected: --manifest + --reth-genesis;
manifest_base64 = "..." # tdx-init 400s if either is absent
reth_genesis_base64 = "..."
```

Authoritative reference:
Expand All @@ -195,13 +229,16 @@ back as 4xx and the deploy script raises.
up (LUKS format + mkfs + TPM enroll takes ~20–40s on first
boot). On success tdx-init writes:

| File | Content | Consumer |
| ------------------------------ | --------------------------------------------------------------- | --------------------------- |
| `/persistent/conf/domain.env` | `DOMAIN_NAME=...`, `DOMAIN_EMAIL=...` | `setup-nginx-ssl` (certbot) |
| `/persistent/conf/enclave.env` | `SEISMIC_ENCLAVE_GENESIS_NODE=...`, `SEISMIC_ENCLAVE_PEERS=...` | `enclave.service` |
| File | Content | Consumer |
| ----------------------------------------- | --------------------------------------------------------------- | --------------------------- |
| `/run/seismic/conf/domain.env` | `DOMAIN_NAME=...`, `DOMAIN_EMAIL=...` | `setup-nginx-ssl` (certbot) |
| `/run/seismic/conf/enclave.env` | `SEISMIC_ENCLAVE_GENESIS_NODE=...`, `SEISMIC_ENCLAVE_PEERS=...` | `enclave.service` |
| `/run/seismic/conf/network-manifest.json` | verbatim manifest bytes (hashed into `network_id`) | `enclave.service` |
| `/run/seismic/conf/reth-genesis.json` | verbatim reth genesis bytes | `reth.service` (`--chain`) |

A sentinel at `/persistent/conf/.tdx-init-done` makes subsequent
boots no-ops.
The conf dir is tmpfs, so every boot needs a fresh POST (`configure` is
recurring, not one-shot); the sentinel `/run/seismic/conf/.tdx-init-done`
only gates against a second POST within the same boot.

### Operator footguns to be aware of

Expand Down
15 changes: 9 additions & 6 deletions deploy_tee/bootstrap_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Seismic-internal, NOT a tool node operators run: it provisions a cohort of
TDX nodes (`up` / `down`) and runs the one-time network-creation steps
(`manifest`, `genesis`). This is the CLI that is *allowed* to wrap Pulumi —
(`manifest`, `genesis-ceremony`). This is the CLI that is *allowed* to wrap
Pulumi —
`up` / `down` drive the seismic_node Automation-API orchestrator. The
operator CLI (`seismic-tee`) deliberately is not; the boundary is the node
descriptor file (see deploy_tee/descriptor.py), which this CLI produces
Expand Down Expand Up @@ -49,19 +50,21 @@ def configure(argv: tuple[str, ...]) -> None:
forward(cohort_configure.main, "seismic-tee-bootstrap configure", argv)


@app.command(name="genesis", context_settings=PASSTHROUGH, add_help_option=False)
@app.command(
name="genesis-ceremony", context_settings=PASSTHROUGH, add_help_option=False
)
@click.argument("argv", nargs=-1, type=click.UNPROCESSED)
def genesis(argv: tuple[str, ...]) -> None:
"""Genesis ceremony: build genesis.toml from the cohort and fan it out."""
def genesis_ceremony(argv: tuple[str, ...]) -> None:
"""One-shot genesis ceremony: build genesis.toml from the cohort, fan it out."""
from deploy_tee import genesis as genesis_mod

forward(genesis_mod.main, "seismic-tee-bootstrap genesis", argv)
forward(genesis_mod.main, "seismic-tee-bootstrap genesis-ceremony", argv)


@app.command(name="manifest", context_settings=PASSTHROUGH, add_help_option=False)
@click.argument("argv", nargs=-1, type=click.UNPROCESSED)
def manifest(argv: tuple[str, ...]) -> None:
"""Assemble / validate a network manifest."""
"""Scaffold (init) / assemble / validate a network artifact-set dir."""
from deploy_tee import manifest as manifest_mod

forward(manifest_mod.main, "seismic-tee-bootstrap manifest", argv)
Expand Down
48 changes: 41 additions & 7 deletions deploy_tee/cohort_configure.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
TDX_INIT_PORT,
build_config,
post_config_to_tdx_init,
resolve_reth_genesis,
)
from deploy_tee.descriptor import load_descriptor, require
from deploy_tee.status import poll_provisioning
Expand Down Expand Up @@ -92,6 +93,7 @@ def build_cohort(genesis_path: Path, join_paths: list[Path]) -> list[Node]:
def _configure_node(
node: Node,
manifest_path: Path,
reth_genesis_path: Path,
email: str,
no_wait: bool,
states: dict[str, str],
Expand All @@ -107,7 +109,12 @@ def _configure_node(
try:
states[node.name] = "building config…"
config = build_config(
manifest_path, node.fqdn, email, genesis_node=node.genesis, peers=node.peers
manifest_path,
node.fqdn,
email,
genesis_node=node.genesis,
peers=node.peers,
reth_genesis_path=reth_genesis_path,
)
states[node.name] = f"POSTing config to tdx-init :{TDX_INIT_PORT}…"
post_config_to_tdx_init(node.public_ip, config)
Expand Down Expand Up @@ -162,7 +169,11 @@ def render(self, states: dict[str, str]) -> None:


def _run_cohort(
nodes: list[Node], manifest_path: Path, email: str, no_wait: bool
nodes: list[Node],
manifest_path: Path,
reth_genesis_path: Path,
email: str,
no_wait: bool,
) -> dict[str, bool]:
"""Configure every node concurrently, refreshing the dashboard until all
workers finish. Returns {node name: ok}. Threads suit this — the work is
Expand All @@ -180,7 +191,14 @@ def _run_cohort(
with ThreadPoolExecutor(max_workers=len(nodes)) as pool:
for node in nodes:
futures[node.name] = pool.submit(
_configure_node, node, manifest_path, email, no_wait, states, stop
_configure_node,
node,
manifest_path,
reth_genesis_path,
email,
no_wait,
states,
stop,
)
try:
# Refresh while workers block on POST/poll.
Expand Down Expand Up @@ -257,6 +275,17 @@ def parse_args() -> argparse.Namespace:
metavar="FILE",
help="Network manifest JSON (from `manifest assemble`); → [network].",
)
parser.add_argument(
"--reth-genesis",
type=Path,
default=None,
metavar="FILE",
help=(
"reth genesis JSON (chain spec) POSTed to every node; → "
"[network].reth_genesis_base64. Default: reth-genesis.json "
"beside --manifest (the artifact-set layout)."
),
)
parser.add_argument(
"--email",
default="ops@seismic.systems",
Expand All @@ -279,12 +308,17 @@ def main() -> None:
setup_logging()
args = parse_args()

# Validate the shared manifest once, so a bad one fails fast here rather
# than as N identical per-worker errors mid-dashboard.
# Validate the shared network artifacts once, so a bad one fails fast here
# rather than as N identical per-worker errors mid-dashboard.
try:
manifest_mod.validate_manifest_schema(args.manifest.read_bytes())
manifest = manifest_mod.validate_manifest_schema(args.manifest.read_bytes())
except manifest_mod.ManifestSchemaError as e:
raise SystemExit(f"--manifest {args.manifest}: invalid manifest: {e}") from None
reth_genesis = resolve_reth_genesis(args.reth_genesis, args.manifest)
try:
manifest_mod.validate_reth_genesis_matches(manifest, reth_genesis.read_bytes())
except manifest_mod.GateError as e:
raise SystemExit(f"--reth-genesis {reth_genesis}: {e}") from None

nodes = build_cohort(args.genesis, args.join)
joiners = [n.name for n in nodes if not n.genesis]
Expand All @@ -293,7 +327,7 @@ def main() -> None:
+ (f", joining={joiners}" if joiners else " (genesis-only)")
)

results = _run_cohort(nodes, args.manifest, args.email, args.no_wait)
results = _run_cohort(nodes, args.manifest, reth_genesis, args.email, args.no_wait)
_report(nodes, results)


Expand Down
Loading
Loading