node-snapshotter

Snapshots the op-reth (katana-opreth) datadir and uploads to Google Cloud Storage via rclone.

The node stops only during tar; upload runs after the container is back online.

stop container  ->  tar  ->  start container  ->  upload to GCS
^-------- node down --------^   ^--- node up ---^

Snapshot contents

Included	Excluded
db/, static_files/, blobstore/ (if present), reth.toml, rocksdb/ (if present)	discovery-secret, known-peers.json, invalid_block_hooks, *.lck

Genesis and JWT in conf/ are not included.

Requirements

docker, tar, sha256sum, flock, curl, rclone — plus a GCS bucket and service account key.

GCP setup (bucket, IAM, JSON key): see gcloud-setup.md.

Setup

Install path: ~/node-snapshotter (e.g. /root/node-snapshotter when logged in as root).

# 1. Clone
git clone https://github.com/katana-network/node-snapshotter ~/node-snapshotter
cd ~/node-snapshotter

# 2. GCP — follow gcloud-setup.md, then add the JSON key to the repo:
cp /path/to/key.json gcs-service-account.json
chmod 600 gcs-service-account.json

# 3. Config + rclone (both live in the repo)
cp config.env.example config.env
cp rclone.conf.example rclone.conf
"$EDITOR" config.env    # set GCS_BUCKET
"$EDITOR" rclone.conf   # set location to your bucket region (e.g. us-central1)

# 4. Validate
chmod +x bin/snapshot.sh
bin/snapshot.sh dry-run

config.env.example already points at the repo-local credential files:

GCS_SERVICE_ACCOUNT_FILE="~/node-snapshotter/gcs-service-account.json"
RCLONE_CONFIG="~/node-snapshotter/rclone.conf"

Config is loaded from $CONFIG_FILE, then /etc/node-snapshotter/config.env, then ~/node-snapshotter/config.env.

Usage

cd ~/node-snapshotter
bin/snapshot.sh dry-run   # check config, no upload
bin/snapshot.sh run       # full snapshot
bin/snapshot.sh list      # list objects in GCS

Use sudo only if your user cannot run docker without it.

Scheduled runs (systemd)

Update paths in the service unit if your install is not at /root/node-snapshotter, then install:

cd ~/node-snapshotter

sed -i "s|/root/node-snapshotter|$HOME/node-snapshotter|g" systemd/katana-snapshot.service

cp systemd/katana-snapshot.* /etc/systemd/system/
systemctl daemon-reload
systemctl start katana-snapshot.service          # run once now
systemctl enable --now katana-snapshot.timer     # daily at 03:00 UTC
journalctl -u katana-snapshot -f                 # follow logs

Cron fallback: cron/katana-snapshot.cron.example.

GCS layout

Default (LATEST_ONLY=true):

gs://<bucket>/katana-opreth/latest/rethdata.tar
gs://<bucket>/katana-opreth/latest/manifest.json

Public download (if bucket is configured per gcloud-setup.md):

https://storage.googleapis.com/<bucket>/katana-opreth/latest/rethdata.tar
https://storage.googleapis.com/<bucket>/katana-opreth/latest/manifest.json

With LATEST_ONLY=false, versioned snapshots go under katana-opreth/snapshots/<timestamp>-block-<N>/. See config.env.example for retention options.

Restore

Use the op-reth version from manifest.json.

docker stop katana-opreth
rm -rf /root/katana/rethdata/{db,static_files,blobstore,reth.toml,rocksdb}

# public bucket — no credentials needed
curl -o /tmp/rethdata.tar https://storage.googleapis.com/YOUR_BUCKET/katana-opreth/latest/rethdata.tar
tar -xf /tmp/rethdata.tar -C /root/katana/rethdata/

docker start katana-opreth

Timing (≈100 GB datadir)

Phase	SSD	HDD
tar (node down)	~10–25 min	~30–60 min
upload (node up)	~30–90 min	~1.5–3 h

Downtime ≈ tar time only.