Document Snapshot Downloads

docs/indexer.md

@@ -195,6 +195,7 @@ services:
       --bootnodes "/dns/bootnode2.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWSvSQNVHGmK965dKcCDGaxtyeY1XPCMwFUSLC8opguG1T"
       --bootnodes "/ip4/51.210.3.173/tcp/30333/p2p/12D3KooWRUd3BqRyiGfhxVb2BSyUDLK5nHHNXTddZpqzqvQ73C9u"
       --bootnodes "/ip4/141.95.65.179/tcp/30683/p2p/12D3KooWQ8xPXuBww4qSumnjycjjKDThFUj4nDgGS3UPoLyRBvqJ"
+      --database paritydb
       --validator
       --port 30333
       --log info

docs/mainnet.md

@@ -92,7 +92,43 @@ The generated key should now be in a file called `subkey.key` in the sxt-node-ke
 
 Here we assume the setup uses the following volumes: `sxt-mainnet-data` is the block storage volume and the volume where the generated node key is stored is `sxt-node-key`.
 
-### 2.1. Docker Run
+### 2.1. Snapshot Download and Extraction
+
+When starting a new node, the initial syncing of blocks with the rest of the network can take multiple days.
+To alleviate this, it is recommended to download and extract a snapshot of the `paritydb` database prior to booting your node.
+Space and Time roughly publishes snapshots on a two-week cadence.
+
+The date of the latest snapshot (in `YYYY-MM-DD` format) can be found at https://snapshots.mainnet.sxt.network/latest.txt
+The snapshot and checksum for the corresponding date can then be downloaded at the following URLs, respectively
+(where `${DATE}` is once again in `YYYY-MM-DD` format).
+
+```
+https://snapshots.mainnet.sxt.network/${DATE}/sxt-mainnet.tar.gz
+https://snapshots.mainnet.sxt.network/${DATE}/sxt-mainnet.sha1
+```
+
+Once downloaded, extract the contents of the archive onto the `/chains` directory in the block storage volume.
+The following snippet downloads and extracts the snapshot as describe above leveraging the [aria2](https://aria2.github.io/)
+utility for parallelizing and retrying the download process.
+
+```bash
+LATEST_DATE="$(curl -s https://snapshots.mainnet.sxt.network/latest.txt)"
+pushd sxt-mainnet-data  # Directory that will be mounted as the block storage volume in docker
+aria2c -x 16 -s 16 "https://snapshots.mainnet.sxt.network/${LATEST_DATE}/sxt-mainnet.tar.gz"
+curl -O "https://snapshots.mainnet.sxt.network/${LATEST_DATE}/sxt-mainnet.sha1"
+sha1sum --check "sxt-mainnet.sha1"
+mkdir -p chains
+tar xf sxt-mainnet.tar.gz -C chains
+rm sxt-mainnet.tar.gz
+popd
+```
+
+Note that before the `sxt-mainnet.tar.gz` archive gets deleted in the snippet above, you'll be required to have enough disk space
+to host both the archive and the exploded directory, which roughly equates to 2.5TB. To get around this requirement, a separate
+volume that gets temporarily mounted to download the snapshot is recommended. A streaming solution would likely be the most space efficient,
+but it has not yet been developed.
+
+### 2.2. Docker Run
 
 Make sure to set VALIDATOR_NAME to make it easier to identify your node in the telemetry dashboard.
 
@@ -121,6 +157,7 @@ docker run -d --restart always \
   --bootnodes "/dns/bootnode2.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWSvSQNVHGmK965dKcCDGaxtyeY1XPCMwFUSLC8opguG1T" \
   --bootnodes "/ip4/51.210.3.173/tcp/30333/p2p/12D3KooWRUd3BqRyiGfhxVb2BSyUDLK5nHHNXTddZpqzqvQ73C9u" \
   --bootnodes "/ip4/141.95.65.179/tcp/30683/p2p/12D3KooWQ8xPXuBww4qSumnjycjjKDThFUj4nDgGS3UPoLyRBvqJ" \
+  --database "paritydb" \
   --validator \
   --port 30333 \
   --log info \
@@ -130,7 +167,7 @@ docker run -d --restart always \
   --name ${VALIDATOR_NAME?}
 ```
 
-### 2.2. Docker Compose
+### 2.3. Docker Compose
 
 Prepare a `docker-compose.yaml` file as follows:
 
@@ -164,11 +201,12 @@ services:
       --keystore-path /key
       --node-key-file /node-key/subkey.key
       --bootnodes "/dns/validator0.mainnet.sxt.network/tcp/30333/p2p/12D3KooWK4MUYTiz8H6gG98JwN3bT11keivvFLYjtwEv5sqhwkAt"
-      --bootnodes "/dns/validator.mainnet-sxt.ethernodes.io/tcp/30333/p2p/12D3KooWMEq9xSwmr8uTthCb6KM86av1T7AYe44kDPxWovEqzq1w" \
-      --bootnodes "/dns/bootnode1.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWN8ZsZDNVr1ooMWJtsqMNHCVaQ4cvxvB8kELqrZeqct79" \
-      --bootnodes "/dns/bootnode2.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWSvSQNVHGmK965dKcCDGaxtyeY1XPCMwFUSLC8opguG1T" \
-      --bootnodes "/ip4/51.210.3.173/tcp/30333/p2p/12D3KooWRUd3BqRyiGfhxVb2BSyUDLK5nHHNXTddZpqzqvQ73C9u" \
-      --bootnodes "/ip4/141.95.65.179/tcp/30683/p2p/12D3KooWQ8xPXuBww4qSumnjycjjKDThFUj4nDgGS3UPoLyRBvqJ" \
+      --bootnodes "/dns/validator.mainnet-sxt.ethernodes.io/tcp/30333/p2p/12D3KooWMEq9xSwmr8uTthCb6KM86av1T7AYe44kDPxWovEqzq1w"
+      --bootnodes "/dns/bootnode1.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWN8ZsZDNVr1ooMWJtsqMNHCVaQ4cvxvB8kELqrZeqct79"
+      --bootnodes "/dns/bootnode2.sxt.blockhunters.services/tcp/30333/p2p/12D3KooWSvSQNVHGmK965dKcCDGaxtyeY1XPCMwFUSLC8opguG1T"
+      --bootnodes "/ip4/51.210.3.173/tcp/30333/p2p/12D3KooWRUd3BqRyiGfhxVb2BSyUDLK5nHHNXTddZpqzqvQ73C9u"
+      --bootnodes "/ip4/141.95.65.179/tcp/30683/p2p/12D3KooWQ8xPXuBww4qSumnjycjjKDThFUj4nDgGS3UPoLyRBvqJ"
+      --database "paritydb"
       --validator
       --port 30333
       --log info

docs/testnet.md

@@ -120,7 +120,43 @@ The generated key should now be in a file called `subkey.key` in the sxt-node-ke
 
 Here we assume the setup uses the following volumes: `sxt-testnet-data` is the block storage volume and the volume where the generated node key is stored is `sxt-node-key`.
 
-### 2.1. Docker Run
+### 2.1. Snapshot Download and Extraction
+
+When starting a new node, the initial syncing of blocks with the rest of the network can take multiple days.
+To alleviate this, it is recommended to download and extract a snapshot of the `paritydb` database prior to booting your node.
+Space and Time roughly publishes snapshots on a two-week cadence.
+
+The date of the latest snapshot (in `YYYY-MM-DD` format) can be found at https://snapshots.testnet.sxt.network/latest.txt
+The snapshot and checksum for the corresponding date can then be downloaded at the following URLs, respectively
+(where `${DATE}` is once again in `YYYY-MM-DD` format).
+
+```
+https://snapshots.testnet.sxt.network/${DATE}/sxt-mainnet.tar.gz
+https://snapshots.testnet.sxt.network/${DATE}/sxt-mainnet.sha1
+```
+
+Once downloaded, extract the contents of the archive onto the `/chains` directory in the block storage volume.
+The following snippet downloads and extracts the snapshot as describe above leveraging the [aria2](https://aria2.github.io/)
+utility for parallelizing and retrying the download process.
+
+```bash
+LATEST_DATE="$(curl -s https://snapshots.testnet.sxt.network/latest.txt)"
+pushd sxt-testnet-data  # Directory that will be mounted as the block storage volume in docker
+aria2c -x 16 -s 16 "https://snapshots.testnet.sxt.network/${LATEST_DATE}/sxt-testnet.tar.gz"
+curl -O "https://snapshots.testnet.sxt.network/${LATEST_DATE}/sxt-testnet.sha1"
+sha1sum --check "sxt-testnet.sha1"
+mkdir -p chains
+tar xf sxt-testnet.tar.gz -C chains
+rm sxt-testnet.tar.gz
+popd
+```
+
+Note that before the `sxt-testnet.tar.gz` archive gets deleted in the snippet above, you'll be required to have enough disk space
+to host both the archive and the exploded directory, which roughly equates to 2.5TB. To get around this requirement, a separate
+volume that gets temporarily mounted to download the snapshot is recommended. A streaming solution would likely be the most space efficient,
+but it has not yet been developed.
+
+### 2.2. Docker Run
 
 Make sure to set VALIDATOR_NAME to make it easier to identify your node in the telemetry dashboard.
 
@@ -146,6 +182,7 @@ docker run -d --restart always \
   --bootnodes "/dns/new-validator0.testnet.sxt.network/tcp/30333/p2p/12D3KooWFSan1dfyXvyJkGoTf3Jyk7pLmWJpHEMriSYSt5JoqmDB" \
   --bootnodes "/dns/new-validator1.testnet.sxt.network/tcp/30333/p2p/12D3KooWLcAKtSNC2fQM8SEPwsLSrNijfihQ7ENFdb3TpqS2WCas" \
   --bootnodes "/dns/new-validator2.testnet.sxt.network/tcp/30333/p2p/12D3KooWHEJSqiT9KGVdU3HC7dvodY2DG8E8cHoVkSevA4ZVaM2d" \
+  --database "paritydb" \
   --validator \
   --port 30333 \
   --log info \
@@ -155,7 +192,7 @@ docker run -d --restart always \
   --name ${VALIDATOR_NAME?}
 ```
 
-### 2.2. Docker Compose
+### 2.3. Docker Compose
 
 Prepare a `docker-compose.yaml` file as follows:
 
@@ -191,6 +228,7 @@ services:
       --bootnodes "/dns/new-validator0.testnet.sxt.network/tcp/30333/p2p/12D3KooWFSan1dfyXvyJkGoTf3Jyk7pLmWJpHEMriSYSt5JoqmDB"
       --bootnodes "/dns/new-validator1.testnet.sxt.network/tcp/30333/p2p/12D3KooWLcAKtSNC2fQM8SEPwsLSrNijfihQ7ENFdb3TpqS2WCas"
       --bootnodes "/dns/new-validator2.testnet.sxt.network/tcp/30333/p2p/12D3KooWHEJSqiT9KGVdU3HC7dvodY2DG8E8cHoVkSevA4ZVaM2d"
+      --database paritydb
       --validator
       --port 30333
       --log info