First Backup Sync Implementation

.gitignore

@@ -19,3 +19,4 @@ coverage
 /agent-linux.bin
 /privateSeeds/
 /test/service/logs.log
+/backup-sync/

CHANGELOG.md

@@ -4,7 +4,18 @@ All notable changes to this project will be documented in this file.
 
 The changelog format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [0.4.2] - Feb-2-2020
+## [0.4.3] - NEXT
+
+**Milestone**: Hippopotamus(0.10.0.6)
+
+| Package          | Version | Link                                                               |
+| ---------------- | ------- | ------------------------------------------------------------------ |
+| Symbol Bootstrap | v0.4.3  | [symbol-bootstrap](https://www.npmjs.com/package/symbol-bootstrap) |
+
+-   Added `--backupSync` to `config` and `start` commands. It downloads a backup with the Mongo and RocksDb databases for faster synchronization.
+-   Added `backup` command. The command backups the Mongo and RocksDb data folder into a Zip file that can be used for `--backupSync` feature.
+
+## [0.4.2] - Feb-2-2021
 
 **Milestone**: Hippopotamus(0.10.0.6)
 
@@ -19,7 +30,7 @@ The changelog format is based on [Keep a Changelog](https://keepachangelog.com/e
 -   Added Symbol Bootstrap version to generated configuration reports.
 -   Renamed command from `supernode` for `enrolSupernode`.
 
-## [0.4.1] - Jan-19-2020
+## [0.4.1] - Jan-19-2021
 
 **Milestone**: Hippopotamus(0.10.0.5)
 
@@ -34,7 +45,7 @@ The changelog format is based on [Keep a Changelog](https://keepachangelog.com/e
 -   Added `CONTROLLER_PUBLIC_KEY` to Supernode's agent configuration
 -   Upgraded Symbol Rest to version 2.3.1.
 
-## [0.4.0] - Jan-14-2020
+## [0.4.0] - Jan-14-2021
 
 **Milestone**: Hippopotamus(0.10.0.5)
 

README.md

@@ -264,6 +264,7 @@ General users should install this tool like any other node module.
 <!-- commands -->
 # Command Topics
 
+* [`symbol-bootstrap backup`](docs/backup.md) - The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.
 * [`symbol-bootstrap clean`](docs/clean.md) - It removes the target folder deleting the generated configuration and data
 * [`symbol-bootstrap compose`](docs/compose.md) - It generates the `docker-compose.yml` file from the configured network.
 * [`symbol-bootstrap config`](docs/config.md) - Command used to set up the configuration files and the nemesis block for the current network

cmds/backup-full-dual.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e
+rm -rf testnet-backup
+mkdir -p testnet-backup
+cp -rf target/databases/db testnet-backup/mongo
+cp -rf target/nodes/api-node/data testnet-backup/data
+rm -rf testnet-backup/data/spool
+
+cd testnet-backup
+zip -r testnet-full-backup.zip *
+cd ..
+
+# push testnet-backup/testnet-full-backup.zip into S3

cmds/backup-full-sync-testnet-backup.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+set -e
+rm -rf backup-sync/testnet-backup
+mkdir -p backup-sync/testnet-backup
+cp -rf target/testnet-dual/databases/db backup-sync/testnet-backup/mongo
+cp -rf target/testnet-dual/nodes/api-node/data backup-sync/testnet-backup/data
+rm -rf backup-sync/testnet-backup/data/spool
+
+cd backup-sync/testnet-backup
+zip -r testnet-local-full-backup.zip *
+cd ../..

cmds/backup-minimal-sync-testnet-backup.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e -x
+rm -rf backup-sync/testnet-backup
+mkdir -p backup-sync/testnet-backup/data
+cp -rf target/testnet-dual/nodes/api-node/data/0* backup-sync/testnet-backup/data
+cp -rf target/testnet-dual/nodes/api-node/data/proof.index.dat backup-sync/testnet-backup/data
+cp -rf target/testnet-dual/nodes/api-node/data/index.dat backup-sync/testnet-backup/data
+rm -rf backup-sync/testnet-backup/data/spool
+touch backup-sync/testnet-backup/data/server.lock # force docker compose to run a recover
+
+cd backup-sync/testnet-backup
+zip -r testnet-local-minimal-backup.zip *
+cd ../..

cmds/backup-testnet-dual.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+set -e
+symbol-bootstrap backup -t target/testnet-dual --destinationFile ./backup-sync/testnet-local-backup.zip $1 $2 $3

cmds/start-testnet-dual.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 set -e
-symbol-bootstrap start -p testnet -a dual -t target/testnet-dual $1 $2 $3
+symbol-bootstrap start -p testnet -a dual -t target/testnet-dual -c  test/testnet-custom-preset.yml $1 $2 $3

cmds/start-testnet-voting.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 set -e
-symbol-bootstrap start -p testnet -a dual -t target/testnet -c test/voting_preset.yml  $1
+symbol-bootstrap start -p testnet -a dual -t target/testnet -c test/voting_preset.yml  $1 $2 $3

docs/backup.md

@@ -0,0 +1,36 @@
+`symbol-bootstrap backup`
+=========================
+
+The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.
+
+Note: This command is designed for NGL to be used when running public main or public test networks. It's not backing up any node specific information.
+
+* [`symbol-bootstrap backup`](#symbol-bootstrap-backup)
+
+## `symbol-bootstrap backup`
+
+The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.
+
+```
+USAGE
+  $ symbol-bootstrap backup
+
+OPTIONS
+  -h, --help                         It shows the help of this command.
+  -t, --target=target                [default: target] The target folder where the symbol-bootstrap network is generated
+
+  --destinationFile=destinationFile  The file location where the backup zip file will be created. Default destination is
+                                     target/backup.zip.
+
+  --nodeName=nodeName                The dual/api node name to be used to backup the data. If not provided, the first
+                                     configured api/dual node would be used.
+
+DESCRIPTION
+  Note: This command is designed for NGL to be used when running public main or public test networks. It's not backing 
+  up any node specific information.
+
+EXAMPLE
+  $ symbol-bootstrap backup
+```
+
+_See code: [src/commands/backup.ts](https://github.com/nemtech/symbol-bootstrap/blob/v0.4.2/src/commands/backup.ts)_

docs/config.md

@@ -30,6 +30,12 @@ OPTIONS
   -u, --user=user                   [default: current] User used to run docker images when creating configuration files
                                     like certificates or nemesis block. "current" means the current user.
 
+  --backupSync                      It downloads a backup with the Mongo and RocksDb databases for faster
+                                    synchronization.
+
+                                    The location of the backup can be found and changed using the 'backupSyncLocation'
+                                    preset configuration. This configuration allows local files and remote URLs
+
   --pullImages                      It pulls the utility images from DockerHub when running the configuration. It only
                                     affects alpha/dev docker images.
 

docs/start.md

@@ -46,6 +46,12 @@ OPTIONS
   --args=args
       Add extra arguments to the docker-compose up command. Check out https://docs.docker.com/compose/reference/up.
 
+  --backupSync
+      It downloads a backup with the Mongo and RocksDb databases for faster synchronization.
+
+      The location of the backup can be found and changed using the 'backupSyncLocation' preset configuration. This 
+      configuration allows local files and remote URLs
+
   --healthCheck
       It checks if the services created with docker compose are up and running.