script: introduce db/dump script command

[giorio94]

Introduce a db/dump script command that dumps the entire contents of StateDB as JSON. This command mimics the behavior of the already existing /dump/ HTTP API, and is intended to be used to retrieve a snapshot of the DB (for instance during sysdump collection), without having to register the HTTP API, and implement a custom client.

I'd additionally nominate this PR for backport to the v0.5 series, which is still used by Cilium v1.19, given that it has no regression risks being only a script command, but it will help with possible troubleshooting, given that it is the first Cilium version in which the operator also includes some statedb usage.

Related: cilium/cilium#46503

[github-actions]

$ make
go build ./...
go: downloading github.com/cilium/hive v1.0.0
go: downloading golang.org/x/time v0.5.0
go: downloading go.yaml.in/yaml/v3 v3.0.3
go: downloading github.com/spf13/cobra v1.8.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/cilium/stream v0.0.0-20240209152734-a0792b51812d
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.8.4
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	404.001s	coverage: 78.6% of statements
ok  	github.com/cilium/statedb/index	0.006s	coverage: 33.7% of statements
ok  	github.com/cilium/statedb/internal	0.036s	coverage: 42.9% of statements
ok  	github.com/cilium/statedb/lpm	4.031s	coverage: 77.9% of statements
ok  	github.com/cilium/statedb/part	61.310s	coverage: 87.3% of statements
ok  	github.com/cilium/statedb/reconciler	0.273s	coverage: 92.5% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	40.455s
ok  	github.com/cilium/statedb/index	1.014s
ok  	github.com/cilium/statedb/internal	1.026s
ok  	github.com/cilium/statedb/lpm	2.793s
ok  	github.com/cilium/statedb/part	36.030s
ok  	github.com/cilium/statedb/reconciler	1.417s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 9V74 80-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	  726504	      1614 ns/op	    619620 objects/sec	    1000 B/op	      16 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 1821859	       656.7 ns/op	   1522705 objects/sec	     520 B/op	       8 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 2163012	       555.7 ns/op	   1799495 objects/sec	     536 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 1851801	       682.5 ns/op	   1465304 objects/sec	     522 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	  866938	      1349 ns/op	    741271 objects/sec	    1100 B/op	      20 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	 1000000	      1102 ns/op	    1112 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 1704753	       704.4 ns/op	     224 B/op	       5 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 1849839	       647.3 ns/op	     200 B/op	       4 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	 1000000	      1094 ns/op	    1096 B/op	       5 allocs/op
BenchmarkDB_NewReadTxn-4                      	567094274	         2.112 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    1496	    727698 ns/op	   1374196 objects/sec	  546075 B/op	    8094 allocs/op
BenchmarkDB_GetInsert-4                       	    1532	    780148 ns/op	   1281808 objects/sec	  530073 B/op	    8094 allocs/op
BenchmarkDB_RandomInsert-4                    	    1722	    672944 ns/op	   1486008 objects/sec	  522021 B/op	    7094 allocs/op
BenchmarkDB_RandomReplace-4                   	     422	   2820647 ns/op	    354529 objects/sec	 2073410 B/op	   29146 allocs/op
BenchmarkDB_SequentialInsert-4                	    1862	    649953 ns/op	   1538573 objects/sec	  522025 B/op	    7094 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     482	   2498114 ns/op	    400302 objects/sec	 3562653 B/op	   45541 allocs/op
BenchmarkDB_Changes_Baseline-4                	    1497	    798601 ns/op	   1252190 objects/sec	  582203 B/op	    9185 allocs/op
BenchmarkDB_Changes-4                         	     919	   1322596 ns/op	    756089 objects/sec	  783704 B/op	   12336 allocs/op
BenchmarkDB_RandomLookup-4                    	   25011	     47641 ns/op	  20990354 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   24163	     49672 ns/op	  20131972 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	    7347	    149058 ns/op	   6708779 objects/sec	  124920 B/op	    1025 allocs/op
BenchmarkDB_FullIteration_All-4               	     814	   1592596 ns/op	  62790551 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	     669	   1738401 ns/op	  57524141 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     178	   6582645 ns/op	  15191462 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	      98	  12388610 ns/op	   8071930 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     170	   7034566 ns/op	  14215519 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	  654459	      1637 ns/op	        14.00 50th_µs	        16.00 90th_µs	        73.00 99th_µs	    1120 B/op	      19 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  517483	      2308 ns/op	    433363 objects/sec	    1825 B/op	      37 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  140274	     14779 ns/op	     67666 objects/sec	   15869 B/op	      82 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     361	   3320633 ns/op	   3011474 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2365077	       503.3 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  772849	      1551 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	   87055	     13779 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	    7921	    141528 ns/op	   73744 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	44.753s
PASS
ok  	github.com/cilium/statedb/index	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_SortableMutex-4   	 5781859	       207.4 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.204s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    2011	    597062 ns/op	   1674868 objects/sec	  838413 B/op	   13975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    3301	    364422 ns/op	   2744072 objects/sec	  385197 B/op	    6668 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    3447	    340026 ns/op	   2940949 objects/sec	  345615 B/op	    6027 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    1569	    751141 ns/op	   1331308 objects/sec	 1286470 B/op	   13976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    3303	    366260 ns/op	   2730302 objects/sec	  372418 B/op	    5769 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    3488	    327810 ns/op	   3050543 objects/sec	  286753 B/op	    5038 allocs/op
Benchmark_LPM_Lookup-4                     	    7741	    155093 ns/op	   6447749 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  134031	      9029 ns/op	 110757175 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_Prefix-4                     	  131367	      9171 ns/op	 109045004 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  242128	      4841 ns/op	 103283571 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	11.908s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_Uint64Map_Random-4                  	    1598	    724207 ns/op	   1380821 items/sec	 2524651 B/op	    6037 allocs/op
Benchmark_Uint64Map_Sequential-4              	    2042	    595830 ns/op	   1678332 items/sec	 2216722 B/op	    5754 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2224	    558671 ns/op	   1789961 items/sec	 2208719 B/op	    4753 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	   12024	     99599 ns/op	  10040239 items/sec	   86352 B/op	    2028 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    1886	    660806 ns/op	   1513304 items/sec	 2518235 B/op	    5033 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    6358	    158260 ns/op	   6318714 items/sec	  118320 B/op	    2404 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	   10000	    101730 ns/op	   9829931 objects/sec	   71504 B/op	    2033 allocs/op
Benchmark_Insert-4                            	    8136	    142360 ns/op	   7024458 objects/sec	  186937 B/op	    3060 allocs/op
Benchmark_Modify-4                            	   12337	     94899 ns/op	  10537469 objects/sec	   58224 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	    9933	    114237 ns/op	   8753752 objects/sec	   58224 B/op	    1007 allocs/op
Benchmark_Replace-4                           	32682494	        35.85 ns/op	  27890245 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	32280253	        36.97 ns/op	  27046964 objects/sec	       0 B/op	       0 allocs/op
Benchmark_txn_1-4                             	 5787306	       208.3 ns/op	   4801374 objects/sec	     168 B/op	       3 allocs/op
Benchmark_txn_10-4                            	10846808	       109.1 ns/op	   9167160 objects/sec	      86 B/op	       2 allocs/op
Benchmark_txn_100-4                           	13132717	        91.21 ns/op	  10963208 objects/sec	      80 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	11538340	       102.9 ns/op	   9717302 objects/sec	      65 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 4927342	       230.7 ns/op	   4333849 objects/sec	     664 B/op	       4 allocs/op
Benchmark_txn_delete_10-4                     	12280554	        95.46 ns/op	  10475736 objects/sec	     106 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	14414358	        83.64 ns/op	  11956695 objects/sec	      47 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	16223114	        73.91 ns/op	  13530211 objects/sec	      24 B/op	       1 allocs/op
Benchmark_Get-4                               	   47946	     25073 ns/op	  39883822 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  139960	      8585 ns/op	 116487278 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  145840	      8229 ns/op	 121526423 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  159001	      7369 ns/op	 135697671 objects/sec	     896 B/op	       1 allocs/op
Benchmark_Hashmap_Insert-4                    	   17582	     68170 ns/op	  14669260 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  152266	      7842 ns/op	 127512256 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  125797	      9556 ns/op	 104651230 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      84	  14154842 ns/op	   7064720 objects/sec	 2111842 B/op	  102364 allocs/op
Benchmark_find16-4                            	218166955	         5.578 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	100000000	        14.06 ns/op	       0 B/op	       0 allocs/op
Benchmark_find48-4                            	426075777	         2.819 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex48_hit-4                   	425200448	         2.816 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex48_miss-4                  	376974115	         3.174 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	423377227	         2.828 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	340143936	         3.526 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/part	42.857s
PASS
ok  	github.com/cilium/statedb/reconciler	0.004s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.85 seconds (batch size 1000)
Throughput 541697.31 objects per second
888MB total allocated, 6015218 in-use objects, 338MB bytes in use

[joamaki]

LGTM.

Perhaps we could think about just removing the HTTP stuff completely once we've switched over to this? As far as I remember it wasn't used for anything else except the dumping. Would also allow removing the 'AnyTable/RemoteTable' thing.

EDIT: Oh wait, 'cilium-dbg/cmd/status.go' uses the 'RemoteTable' construct. Maybe that could be rewritten to use a shell command instead (it's just doing basically the 'health' command as part of 'cilium status').

[giorio94]

Perhaps we could think about just removing the HTTP stuff completely once we've switched over to this?

Yeah, I was wondering about the same as well. It shouldn't be difficult to rewrite the current usages to directly invoke the corresponding shell command once this is merged and the dependency bumped on the Cilium side. The actual removal will likely need to be postponed to a new minor version, though, as it is technically a breaking change.