[performance] CRDB sharding influences Locust test results

[the-glu]

Describe the bug

Even on the same hardware (see #1557 for issues across hardware), we noticed significant performance variance when running locust load tests without changing any parameters.

As an example, using a dedicated machine in aws (c8a.4xlarge), with the following parameters that ensure all CRDB instances have the same latency (one node per USS, so CRDB needs to cross INTER USS for each connection):

INTER_USS_NETEM_CONF=delay 25ms 7.5ms 50% distribution paretonormal loss 0.025% 25%'
INTRA_USS_NETEM_CONF='delay 600us 40us 25% distribution normal loss 0.0005%'
NUM_USS=3
NUM_NODES=1

Running locust with 3 locust instances running in parallel, one against each DSS and the following command:

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 -v .:/app/ interuss/monitoring-dev uv run locust -f loadtest/locust_files/FlightsInSub.py -H http://dss1.uss1.localutm -u 10 --cluster-count 11 --base-lat 34 --base-lng -118 --area-radius 15182 --max-flight-distance 10000  --uss-base-url http://dss1.uss1.localutm

(with base lat of 34, 35, 36).

The 3 DSS show different q/s results:

Restarting from scratch but restarting CRDB node 1 to make it not leader, and waiting for CRDB to stabilize before running tests:

Now at the beginning, DSS 3 was faster, but quickly all DSS reach the same q/s with high errors.
(Both cases are reproducible)

---

If we switch to a different scenario with 3 USS and 3 nodes the issue is even worse, as shard repartition may be lucky. If CockroachDB needs to contact only local peers inside a USS, queries will be quick. If data is spread across 'external USS', queries will be slow.

INTER_USS_NETEM_CONF='delay 25ms 7.5ms 50% distribution paretonormal loss 0.025% 25%'
INTRA_USS_NETEM_CONF='delay 600us 40us 25% distribution normal loss 0.0005%'
NUM_USS=7
NUM_NODES=3

The test is run only on the first DSS.

Re-creating the cluster from scratch and running it again:

In both cases, in the first run data was locally present, in the second run it was not:

Even when doing direct queries on CRDB, looking at index repartition and querying where data is local vs not local to the DSS:

I also noticed when taking the last screenshot that data repartition also changes randomly. Initially, the numbers were inverted.

Expected behavior

We should define a way to have reproducible results, removing as many variables as possible. To do that, we should probably (and that may not be enough):

• Not have possible variance due to data spread in CockroachDB on nodes with different latencies, and always use NUM_NODES=1
• Always test and use the average query/s across all DSS to remove variance related to data locality relative to the DSS.

and/or

• Look into solutions to guide CockroachDB (e.g. with locality) to be consistent with data spreading.

[the-glu]

A simple solution to reduce impact of issue 1 would be to automatically introduce a loadbalancer in various scenario (especially ones who start-locally), and always target the loadbalancer with locus tests.

[mickmis]

While we cannot use advanced multi-regions capabilities of CRDB, we still have some control over the zones configuration. I think we can quite easily ensure that there is at least one replica per USS by:

• setting a locality to each CRDB node with the USS ID: --locality=uss=1
• setting the default zone to have as many replicas as there are USSs with ALTER RANGE default CONFIGURE ZONE USING num_replicas = 3; (if 3 USSs)

(we could have additional constraints such as constraints = '{"+uss=1": 1, "+uss=2": 1, "+uss=3": 1}' set, however that should not be needed as CRDB by default spreads replicas evenly by locality)

See the data distribution I end up getting (with NUM_USS=3;NUM_NODES=2):

Local runs of your first test result to the following:

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 -v .:/app/ interuss/monitoring-dev uv run locust -f loadtest/locust_files/FlightsInSub.py -H http://dss1.uss1.localutm -u 10 --cluster-count 11 --base-lat 34 --base-lng -118 --area-radius 15182 --max-flight-distance 10000 --uss-base-url http://dss1.uss1.localutm:

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8090:8089 -v .:/app/ interuss/monitoring-dev uv run locust -f loadtest/locust_files/FlightsI nSub.py -H http://dss1.uss2.localutm -u 10 --cluster-count 11 --base-lat 35 --base-lng -118 --area-radius 15182 --max-flight-distance 10000 --uss-base-url http://dss1.uss2.localutm

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8091:8089 -v .:/app/ interuss/monitoring-dev uv run locust -f loadtest/locust_files/FlightsI nSub.py -H http://dss1.uss3.localutm -u 10 --cluster-count 11 --base-lat 36 --base-lng -118 --area-radius 15182 --max-flight-distance 10000 --uss-base-url http://dss1.uss3.localutm

I'm not completely sure what to make of those results, your input would be appreciated to check whether that addresses the issue or not.