Investigate performance issues of the 'Busy S2cell Test' scenario

[mickmis]

Sub-issue of #1541 used to report the findings of investigating performance issues for the 'Busy S2cell Test' described here.

[mickmis]

In this comment I report various tests I've made in an attempt to understand better the issue and find a potential fix. Right now they are mostly inconclusive but reporting them can avoid doing twice the effort. Note that those tests were ran on my own laptop, meaning they can be compared among themselves, but hardly with other tests.

DSS started locally with:

INTER_USS_NETEM_CONF=delay 16.5ms 7.5ms 50% distribution paretonormal loss 0.025% 25%
INTRA_USS_NETEM_CONF=delay 600us 40us 25% distribution normal loss 0.0005%
NUM_NODES=3
NUM_USS=3

Locus start command:

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 -v .:/app/ interuss/monitoring-dev \
uv run locust -f loadtest/locust_files/FlightsInSub.py -H http://dss.lb.localutm \
-u 10 --cluster-count 1 --base-lat -34.928683 --base-lng 138.593795 \
--area-radius 550 --max-flight-distance 100 --oi-duration 60 --uss-base-url http://dss.lb.localutm

Tests

Baseline without time-based index

DSS setup:

• v0.23.0-rc2
• time-based notification index disabled (i.e. locks cells ahead of time)

With time-based index

DSS setup:

• v0.23.0-rc2
• time-based notification index enabled (i.e. NO cell locking)

As expected, worse performances in this case.

Replicas on all nodes

DSS setup:

• v0.23.0-rc2
• time-based notification index disabled (i.e. locks cells ahead of time)
• CockroachDB configured to have a replica store in 100% of the nodes

No impact.

Time-based index with alternative locking implementation

DSS setup:

• v0.23.0-rc2
• time-based notification index enabled
• BUT locking cells ahead of time anyway with alternative cell locking implementation:
  • avoid full scan by:
    • using index hint
    • filtering only by cell ID (and not by sub ID nor start/end)

Mostly no impact. Maybe slight improvement on the 95th percentile, but could be statistically irrelevant.

Time-based index with with alternative locking implementation BIS

DSS setup:

• v0.23.0-rc2
• time-based notification index enabled
• BUT locking cells ahead of time anyway with alternative cell locking implementation, building on the previous one:
  • avoid full scan by:
    • using index hint
    • filtering only by cell ID (and not by sub ID nor start/end)
  • split retrieving cells to lock from actually locking them
  • enable cockroachDB session setting optimizer_use_lock_op_for_serializable

Slightly more interesting results, as the 50th percentile response time is significantly reduced, but no change for the 95th percentile (expected since there are failures). We may want to dig a bit more in that direction, starting by isolating the change that produced this improvement and assessing its impact on other test scenarios.

Time-based index with original locking, disabling increment of notification indexes

DSS setup:

• v0.23.0-rc2
• time-based notification index enabled
• BUT locking cells ahead of time anyway with the orinal cell locking implementation
• do not perform any subscription notification index increment

Interestingly no impact on performances. Could this suggest the increment of the notification index is not the bottleneck here?

Original time-based index, disabling increment of notification indexes

DSS setup:

• v0.23.0-rc2
• time-based notification index enabled, no locking as in the original implementation
• do not perform any subscription notification index increment

Made this to see if that makes a different vs previous test. Not incrementing the notification index does not make a different either here.