Describe the bug
When running performance tests to validate and debug issues, we noticed a significant difference in results between machines, even with identical parameters.
The CPU type and architecture create major variations, and the local machine load likely also plays a role.
For instance, running FlightInSubs on my machine (Intel(R) Core(TM) Ultra 9 185H) with DSS_IMAGE=interuss/dss:v0.22.0, NUM_USS=7, NUM_NODES=3, and INTER_USS_NETEM_CONF="delay 36ms 40ms 50% distribution paretonormal loss 0.25% 15%" is extremely slow- getting about half the q/s of a Mac M1. Furthermore, running the same test with datastore_max_open_conns set to 30 is impossible on my machine, whereas it actually increases performance on the M1.
Expected behavior
A solution would be to define a specific cloud-based VM instance and OS (and probably write a deployment script) to establish a common baseline and ensure comparable load test results.
Describe the bug
When running performance tests to validate and debug issues, we noticed a significant difference in results between machines, even with identical parameters.
The CPU type and architecture create major variations, and the local machine load likely also plays a role.
For instance, running
FlightInSubson my machine (Intel(R) Core(TM) Ultra 9 185H) withDSS_IMAGE=interuss/dss:v0.22.0,NUM_USS=7,NUM_NODES=3, andINTER_USS_NETEM_CONF="delay 36ms 40ms 50% distribution paretonormal loss 0.25% 15%"is extremely slow- getting about half the q/s of a Mac M1. Furthermore, running the same test withdatastore_max_open_connsset to 30 is impossible on my machine, whereas it actually increases performance on the M1.Expected behavior
A solution would be to define a specific cloud-based VM instance and OS (and probably write a deployment script) to establish a common baseline and ensure comparable load test results.