Skip to content

Add sequential test method execution and support for HZ_LICENSEKEY env variable [CTT-1276] #2336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k-jamroz merged 2 commits into from
Jul 20, 2026
Merged

Add sequential test method execution and support for HZ_LICENSEKEY env variable [CTT-1276] #2336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6e9e3b3
Add sequential test method execution and support for HZ_LICENSEKEY en…
k-jamroz Jul 16, 2026
3bef8d7
Improve code gen and test
k-jamroz Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 22 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -548,27 +548,27 @@ Let's explain the lines one by one.
Each section within the `tests.yaml` file contains properties for the environment the defined tests for that section
should be conducted in.

| Property | Example value | Description |
|----------------------------------------|------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| `name` | `read_only` | The name of the test suite (overriden by test-specific values) |
| `repetitions` | `1` | The number of times this test suite should run (1 or more) |
| `duration` | `300s` | The amount of time this test suite should run for (45m, 1h, 2d, etc.) |
| `clients` | `1` | The number of loadgenerator workers to use in this test suite (hosted on `loadgenerator_hosts` |
| `members` | `1` | The number of Hazelcast Members to use in this test suite (hosted on `node_hosts`) |
| `loadgenerator_hosts` | `loadgenerators` | Defines the host for Clients, based on either `loadgenerators` or `nodes`, allowing both separate and mixed client/member setups |
| `node_hosts` | `nodes` | Defines the host for Members - this should generally always be `nodes`, and only `loadgenerator_hosts` should be changed for mixed testing. |
| `driver` | `hazelcast5` | The Hazelcast Driver to use - for 5.0+ testing, this is either `hazelcast5` or `hazelcast-enterprise5` for OS or EE respectively |
| `version` | `maven=5.1` | The Hazelcast version to use - typically provided by maven, i.e. `maven=5.3.0-SNAPSHOT` |
| `client_args` | `-Xms3g -Xmx3g` | The command-line Java parameters passed to all clients in this test suite |
| `member_args` | `-Xms3g -Xmx3g` | The command-line Java parameters passed to all members in this test suite |
| `performance_monitor_interval_seconds` | `1` | The interval of the Simulator performance monitor |
| `verify_enabled` | `True` | Defines whether tests should be verified after completion or not (default true) |
| `warmup_seconds` | `0` | The number of seconds from the start of the test to exclude in reporting (only used for report generation) |
| `cooldown_seconds` | `0` | The number of seconds before the end of the test to exclude in reporting (only used for report generation) |
| `license_key` | `your_ee_key` | The Hazelcast Enterprise Edition license to use in your test, if using `hazelcast-enterprise5` drivers |
| `parallel` | `True` | Defines whether tests should be run in parallel when multiple tests are defined within 1 suite (default false) |
| `cp_priorities` | <pre>- address: internalIp<br> &nbsp;priority: 1</pre> | Defines the leadership priority of the CP Subsystem members in the cluster. Use the internal IP address of the agent(s) you wish to configure. |
| `clients_per_loadgenerator` | `1` | The number of Hazelcast client instances per loadgenerator worker (default 1) |
| Property | Example value | Description |
|----------------------------------------|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `name` | `read_only` | The name of the test suite (overriden by test-specific values) |
| `repetitions` | `1` | The number of times this test suite should run (1 or more) |
| `duration` | `300s` | The amount of time this test suite should run for (45m, 1h, 2d, etc.) |
| `clients` | `1` | The number of loadgenerator workers to use in this test suite (hosted on `loadgenerator_hosts` |
| `members` | `1` | The number of Hazelcast Members to use in this test suite (hosted on `node_hosts`) |
| `loadgenerator_hosts` | `loadgenerators` | Defines the host for Clients, based on either `loadgenerators` or `nodes`, allowing both separate and mixed client/member setups |
| `node_hosts` | `nodes` | Defines the host for Members - this should generally always be `nodes`, and only `loadgenerator_hosts` should be changed for mixed testing. |
| `driver` | `hazelcast5` | The Hazelcast Driver to use - for 5.0+ testing, this is either `hazelcast5` or `hazelcast-enterprise5` for OS or EE respectively |
| `version` | `maven=5.1` | The Hazelcast version to use - typically provided by maven, i.e. `maven=5.3.0-SNAPSHOT` |
| `client_args` | `-Xms3g -Xmx3g` | The command-line Java parameters passed to all clients in this test suite |
| `member_args` | `-Xms3g -Xmx3g` | The command-line Java parameters passed to all members in this test suite |
| `performance_monitor_interval_seconds` | `1` | The interval of the Simulator performance monitor |
| `verify_enabled` | `True` | Defines whether tests should be verified after completion or not (default true) |
| `warmup_seconds` | `0` | The number of seconds from the start of the test to exclude in reporting (only used for report generation) |
| `cooldown_seconds` | `0` | The number of seconds before the end of the test to exclude in reporting (only used for report generation) |
| `license_key` | `your_ee_key` | The Hazelcast Enterprise Edition license to use in your test, if using `hazelcast-enterprise5` drivers. License key can be also set using `HZ_LICENSEKEY` environment variable. |
| `parallel` | `True` | Defines whether tests should be run in parallel when multiple tests are defined within 1 suite (default false) |
| `cp_priorities` | <pre>- address: internalIp<br> &nbsp;priority: 1</pre> | Defines the leadership priority of the CP Subsystem members in the cluster. Use the internal IP address of the agent(s) you wish to configure. |
| `clients_per_loadgenerator` | `1` | The number of Hazelcast client instances per loadgenerator worker (default 1) |

### Specify test class(es) and number of threads per worker

Expand All @@ -580,6 +580,7 @@ properties shown in the above example are built-in "magic" properties of Simulat
| `class` | `com.hazelcast.simulator.tests.map.IntByteMapTest` | Defines the fully qualified class name for the Simulator Test. Used to create the test class instance on the Simulator Worker. This is the only mandatory property which has to be defined. |
| `name` | `MyByteTest` | Defines a unique name for this Test. This property is only required when running multiple tests on the same test class, without it only 1 test will run per class type (as the class name is used as the name if not defined here). The tests are ordered alphabetically by name. |
| `threadCount` | `40` | Defines how many threads are running the Test methods in parallel. In other words, defines the number of worker threads for Simulator Tests which use the `@RunWithWorker` annotation. |
| `sequential` | `true` | By default test operations are chosen randomly based on probabilities (`sequential` not configured or `false`). Sequential mode causes test operations to be executed in round-robin, deterministic order, sorted alphabetically. |

> :books: For details about available values for `class`, refer to the provided classes in the [drivers](java/drivers)
> directory or the [Writing a Simulator test](#writing-a-simulator-test) section.
Expand Down
2 changes: 1 addition & 1 deletion java/drivers/driver-hazelcast4plus/conf/configure.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def _configure_hazelcast_xml(nodes, args: DriverConfigureArgs, is_lite_member: b
config = read_file(src_file)

# configure <!--LICENSE-KEY-->
license_key = args.test.get("license_key")
license_key = args.test.get("license_key") or os.getenv("HZ_LICENSEKEY")
if license_key is not None:
license_key_config = f"<license-key>{license_key}</license-key>"
config = config.replace("<!--LICENSE-KEY-->", license_key_config)
Expand Down
27 changes: 27 additions & 0 deletions ...-hazelcast4plus/src/main/java/com/hazelcast/simulator/tests/map/LongByteArrayMapTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,24 @@ public void deleteAndPut(ThreadState state) {
map.put(key, state.randomValue());
}

/**
* {@link #deleteNewCurrent(ThreadState)} and {@link #setCurrent(ThreadState)} form a sequential scenario,
* that ensures that entry exists before delete (if filled during setup) and does not exist before insertion.
*/
@TimeStep(prob = 0)
public void deleteNewCurrent(ThreadState state) {
var map = getRandomMap();
long key = state.newRandomCurrentKey();
map.delete(key);
}

@TimeStep(prob = 0)
public void setCurrent(ThreadState state) {
var map = getRandomMap();
long key = state.currentKey();
map.set(key, state.randomValue());
}

@TimeStep(prob = 0.0)
public CompletableFuture putAsync(ThreadState state) {
return getRandomMap().putAsync(state.randomKey(), state.randomValue()).toCompletableFuture();
Expand Down Expand Up @@ -204,6 +222,7 @@ public class ThreadState extends BaseThreadState {
public static final int HIGHEST_PROBABILITY = 100;
private Pipelining<byte[]> pipeline;
private int i;
private long currentKey = randomKey();

private long fixedKeyOrRandom() {
if (fixedKeyDomain > 0 && fixedKeyDomain < keyDomain && fixedKeyProbability > 0 &&
Expand All @@ -220,6 +239,14 @@ private long randomKey() {
private byte[] randomValue() {
return values[randomInt(values.length)];
}

public long currentKey() {
return currentKey;
}

public long newRandomCurrentKey() {
return currentKey = randomKey();
}
}

private static final class UpdateEntryProcessor implements EntryProcessor<Long, byte[], Object> {
Expand Down
6 changes: 6 additions & 0 deletions java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,12 @@
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.27.7</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
21 changes: 16 additions & 5 deletions ...src/main/java/com/hazelcast/simulator/worker/testcontainer/TimeStepLoopCodeGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import java.net.URL;
import java.net.URLClassLoader;
import java.security.PrivilegedAction;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand All @@ -66,7 +67,8 @@ Class compile(
Class<? extends LatencyProbe> probeClass,
long logFrequency,
long logRateMs,
boolean hasIterationCap) {
boolean hasIterationCap,
boolean sequential) {

ensureExistingDirectory(targetDirectory);

Expand All @@ -81,7 +83,7 @@ Class compile(
className += testCaseId;
}
JavaFileObject file = createJavaFileObject(
className, executionGroup, metronomeClass, timeStepModel, probeClass, logFrequency, logRateMs, hasIterationCap);
className, executionGroup, metronomeClass, timeStepModel, probeClass, logFrequency, logRateMs, hasIterationCap, sequential);
return compile(javaCompiler, file, className);
}

Expand Down Expand Up @@ -123,15 +125,17 @@ Class compile(JavaCompiler compiler, JavaFileObject file, final String className
});
}

private JavaFileObject createJavaFileObject(
// visible for tests
JavaFileObject createJavaFileObject(
String className,
String executionGroup,
Class<? extends Metronome> metronomeClass,
TimeStepModel timeStepModel,
Class<? extends LatencyProbe> probeClass,
long logFrequency,
long logRateMs,
boolean hasIterationCap) {
boolean hasIterationCap,
boolean sequential) {
try {
Configuration cfg = new Configuration(Configuration.VERSION_2_3_24);
cfg.setClassForTemplateLoading(this.getClass(), "/");
Expand All @@ -142,7 +146,11 @@ private JavaFileObject createJavaFileObject(
Map<String, Object> root = new HashMap<>();
root.put("testInstanceClass", getClassName(timeStepModel.getTestClass()));
root.put("metronomeClass", getMetronomeClass(metronomeClass));
root.put("timeStepMethods", timeStepModel.getActiveTimeStepMethods(executionGroup));
List<Method> activeTimeStepMethods = timeStepModel.getActiveTimeStepMethods(executionGroup);
if (sequential) {
activeTimeStepMethods.sort(Comparator.comparing(Method::getName));
}
root.put("timeStepMethods", activeTimeStepMethods);
root.put("probeClass", getClassName(probeClass));
root.put("isStartNanos", new IsStartNanos(timeStepModel));
root.put("isAssignableFrom", new IsAssignableFromMethod());
Expand All @@ -162,6 +170,9 @@ private JavaFileObject createJavaFileObject(
if (hasIterationCap) {
root.put("hasIterationCap", "true");
}
if (sequential) {
root.put("sequential", "true");
}

Template temp = cfg.getTemplate("TimeStepLoop.ftl");
StringWriter out = new StringWriter();
Expand Down
5 changes: 4 additions & 1 deletion .../simulator/src/main/java/com/hazelcast/simulator/worker/testcontainer/TimeStepRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ class TimeStepRunner extends TestRunner {
long iterations = binding.loadAsLong(toPropertyName(executionGroup, "iterations"), 0);
runIterationMap.put(executionGroup, iterations);

boolean sequential = binding.loadAsBoolean(toPropertyName(executionGroup, "sequential"), false);

Class loopClass = new TimeStepLoopCodeGenerator().compile(
testContainer.getTestCase().getId(),
executionGroup,
Expand All @@ -87,7 +89,8 @@ class TimeStepRunner extends TestRunner {
binding.getProbeClass(),
logFrequency,
logRateMs,
iterations > 0);
iterations > 0,
sequential);

loopClassMap.put(executionGroup, loopClass);
}
Expand Down
7 changes: 5 additions & 2 deletions java/simulator/src/main/resources/TimeStepLoop.ftl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public class ${className} extends TimeStepLoop {
final ThrottlingLogger throttlingLogger = new ThrottlingLogger(logger, ${logRateMs});
</#if>

<#if timeStepMethods?size gt 1>
<#if timeStepMethods?size gt 1 && !sequential??>
final byte[] probs = this.timeStepProbabilities;
</#if>

Expand Down Expand Up @@ -82,8 +82,11 @@ public class ${className} extends TimeStepLoop {
${method.name}Probe.recordValue(System.nanoTime() - startNanos);
</#if>
<#else>

<#if sequential??>
switch((int) (iteration % ${timeStepMethods?size})){
<#else>
switch(probs[random.nextInt(probs.length)]){
</#if>
<#list timeStepMethods as method>
<#assign index = method?counter-1>
case ${index}:
Expand Down
Loading
Loading