Fix 2-core deadlock in trust-estimation cooperative thread join

[AlipourIm]

Summary

• Replace native Thread.join() in JmcThread with cooperative JmcRuntime.waitForTaskTermination(), so stop-all during trust-estimation can wake joiners instead of leaving OS threads stuck outside JMC pause/resume.
• Harden the scheduler and trust-estimation strategy against null scheduling choices and stalls when every live task is paused waiting for a decision.

Root cause

On 2-core CI runners (taskset -c 0,1), EstimationSynTest.runIncnTrustEstimation could hang indefinitely. Native Thread.join() blocked OS threads that were no longer tracked in TaskManager.taskFutures, so trust-estimation stop-all could not resume them. On machines with more cores the race often did not reproduce.

Test plan

• EstimationSynTest.runIncnTrustEstimation in Docker with --cpus=2 + taskset -c 0,1 (294 and 310 iterations, ~15–22s each; previously timed out at 180s)
• ./gradlew :core:test

this reminded me why I hate Java so so much

[joulook]

Dear @AlipourIm,

Thank you very much for considering and helping us address the issue I mentioned regarding the 2-core setting. The first time I attempted to tackle this issue using Claude Code, it suggested the same idea of implementing a cooperative join mechanism. We initially avoided this approach to enhance performance during exploration, which you noted after implementing it, as each test took 15-22 seconds.

However, after a careful investigation of the complete log files from JMC running in debug mode, I realized that the issue is not related to the join mechanism in JCM's runtime. Instead, the problem lies within the doNextStop method of the scheduler. In this method, the scheduler attempts to terminate each task (thread) running in the program under test using the taskManager. When the scheduler enters this mode, it sets a flag (stopAllMode) to indicate this state. Furthermore, it ensures that the main thread of the program under test (which always has the threadId of 1) is the last thread to be stopped, after which the scheduler completes the stopAllMode.

However, the logic in this method was such that it first requested the taskManager to terminate the task and then checked if it was the main thread to reset the stopAllMode flag. This race condition had not been encountered before, as we had tested our tool and benchmarks on platforms with more than 8 cores. In those systems, the scheduler thread was likely able to reset the flag before the main thread completed its execution and started the next iteration of testing. However, on a 2-core platform, the scheduler thread could be slower due to limited resources, resulting in a loss in the race.

In PR #4, I addressed this issue by reordering these operations, thereby resolving the data race without sacrificing performance. Testing concurrent programs under restricted resources also led me to identify another data race in the taskManager, which I also addressed in the same PR. This experience taught me that concurrent programs must be tested on various platforms with different resource constraints.

Regarding your last comment, I’m sorry to hear that you don’t enjoy working with Java. I hope you can find a better way to engage with it and appreciate its features and beauty. If this bug or any similar issues have contributed to your dislike of Java, it’s important to remember that the existing nondeterminism in concurrent programming is often the culprit, rather than the language itself. This underscores the necessity of developing tools like JMC to make concurrent programming a more enjoyable experience for programmers :)

Thank you once again for your contributions to JMC and for helping us improve it. I eagerly look forward to seeing more PRs from you on this project.

All my best,
Mohammad