Add stress testing script for multiple concurrent jobs

[AdarshAlex751]

I finished the stress testing script and tested the system with 2, 5, and 10 concurrent jobs. All the jobs were submitted successfully, the worker processes finished without any errors, and the coordinator stayed responsive.

One issue I noticed is that the jobs stayed in the QUEUED state even after the workers completed and notified the coordinator. I didn't change this since it seems to be a coordinator issue and not included in my task.

[Copilot]

Pull request overview

Adds a standalone stress-test utility script to exercise the DistroML coordinator/worker stack by submitting multiple jobs concurrently, launching local worker processes for each job, polling /api/jobs for status, and emitting logs + a JSON summary report.

Changes:

• Introduces scripts/stress_test_multiple_jobs.py to submit N jobs concurrently and run N local workers.
• Adds periodic status polling and a JSON report (latencies, outcomes, per-job details) written to stress_test_logs/.

Suppressed comments (3)

scripts/stress_test_multiple_jobs.py:288

• If launching one of the workers fails, the script currently aborts before entering the polling try/finally, leaving any already-started worker processes running and their log files open. Consider handling per-worker launch failures by terminating previously-started workers, closing log handles, and returning a non-zero exit code.

    for job in submitted:
        process, log_file = launch_worker(
            job,
            coordinator_url,
            args.steps,
            log_directory,
        )

scripts/stress_test_multiple_jobs.py:379

• After adding a dedicated cancelled count, include it in the JSON report so the summary matches the printed/exit behavior and doesn't lump cancellations into stuck_or_unknown_jobs.

    report = {
        "requested_jobs": args.jobs,
        "submitted_jobs": len(submitted),
        "submission_failures": submission_errors,
        "completed_jobs": completed,
        "failed_jobs": failed,
        "stuck_or_unknown_jobs": stuck,
        "steps_per_job": args.steps,

scripts/stress_test_multiple_jobs.py:413

• If cancelled is tracked separately, the script should still exit non-zero when any job is cancelled (since it isn't a successful completion). Update the exit condition to include cancelled == 0.

    return 0 if failed == 0 and stuck == 0 else 1

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.