[Bug] Tune driver RSS grows linearly with trial count due to orphaned logger state

[goingforstudying-ctrl]

Was running a large hyperparam sweep with tune.run() and noticed the driver process RSS just kept climbing. After a few thousand trials it was eating multiple GBs and eventually got killed by the OOM killer.

I dug into the per-trial accumulators and found three separate leaks layered on top of each other:

1. CSVLoggerCallback keeps a _trial_continue dict to know whether to write a CSV header. The entry is never removed when the trial ends, so it grows linearly with the number of trials. Same issue in JsonLoggerCallback with _trial_configs.

2. TuneController queues scheduler decisions in _queued_trial_decisions, but if a trial stops or fails outside the normal result path (e.g. via stop_trial() or an exception), the queued entry never gets popped by _maybe_execute_queued_decision. It just sits there forever.

3. Trial.get_json_state() serializes trial state with indent=2 for pretty printing. This is the largest per-trial blob — it's cached in memory and rewritten to the experiment-state checkpoint file every period. Dropping the indentation saves about 25-30% on both the retained footprint and the on-disk experiment state. Since this JSON is machine-read anyway, the pretty formatting is just waste.

The first two are straightforward — just clean up the dict entries in log_trial_end() and _schedule_trial_stop(). The third is a one-line change to remove indent=2 from json.dumps.

I also added a test file (test_logger_memory_leak.py) that spins up a bunch of mock trials, runs them through the loggers, and asserts the internal dicts are empty afterwards. Caught all three issues during development.

Not 100% sure if removing indent=2 breaks any downstream tools that parse the experiment state checkpoint as plain text. From what I can tell it's always loaded back via json.loads, so it should be fine. Let me know if there's a tool I'm missing.

This is the same thing reported in #64231.

[gemini-code-assist]

Code Review

This pull request addresses memory leaks in Ray Tune when running thousands of trials by cleaning up trial-related state in the controller and logger callbacks upon trial completion, and by using compact JSON serialization for trial state checkpoints. The feedback suggests extending the controller cleanup to include _cached_trial_decisions when a trial is not saving, and further optimizing the JSON serialization by using compact separators to minimize memory footprint.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

In addition to cleaning up _queued_trial_decisions, we should also clean up _cached_trial_decisions when the trial is not saving. If a trial is stopped or fails outside the normal path while not saving, any entry in _cached_trial_decisions will be orphaned and leak memory, as the trial actor is terminated and no further results or saves will be processed to trigger its cleanup.

Suggested change

	# Clean up any queued decision for this trial to prevent memory leak
	# when trials stop or fail outside the normal result path (see #64231)
	self._queued_trial_decisions.pop(trial.trial_id, None)
	# Clean up any queued or cached decision for this trial to prevent memory leak
	# when trials stop or fail outside the normal result path (see #64231)
	self._queued_trial_decisions.pop(trial.trial_id, None)
	if not trial.is_saving:
	self._cached_trial_decisions.pop(trial.trial_id, None)

[pseudo-rnd-thoughts]

@goingforstudying-ctrl Is gemini right for

In addition to cleaning up _queued_trial_decisions, we should also clean up _cached_trial_decisions when the trial is not saving. If a trial is stopped or fails outside the normal path while not saving, any entry in _cached_trial_decisions will be orphaned and leak memory, as the trial actor is terminated and no further results or saves will be processed to trigger its cleanup.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 0694bac. Configure here.}

[pseudo-rnd-thoughts]

Thanks for the changes, it overall looks good. Do you have a minimal example script to reproduce the original issue? Do you have a graph of the memory usage over time with the Ray 2.55 and this PR?

Please reduce the length of comments that are LLM generated. For the test added, how long does it take to run?

[pseudo-rnd-thoughts]

@goingforstudying-ctrl Is gemini right for

In addition to cleaning up _queued_trial_decisions, we should also clean up _cached_trial_decisions when the trial is not saving. If a trial is stopped or fails outside the normal path while not saving, any entry in _cached_trial_decisions will be orphaned and leak memory, as the trial actor is terminated and no further results or saves will be processed to trigger its cleanup.