GH-50037: [Python] test_table_uses_memory_pool flaky on macOS 14 job

[tadeja]

Rationale for this change

Fixes #50037

What changes are included in this PR?

Add gc.collect() before prior_allocation = pa.total_allocated_bytes() in test_table_uses_memory_pool
and also add to _check_to_pandas_memory_unchanged helper with the same pattern.
Same fix as #44793

Are these changes tested?

Not reproduced locally yet.

Are there any user-facing changes?

No.

• GitHub Issue: [Python][CI] test_table_uses_memory_pool flaky on macOS 14 job #50037

[github-actions]

⚠️ GitHub issue #50037 has been automatically assigned in GitHub to PR creator.

[tadeja]

Recent occurrence June 1st: https://github.com/apache/arrow/actions/runs/26750696954/job/78837634838#step:10:218

[tadeja]

Root cause + deterministic repro

Scanned the suite with Opus and found the "polluter" is parquet/test_dataset.py. Its dataset/filter tests leave small cyclic buffers (64–768 B) reclaimed only gc.collect(), which accumulate in the xdist worker. Occasionally a remnant survives auto-GC into test_table_uses_memory_pool. THere gc.collect() frees it so final total drops below the baseline at start of the test causing AssertionError in CI. Since pa.total_allocated_bytes() is process-global, the fix is to take the baseline after a gc.collect(), same as test_array_uses_memory_pool and #44793.

Repro (disabling auto-GC just removes the rarity, mechanism is unchanged)
File nogc.py:

import gc
def pytest_configure(config): gc.disable()

python -m pytest -n0 -q -p nogc \
  pyarrow/tests/parquet/test_dataset.py \
  pyarrow/tests/test_pandas.py::test_table_uses_memory_pool

--
On main (no fix):

FAILED ../pyarrow/tests/test_pandas.py::test_table_uses_memory_pool - assert 0 == 4736
1 failed, 56 passed, 6 skipped, 1 xfailed in 0.75s

With this PR fix (gc.collect() also before the baseline) makes baseline 0 and tests pass:
57 passed, 6 skipped, 1 xfailed in 0.66s

[AlenkaF]

Thanks for looking into this @tadeja!
I think this is worth merging and keeping track if any more flaky failures happen.

@raulcd any thoughts?

[raulcd]

I agree, I think this is reasonable. Thanks @tadeja

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit bfc9cdb.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.