[Feat] Memory Profiling and Automated Performance Regression Testing

[yuvraajnarula]

Description

Following the introduction of runtime scaling and flamegraph benchmarks in #117, I suggest we can expand the diagnostic suite to handle two critical areas: Memory Usage and Regression Prevention.

Proposed Enhancements

1. Memory Profiling Integration

• Goal: Track peak memory consumption alongside runtime.
• Implementation: Integrate a tool like memory-profiler or tracemalloc into the existing benchmark CLI.
• Value: Helps identify if high-resolution grids ($10^7$+ nodes) require a transition from networkx to more memory-efficient data structures (e.g., adjacency matrices or sparse tensors).

2. Automated Performance Regression

• Goal: Prevent performance "drift" where new features accidentally slow down graph construction.
• Implementation: * Add a --check flag to the scaling script.
  • Define a "performance budget" for specific node counts (e.g., $10^5$ nodes must be processed in under 10s).
  • Integrate this check into the CI pipeline (GitHub Actions) to flag PRs that significantly exceed the runtime budget.

3. Machine-Readable Metrics (optional)

• Goal: Enable long-term tracking of performance trends.
• Implementation: Add an --output-json or --csv flag to the benchmark scripts so results can be stored and compared across different versions of the codebase.

Context & Links

• Initial Benchmarks: Established in [Feat] runtime scaling and flamegraph benchmark CLIs #117.
• Target Scale: Ensuring stability for $10^7$ node operations.
• Dev Meeting: Discussed on 2026-04-13 regarding the need for deeper hardware utilization insights.

Task List

• Add memory tracking to graph_creation_scaling.
• (optional) Implement JSON/CSV export for benchmark results.
• Define baseline performance thresholds for CI.
• (Optional) Add a "Comparison Mode" to benchmark current branch vs main.

[yuvraajnarula]

@leifdenby and @joeloskarsson
If you guys think this is fine, I would love to work on Leif's branch as an extension

[leifdenby]

Looks good @yuvraajnarula! A few thoughts:

• I would create a memory-scaling script first, i.e. this would be an interactively run script to determine how memory usage scales with the number of grid nodes
• automated performance regression testing sounds great, but I think we will need to make the benchmarks output machine-readable metrics first (since the regression tests will need to access the numerical values). These won't be scaling tests though, but will pick a point on the scaling plots that we don't want to change.
• So how about splitting this into 2 PRs:
  1. Memory-scaling profiling script. These could sit along-side the runtime-scaling scripts I have already done, or be integrated into that same PR. It maybe makes most sense to do both runtime and memory scaling at the same time. Happy for you to take over my PR, or we can close mine and you can open a new one that takes my branch and adds memory profiling plots. As part of this work you could refactor my code to output what is being plotted (e.g. runtime vs number of gridpoints, memory usage vs number of gridpoints). Instead of using CSV/json we could just return these scaling results from a function, this function could then be used in CI later.
  2. Separate PR that adds CI action to run a runtime_memory_regression_test, that in turn runs the memory_runtime_scaling_benchmark but only with a single number of grid-points. This will need tweaking for the CI platform etc, so there will be some iteration we have to do here. Which is why I am thinking we should do this in a separate PR.

What do you think?

In terms of prioritizing work though I would actually say working on getting the runtime down is more important that adding more tooling for measuring runtime/memory scaling functionality. But maybe you feel like it would be best to get the functionality in that you are suggesting here, before we start working on actually making the code run faster or use less memory? :)

[yuvraajnarula]

Thank you, Leif, for the detailed breakdown.

On the two‑PR split: Agreed. I'll take your runtime‑scaling branch as a starting point, extend it to also profile memory, and refactor the output to return the raw data (runtime vs gridpoints, memory vs gridpoints) instead of just plotting. That gives us a reusable function for both interactive analysis and CI.

On priorities: You're right, reducing runtime/memory is the ultimate goal. The tooling (benchmarking + regression tests) is a means to that end. I'd argue we need the tooling first to measure the impact of any optimisation we later introduce. Without baselines and automated checks, we risk regressions or waste time on ineffective changes.

So I propose:

1. First: Memory + runtime scaling script (this issue, PR 1). Establish baselines on main.
2. Then: Optimise the graph generation (e.g., vectorisation, data structures), happy to help there too.
3. Finally: CI regression test (PR 2) to protect the gains.

If you'd rather I focus directly on optimisation now, I can do that, just let me know. Either way, I'm happy to lead the scaling tooling as a foundation step.
@leifdenby
I'll open a draft PR soon that builds on your branch and adds memory profiling. Let me know if you want me to close #137 and work directly from your existing branch.

[leifdenby]

Just picking this up now @yuvraajnarula because I think it would be great to your work here on the roadmap. Could you have a look at #140 and add a bit more detail for me?