Skip to content

[fix/enhancement] move SN_models out of final_values#848

Open
maxbriel wants to merge 16 commits into
v2.3from
mb_post_processing_fix
Open

[fix/enhancement] move SN_models out of final_values#848
maxbriel wants to merge 16 commits into
v2.3from
mb_post_processing_fix

Conversation

@maxbriel

@maxbriel maxbriel commented May 18, 2026

Copy link
Copy Markdown
Collaborator

We've likely hit the maximum number of columns in a single HDF5 dataset for the final_values with the increased number of SN models.
As such, we need to separate out the SN_models to make sure we will not hit the same issue in the future.

I've rewritten the add_post_processed_quantities to:

  1. Use the faster recfuntions from numpy to add all columns in one go to the structure array, final_values. Originally, we were writing one column at the time. deleting the existing final_values and then writing the new final_values with 1 extra column. Repeating this process for each new column we wanted to add (several hundred times given the number of SN_MODELS we have). This is a massive I/O overhead.
  2. Store the SN_MODELS in their own group (grid/SN_MODELS) in the HDF5 file. Each SN_MODEL has its own dataset + in the attrs the parameters for the model are stored.
  3. grid.compression_args are not actually populated correctly when loading a grid with grid(path_to_file) as such when writing the "new" datasets to file, they were uncompressed.

Caution

Bullet point 3 means that the final_values dataset in the release dataset are uncompressed!. I verified that this is the case!

Screenshots of uncompressed final_values

final_values
image

initial_values
image

This PR now also changes the IF_interpolator and interpolator. If required, the IF_interpolator now uses _get_grid_column to get the final_values or SN_models. It only does this if it looped over the SN_MODELS before.

Changes to step_MESA is just how we access the SN_MODEL data. It should be checked if this stored everything exactly the same as before. step_SN is able to access this correctly at the moment.

Warning

There is a bug I noticed with the LazyHDF5 accessing the HDF5 file when adding a column. When adding something to the HDF5f file, we reload the file, e.g. grid._reload_hdf5_file(). This invalidates the LazyHDF5 linking. We should fix this, if possible, but not in this PR.

@maxbriel maxbriel changed the title final_values move SN_models out [fix/enhancement] move SN_models out of final_values May 18, 2026
@maxbriel maxbriel marked this pull request as ready for review June 3, 2026 14:24
@maxbriel maxbriel self-assigned this Jun 4, 2026
@sgossage sgossage self-requested a review June 10, 2026 01:02
@maxbriel

Copy link
Copy Markdown
Collaborator Author

Here are the output files from the binaries that I ran with the fix and v2.3. I originally posted these on Slack and forgot to add them here:

evolve_1Zsun_fix.log
evolve_1Zsun_main_.log

Comment thread bin/posydon-run-pipeline
Comment thread bin/posydon-run-pipeline
Comment on lines 408 to 410
# TODO: we need to train core collapse for secondary star as well
# to catch reverse mass transfer cases where the secondary undergoes
# core collapse first

@sgossage sgossage Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep this in mind as a to do item. (Not necessarily for this PR though.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants