Upgrading fermipy to numpy>2 and astropy>7#679
Open
facero wants to merge 2 commits into
Open
Conversation
Fill missing (masked) values in key columns before joining In astropy >v6 table.join() strictly rejects missing values in key columns while it was cool with this in astropy v5.
Contributor
Author
|
if the tests are running with numpy 1.26, they might fail. One would need to upgrade the CI environment to astropy >7, numpy 2>, gammapy>2 |
Contributor
Author
python 3.9 could be removed from CI |
Contributor
|
Thanks @facero! I just dropped support for which is called by |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR proposes to fix remaining issues with numpy2 and also issues that I discovered with astropy 6&7.
This follows the PR fixing numpy2 in the fermitools and this PR should be tested in a numpy and astropy 7 environment in conjunction with the fermitools numpy2 version.
Issues solved:
All issues (for now) were related to
catalog.py.I haven't yet tested a longer fermipy analysis than just
gta.setup().I will try later and report if any other bugs are found.
So far this is what I've done.
Numpy 2 related
As numpy2 deprecated the
np.string_call here was failing.the fix was to use :
(np.bytes_, np.str_)to handle both string and bytes columnsMaybe just
np.str_was enough. But at least we should be safe with both.Astropy >6 related
In astropy >6 ,
table.join()here strictly rejects missing values in key columns. The join_tables() function in catalog.py was calling join() directly on tables where the left key column (Extended_Source_Name) had masked/NaN values for non-extended sources, causing:astropy.table.np_utils.TableMergeError: Left key column 'Extended_Source_Name' has missing valuesThe fix here is more painful and I'm not completly sure that the proposed solution is the best.
The solution is to add a pre-processing step before the
join()call that fills masked values in the key column for both tables with 0 or empty strings.Maybe this entire
join_tablesfunction could be revisited and simplified as it is a bit cryptic now.But let's try this fix first.