Added `splice` module by ladsantos · Pull Request #152 · spacetelescope/stistools

ladsantos · 2023-03-01T14:18:08Z

Added the splice module, which is based on the standalone code stissplice, including a simple test suite and documentation.

…into dev_ldsantos

sean-lockwood · 2023-03-01T23:00:33Z

Testing would be more robust if you upload an input file and a "truth" output file. Then you could compare all data values.

I'm concerned that testing one flux value might not be sensitive to changes in all of your code's logic.

ladsantos · 2023-03-02T16:43:42Z

Improved the test suite of splice in commit bece227. Now the whole spectrum is tested against a truth file.

sean-lockwood · 2023-03-02T16:58:59Z

Leo,

Will most users want to use stistools.splice.splice() or stistools.splice.splice_pipeline()?

I'm wondering if splice_pipeline should be renamed to splice and the original splice to something like splice_overlap (or something else descriptive). I know that if I were trying it for the first time, I would either check stistools.splice() (which is actually module-level and not how routines are stored in stistools) or stistools.splice.splice, following the pattern set by most of the other routines.

…he other `stistools` modules

sean-lockwood · 2023-03-14T18:58:15Z

Docstrings should follow numpy/scipy docstring format:

Current:

    Parameters
    ----------
    unique_spectra_list (``list``):
        List of unique spectra.

    merged_pair_list (``list``):
        List of merged overlapping pair spectra.
    ...

Suggested:

    Parameters
    ----------
    unique_spectra_list : list
        List of unique spectra.

    merged_pair_list : list
        List of merged overlapping pair spectra.
    ...

Note that the colons won't currently render properly on RTD, but we're working on that in a separate PR.

…ting scheme

sean-lockwood · 2023-03-31T19:18:09Z

+    dq_weights_interp = np.zeros_like(dq_interp)
+    # And then for each acceptable dq, if the element of the dq array is one
+    # of the acceptable flags, we set its dq weight to one
+    for adq in acceptable_dq_flags:


Should probably check bitwise, i.e.:

np.where(dq_ref & adq)

and

np.where(dq_interp & adq)

In case more than one flag is present at a particular location. Perhaps this doesn't occur with the default flags, but could with other options provided by the user.

In fact, you can probably skip the loop over the various acceptable_dq_flags values and compare to the scalar value that is the bitwise-or or the adq flags (sum in this case).

Ok, so wrapping it up together:

from functools import reduce acceptable_dq_flags = reduce(np.bitwise_or, acceptable_dq_flags) # scalar value dq_weights_ref[np.where(dq_ref & acceptable_dq_flags)] = 1 dq_weights_interp[np.where(dq_interp & acceptable_dq_flags)] = 1

(Should be tested.)

It seems that this code snippet catches all the good flags, except for 0. But I should be able to add a line or two that catches it.

Good catch! We don't typically flag acceptable pixels, so that slipped by.

Perhaps:

bad_dq_flags = ~reduce(np.bitwise_or, acceptable_dq_flags) # scalar value dq_weights_ref = np.ones(..., dtype=...) dq_weights_interp = np.ones(..., dtype=...) dq_weights_ref[dq_ref & bad_dq_flags] = 0 dq_weights_interp[dq_interp & bad_dq_flags] = 0

Looking at it now, we probably don't need the np.where either.

…one pixel

…into dev_ldsantos

sean-lockwood · 2024-02-16T19:23:59Z

@ladsantos -
Sorry, I lost track of where we were on this. Do you think this is ready for some sanity checks and almost ready to merge?

ladsantos · 2024-02-16T19:28:59Z

Hi @sean-lockwood, no worries! It is good to go for sanity checks and merge.

sean-lockwood · 2024-02-26T19:05:11Z

@ladsantos -
We just did a quick test on oewia2010_x1d.fits and found the splice code is introducing spikes at the end of some orders. Applying DQ filtering doesn't seem to help.

Full range:

cc: @Jackie-Brown

sean-lockwood · 2024-03-12T14:50:24Z

I'm noticing near the spikes there are a lot of DQ = 32768 = 2**15, which isn't defined in the STIS DQ values.

Might be indicative of a problem.

ladsantos · 2024-05-23T19:28:20Z

It seems to me that splice is not really "introducing" these spikes, they are already there in the x1d files in the first place. It is difficult to correct them without manual inspection/manipulation. If I change the weight parameter to 'snr' instead of 'sensitivity' (which is the default), and I remove the DQ flag 2048 as an acceptable flag, the results are much better -- although not completely alleviated.

ladsantos · 2024-05-23T19:36:27Z

Also, the DQ flag 32768 simply means a merged pixel where there was an overlap.

ladsantos · 2024-05-23T19:43:02Z

Here's a closer look at the spikes at longer wavelengths (spliced spectrum is offset for visualization purposes). It seems to me that the issue lies in the x1d orders not overlapping, so there is no merging to be done to correct for the edge effects. In such cases, there is not much that the splice code can do besides just concatenating the orders.

…laps

…into dev_ldsantos

ladsantos · 2024-05-24T15:39:42Z

Found a solution to the spikes near the order edges. The new version of the code simply adds a DQ flag of 4096 at a user-specified edge truncation number. Note that it does not fix spikes that are present in the x1d data when there is no overlap.

sean-lockwood · 2024-08-14T21:44:36Z

@ladsantos -

The spikes on the right have DQ = (512 | 2048), whereas the ones on the left are 2^15.

I think making the current DQ 2^15 regions be (2^15 | 2048) makes the most sense, as we'd be able to filter all the spikes using the prior definition of SDQFLAGS (preserving behavior in legacy code) and your distinction using 2^15 would still be present.

(Pending discussion with the team.)

sean-lockwood

I'm still looking at the DQ behavior in more detail, but here are a few issues I noticed today.

sean-lockwood · 2024-08-15T22:11:14Z

+    # We interpolate the lower-SNR spectra to the wavelength bins of the higher
+    # SNR spectrum.
+    max_sens_idx = np.where(avg_sensitivity == np.nanmax(avg_sensitivity))[0][0]
+    overlap_ref = overlap_sections.pop(max_sens_idx)


This has the side effect of modifying the input variable. You should probably make a copy inside the function instead.

Corrected in commit a2f5407.

sean-lockwood · 2024-08-15T22:12:26Z

+    if truncate_edge_right is not None:
+        for sk in spectrum:
+            sk['data_quality'][-truncate_edge_right:] = 4096
+    elif truncate_edge_left is not None:


elif -> if allows both edges to be truncated with default values. Is that the desired behavior?

Corrected in commit a2f5407.

sean-lockwood · 2024-08-15T22:12:54Z

+
+    # Merge the overlapping spectral sections
+    merged_pairs = [
+        merge_overlap(overlap_pair_sections[k], acceptable_dq_flags)


Should the weight parameter be passed here too?

Corrected in commit a2f5407.

sean-lockwood · 2024-08-15T22:31:10Z

+        overlap_s_u = np.copy(overlap_sections[i]['uncertainty'])
+        overlap_s_n = np.copy(overlap_sections[i]['net'])
+        overlap_s_dq = overlap_sections[i]['data_quality']
+        where_dq_bad = np.where(overlap_s_dq & bitwise_or_acceptable_dq_flags)


If bitwise_or_acceptable_dq_flags are the flags that we're ok ignoring, shouldn't we use the inverse here to find bad locations?

unacceptable_dq_flags = int(~np.uint16(bitwise_or_acceptable_dq_flags)) ... where_dq_bad = (overlap_s_dq & unacceptable_dq_flags) != 0

I tried this, and it does not remove the spikes on the right. I think there's something we are missing with the logic here (see my comment below).

ladsantos · 2025-01-31T21:33:41Z

@ladsantos -

The spikes on the right have DQ = (512 | 2048), whereas the ones on the left are 2^15.

I think making the current DQ 2^15 regions be (2^15 | 2048) makes the most sense, as we'd be able to filter all the spikes using the prior definition of SDQFLAGS (preserving behavior in legacy code) and your distinction using 2^15 would still be present.

(Pending discussion with the team.)

The 2^15 DQ flag represents pixels that were co-added from good pixels (those with acceptable flags). This is odd, because the 2048 and 512 are not an acceptable DQ flag, so it shouldn't be co-adding the DQ = (512 | 2048) pixels. Or am I getting this logic wrong?

sean-lockwood · 2026-06-25T21:39:12Z

I've been working on splice and uploaded a draft (still in progress) to this branch:
https://github.com/spacetelescope/stistools/tree/sl_splice

Among other changes, this fixes DQ flags and implements sensitivity weighting derived from reference files. Algorithm improvements and tests are still in progress.

ladsantos added 5 commits February 28, 2023 14:42

First commit of the splice module

598ee17

Merge branch 'spacetelescope:master' into dev_ldsantos

fb4b4f2

Added documentation for the splice module

0c7da4b

Merge branch 'dev_ldsantos' of https://github.com/ladsantos/stistools …

fe21f19

…into dev_ldsantos

Merge branch 'spacetelescope:master' into dev_ldsantos

637460b

ladsantos added the enhancement label Mar 1, 2023

Updated the test suite for splice

bece227

Small correction in the docs of splice

0bd3287

ladsantos added 2 commits March 2, 2023 12:09

Renamed some functions in the splice module to be consistent with t…

19f9ba7

…he other `stistools` modules

More small corrections to the docs of splice

1dcdbe4

Corrected docstrings, added note about ULLYSES, implemented new weigh…

925b033

…ting scheme

sean-lockwood reviewed Mar 31, 2023

View reviewed changes

ladsantos added 2 commits April 6, 2023 14:04

Changed how the DQ flags are dealt with in interpolated pixels

353a7ac

Changed how the wavelength grid of the merged section is calculated

507634e

ladsantos requested a review from a team as a code owner April 25, 2023 15:08

ladsantos added 5 commits April 28, 2023 13:29

Corrected bug that occurred when there is a unique section with only …

613353b

…one pixel

Fixed bug that occurred when there are zero fluxes in spectrum

d4112bd

Fixed bug that occurred when raising error about weighting mode

5a4bf9c

Merge branch 'spacetelescope:master' into dev_ldsantos

a427b8d

Merge branch 'dev_ldsantos' of https://github.com/ladsantos/stistools …

e8c1b50

…into dev_ldsantos

sean-lockwood requested a review from Jackie-Brown February 12, 2024 20:37

ladsantos added 2 commits May 24, 2024 11:36

Added a new feature to set an order-edge truncation when merging over…

bfe6bd5

…laps

Merge branch 'dev_ldsantos' of https://github.com/ladsantos/stistools …

604bf60

…into dev_ldsantos

sean-lockwood requested changes Aug 15, 2024

View reviewed changes

sean-lockwood reviewed Aug 15, 2024

View reviewed changes

Small bug corrections suggested by S. Lockwood on Aug 15, 2024

a2f5407

Uh oh!

Conversation

ladsantos commented Mar 1, 2023

Uh oh!

sean-lockwood commented Mar 1, 2023

Uh oh!

ladsantos commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sean-lockwood commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sean-lockwood commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sean-lockwood Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladsantos Apr 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sean-lockwood commented Feb 16, 2024

Uh oh!

ladsantos commented Feb 16, 2024

Uh oh!

sean-lockwood commented Feb 26, 2024

Uh oh!

sean-lockwood commented Mar 12, 2024

Uh oh!

ladsantos commented May 23, 2024

Uh oh!

ladsantos commented May 23, 2024

Uh oh!

ladsantos commented May 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ladsantos commented May 24, 2024

Uh oh!

sean-lockwood commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sean-lockwood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladsantos commented Jan 31, 2025

Uh oh!

sean-lockwood commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ladsantos commented Mar 2, 2023 •

edited

Loading

sean-lockwood commented Mar 2, 2023 •

edited

Loading

sean-lockwood commented Mar 14, 2023 •

edited

Loading

sean-lockwood Mar 31, 2023 •

edited

Loading

ladsantos Apr 5, 2023 •

edited

Loading

ladsantos commented May 23, 2024 •

edited

Loading

sean-lockwood commented Aug 14, 2024 •

edited

Loading