Unify q-coordinate semantics between NRSS simulation outputs and PyHyperScattering reduction workflows

[delongchamp]

Unify q-coordinate semantics between NRSS simulation outputs and PyHyperScattering reduction workflows

Summary

There is a persistent mismatch between how q-coordinates are handled in:

• conventional experimental RSoXS reduction workflows in PyHyperScattering, and
• current NRSS simulation reduction workflows, especially when NRSS outputs are reduced with WPIntegrator.

The core problem is that experimental data and simulated data are not currently passing through equivalent reduction semantics.

For SST1 / conventional RSoXS data, PyHyperScattering uses pyFAI-backed integrators (PFGeneralIntegrator / PFEnergySeriesIntegrator) that interpret detector geometry and produce a corrected chi,q representation from raw detector pixels plus geometry metadata.

For NRSS outputs, the common workflow instead uses FFT-derived qx/qy coordinates and WPIntegrator, which performs a warp-polar remesh and labels the radial axis as q = sqrt(qx^2 + qy^2). That is a q_perp remesh, not a detector-geometry-aware reduction into corrected q-space.

This becomes especially important for:

• 3D NRSS simulations, where the virtual instrument is geometry-sensitive and the effective sample-detector distance varies trigonometrically with detector position,
• form-factor validation tests, where analytical references are currently adjusted into detector coordinates to match the reduction path, and
• attempts to compare simulation and experiment under a single interpretation of "q".

I've determined that the entire corpus of NRSS 3D work to this point is contaminated by this issue; it affects physical realism more strongly at higher q and might explain why some of our simulation data don't match as well at high q. I think this happened because an early (likely my own) misunderstanding / failure-to-RTFM about what WPIntegrator was doing became proliferated in later workflows without re-examination. An additional issue is that the WPIntegrator are indeed correct for NRSS 2D simulations, which we used a lot in the early days and likely hid the issue. We will correct and move forward!

Problem statement

We currently have three distinct q concepts in play:

1. Experimental detector pixels

   • Raw images on a planar detector.
   • Correct reduction requires geometry-aware mapping from pixel position to scattering vector coordinates.

2. NRSS / CyRSoXS FFT reciprocal-plane coordinates

   • Output qx/qy are generated from PhysSize and array dimensions.
   • These are reciprocal-grid coordinates derived from FFT sampling, not inherently the same thing as beamline-reduced q-space.

3. WPIntegrator radial remesh coordinates

   • WPIntegrator remeshes qx/qy images via warp_polar.
   • Its output radial axis is derived from sqrt(qx^2 + qy^2).
   • This is effectively a q_perp axis, not a detector-geometry-corrected |q| axis.

These three are often treated interchangeably in downstream discussions and tests, but they are not equivalent.

Why this is a problem

1. Simulation and experiment are being reduced through different semantics

Experimental SST1 RSoXS data in PyHyperScattering typically follows this path:

raw detector pixels -> geometry metadata -> pyFAI integrator -> chi,q

NRSS outputs in most workflows - the MWCNT tutorials are an example - currently follow this path:

simulation qx/qy -> WPIntegrator -> chi,q_perp

Those are not equivalent reduction pipelines.

2. The ambiguity is hidden by naming

The radial coordinate returned by WPIntegrator is named q, which encourages users to interpret it as corrected q-space. In reality it is a radial remesh over the provided qx/qy grid.

3. 3D NRSS simulation physics and current reduction labels can diverge

The modern NRSS 3D backend (both cyrsoxs and new cupy variants) includes detector-projection geometry internally during intensity formation, but the returned coordinate labels are still uniform FFT-style qx/qy. If these outputs are then passed through WPIntegrator, the intensity was produced with geometry-aware projection, but the radial remesh is still labeled with q_perp.

In other words, the NRSS 3D simulation is a true virtual instrument that produces what a planar detector would "see" and its q-coordinates need to be corrected for the virtual geometry. They are not being corrected with WPIntegrator.

This can produce a workflow where:

• the projection physics is more realistic than a naive reciprocal-plane interpretation,
• but the exported radial coordinate still does not represent the corrected magnitude of the scattering vector.

This is further complicated by that fact that 2D NRSS simulations do not need correction and receive correct coordinates from WPIntegrator since there is no 3-dimensionality to the Ewald volume that is created in those sims. That is

4. Form-factor tests currently encode the mismatch rather than resolving it

Some validation tests currently compare simulation data reduced with WPIntegrator against analytical references that are adjusted into flat-detector coordinates. This is valid as a detector-coordinate comparison, but it is not representative of how conventional experimental data should be reduced for comparison.

That makes the tests useful, but semantically narrower than they appear.

Evidence from current code and tutorials

Experimental SST1 / conventional RSoXS workflow

PyHyperScattering SST1RSoXSDB.loadRun() returns raw images with pixel dimensions and pyFAI-style geometry metadata:

• pix_y, pix_x
• dist
• pixel1, pixel2
• poni1, poni2
• rot1, rot2, rot3
• wavelength

Relevant files:

• PyHyperScattering/src/PyHyperScattering/SST1RSoXSDB.py
• PyHyperScattering/src/PyHyperScattering/PFGeneralIntegrator.py
• PyHyperScattering/src/PyHyperScattering/PFEnergySeriesIntegrator.py

The pyFAI-backed integrators then perform geometry-aware integration into chi,q.

Legacy / reference linear q conversion exists but is not the active reduction path

SST1RSoXSLoader.loadSingleImage(..., return_q=True) computes a linearized q-per-pixel conversion from pixel size, wavelength, and SDD. The databroker version preserves a similar path only as a disabled / reference method.

This is not the same as the standard pyFAI reduction path and should not be treated as the authoritative source of SST1 reduced q-space.

NRSS / CyRSoXS output workflow

NRSS simulation outputs and the PyHyperScattering CyRSoXS loader construct qx/qy from FFT frequencies and PhysSize.

Relevant files:

• NRSS/src/NRSS/morphology.py
• PyHyperScattering/src/PyHyperScattering/cyrsoxsLoader.py

This is a reciprocal-grid description of the simulated detector plane, not an experimental beamline reduction.

WPIntegrator behavior

WPIntegrator:

• finds the center by interpolating where qx=0 and qy=0,
• performs warp_polar,
• assigns a radial axis from 0 to max(sqrt(qx^2 + qy^2)).

Relevant file:

• PyHyperScattering/src/PyHyperScattering/WPIntegrator.py

This is a radial remesher for existing qx/qy images. It does not apply planar-detector q correction.

Illustrative example: NRSS MWCNT tutorial split

The NRSS MWCNT experimental notebook uses PFEnergySeriesIntegrator for experimental WAXS/SAXS data.

The NRSS MWCNT simulation notebook uses WPIntegrator on morph.run(...) output.

Relevant files:

• NRSS/src/NRSS_tutorials/MWCNTs/nb1_rsoxs.ipynb
• NRSS/src/NRSS_tutorials/MWCNTs/nb3_nrss.ipynb

This tutorial pair makes the mismatch visible:

• experiment is reduced through geometry-aware PF integration,
• simulation is reduced through WPIntegrator.

Desired outcome

We want simulated NRSS data to be reducible through semantics that mirror conventional experimental RSoXS reduction as closely as practical.

That means:

• preserving detector / reciprocal-plane coordinates when needed,
• making the distinction between q_perp and corrected |q| explicit,
• enabling a reduction path for NRSS outputs that parallels experimental pyFAI reduction where appropriate,
• avoiding ambiguous use of q for multiple incompatible meanings.

Proposed solution

Implement a PyHyperScattering-side integrator or adapter specifically for NRSS outputs.

Working name:

• NRSSIntegrator

High-level design

This integrator should:

1. Accept NRSS-native inputs

   • PhysSize
   • simulation shape / dimensions
   • energy or energy axis
   • optionally explicit mode flags
   • It should first search for these in NRSS results metadata (NRSS likely to be updated to provide them as part of results package) and fall back to accepting them as arguments if none are found in the package.

2. Understand NRSS output semantics

   • detect whether the input corresponds to a 2D reciprocal-plane projection or a 3D detector-aware projection
   • distinguish z_dim == 1 from z_dim > 1

3. Use pyFAI machinery where appropriate

   • synthesize pyFAI-compatible geometry metadata or an internal geometry object
   • reuse PFGeneralIntegrator / PFEnergySeriesIntegrator patterns rather than duplicating reduction logic

4. Produce explicit and semantically correct reduced output

   • chi,q
   • with documented interpretation of the returned q-axis

Behavioral proposal

Case A: 2D NRSS / reciprocal-plane-style output (z_dim == 1)

Two reasonable options exist:

• preserve current semantics and explicitly document the output as radial remeshing of reciprocal-plane coordinates, or
• allow an optional detector-geometry interpretation if users explicitly request it.

Proposed default:

• keep the default simple and explicit,
• do not silently reinterpret 2D reciprocal-plane output as detector-corrected q-space unless the user opts in. The idea would be that the opt-in makes the 2D sim look like what the detector "sees" for a 3D sample without geometry-aware reduction and geometry-aware q-axis correction.

Case B: 3D NRSS / detector-aware output (z_dim > 1)

The integrator should apply the planar-detector trigonometric correction when generating reduced radial q coordinates.

This is the case where the mismatch matters most and where a geometry-aware reduction path is strongly justified.

Why this should live in PyHyperScattering, not NRSS

This is primarily a reduction-layer concern, not a simulation-layer concern.

Reasons to prefer the PyHyperScattering side

1. Correct ownership boundary

   • NRSS should focus on generating physically meaningful scattering outputs.
   • PyHyperScattering already owns data loading, geometry handling, remeshing, and reduction abstractions.

2. Closer alignment with experimental workflows

   • Conventional beamline data reduction already lives in PyHyperScattering.
   • Putting the NRSS bridge there makes simulated and experimental data meet in the same layer.

3. Avoids beamline-reduction logic leaking into NRSS

   • pyFAI-specific geometry accommodations do not belong in NRSS core.

4. Cleaner API surface

   • Users comparing simulation to experiment can stay inside one reduction framework.

What NRSS should provide instead

NRSS should expose enough metadata for PyHyperScattering to reduce its outputs correctly.

Examples:

• phys_size_nm
• original array shape
• an explicit flag describing projection semantics
• enough attrs to distinguish reciprocal-plane output from detector-aware projection output

This is a metadata accommodation, not a reduction implementation.

Recommended implementation plan

Phase 1: add an NRSS-aware PyHyperScattering integrator

1. Add a new integrator class in PyHyperScattering.
2. Reuse PF integrator internals where possible.
3. Define explicit behavior for:
   • 2D reciprocal-plane NRSS outputs,
   • 3D detector-aware NRSS outputs.
4. Ensure the class can operate directly on NRSS-produced xarrays.
   • current NRSS does not offer the necessary metadata as part of results package, but that can change. The class should be able to try to find the metadata on the xarray and fall back to explicit arguments if they are not found.

Phase 2: add metadata bridges in NRSS

1. Add attrs needed by the new integrator.
2. Ensure metadata survives serialization and xarray export.
3. Document the export contract between NRSS and PyHyperScattering.

Phase 3: align NRSS tests with intended semantics

1. Keep detector-coordinate tests where that is the intended validation target.
2. Add explicit reduction-path tests for the new integrator.
3. Add tests that compare:
   • WPIntegrator output (q_perp semantics)
   • new NRSS-aware PF-style output (corrected q semantics)
   • analytical references constructed under matching assumptions

Acceptance criteria

This issue should be considered resolved when:

• NRSS and PyHyperScattering documentation clearly distinguish reciprocal-plane coordinates from corrected q-space.
• There is an officially supported PyHyperScattering reduction path for NRSS outputs that is appropriate for conventional experiment-vs-simulation comparison.
• WPIntegrator is no longer implicitly treated as the standard reduction path for NRSS outputs when corrected q-space is desired.
• At least one NRSS tutorial demonstrates the new reduction path.
• Validation tests explicitly state which q semantics they target.

Open design questions

1. Should the output radial coordinate be named differently in some modes, for example:

   • q_perp
   • q_detector
   • q_corrected
     to prevent semantic drift? I kind of hate this idea because only one is correct, but it might make sense to be explicit.

2. Should PF-style reduction from NRSS outputs be implemented as a new user-facing class, implemented as an adapter around PFEnergySeriesIntegrator, with a private preprocessing step? That's my strong recommendation and likely first proposed solution.

3. I think we should keep WPIntegrator workflow as a supported detector-coordinate comparison tool. Should it have some added warnings?

Proposed initial direction

• Implement a PyHyperScattering-side NRSSIntegrator that bridges NRSS outputs into PF-style geometry-aware reduction.
• Keep WPIntegrator unchanged, but document it as a radial remesher over provided qx/qy, not as a corrected q reducer.
• Add lightweight NRSS metadata accommodations to support the new integrator cleanly.

This keeps responsibilities separated cleanly:

• NRSS produces scattering,
• PyHyperScattering reduces scattering,
• simulation and experiment can then be compared through a common reduction layer.

Suggested labels

• enhancement
• geometry
• integration
• nrss
• pyhyperscattering
• validation
• tutorials

Cross-repo note

This likely requires coordinated changes across:

• PyHyperScattering
• NRSS

but the primary API and ownership should live in PyHyperScattering, with NRSS providing metadata support and updated tutorial / validation usage.

Related existing PyHyperScattering issues

This issue is related to, but broader than, several existing PyHyperScattering issues:

• #84 - Incorrect q axis in WPIntegrator in PyHyperScattering version 0.2.0
• #52 - feat: Implement q-axes for xarray data
• #57 - feat: general beamcenter calibration
• #86 - feat possibility: pyfai multi geometry for energy scans

Why this is not a duplicate

Issue #84 is the closest prior report, but it appears to focus on incorrect or shifted q values in WPIntegrator output. The problem described in this draft is broader:

• it includes the semantic mismatch between WPIntegrator and pyFAI-backed PF reduction,
• it includes the disconnect between experimental and NRSS reduction workflows,
• it includes the special case of NRSS 3D / detector-aware projection outputs,
• and it proposes an architectural bridge solution rather than only a local q-axis correction.

This issue is intended to capture a broader design problem than any one of the above issues:

• a unification and ownership/design issue for q-coordinate reduction semantics,
• informed by prior WPIntegrator q-axis concerns,
• and related to existing pyFAI / geometry / q-axis enhancement work.

In particular, this issue is related to prior q-axis and geometry concerns, especially #84, but is intended to capture the larger design problem of how NRSS outputs should be reduced relative to conventional pyFAI-backed experimental workflows.

[delongchamp]

I discussed this with @andrewjlevin and I think he and I are on the same page about it. @andrewjlevin I might ask for your help moving this through.

[delongchamp]

initial implementation results here.