Loading in interpolated simulation extinction coefficient

[vasanchez16]

We have access to flight data measuring extinction coefficient and this gives us another avenue for model-observation comparison. However, this flight data is on a significantly different resolution than the 2D grid of our simulation data. Thus, we will need to upload data that is already interpolated/ mapped onto the path of the flights. This interpolation was already done and saved to a csv, however this csv has a column for every simulation number. This will need to be reorganized as our current data structure dedicates one row to every simulation number.

Proposed Solution

1. Create an entity type to store data

entity type InterpolatedExtinctionCoeff schema name ... {
    lat: double
    long: double
    time: datetime
    simulationNumber: int
    coeff: double
} 

We could also try to use FK refs to the SppeTatzFullGeoTimeGrid here instead of the lat,lon,time directly.

2. Create a Source type, that defines a field for each csv column header eg:

type SourceInterpolatedExtinctionCoeff mixes Source {
  lat: double
  long: double
  ens_0_...
  ens_1_...
...

3. Define transform type(s) and FileSourceCollection and/or seed data

Refs

• https://community.c3.ai/t/create-multiple-objects-from-one-row-in-canonical/4130

Notes As This Relates To Simulation Data Model

The proposal here is to directly import flight track (extinction coefficient) data that has been interpolated onto the UM-model full resolution simulation data grid and saved to a csv file.

We may want to consider this a temporary step and plan to fully integrating the process of interpolating the flight tracks for the following reasons:

1. We may want to interpolate this dataset onto different grid resolutions
2. We may want to more tightly integrate this dataset into the data model we have developed around the simulation data eg: SimulationEnsembleDataset, etc.
   Perhaps mixing this type:

type DatasetSimulationOutputBase<DT,ST,GT> {
    //
    dataset: !DT
    // the {@link SmokePPESimulationSample} that this output is associated with
    simulationRun: !ST
    // the {@link GeoSurfaceTimePoint} that this output is associated with
    geoTimeGridPoint: !GT
    // the pseudoLevel index (starts at 0)
}

7. We may want to support integration of additional simulation and/or flight track data.

[vasanchez16]

@kgohil27 Can you list the column names currently in your csv with the interpolated simulation data?

[dadamsncsa]

@kgohil27 , Perhaps if the csv is not bigger than say 100Mb or so, you could just commit it to this repo. Or, upload to Azure or to the c3 jupyter notebook space.

[kgohil27]

@vasanchez16 Currently 121 columns contain interpolated data corresponding to each timestamp during the flight. These variables all have the following format -- ens_<ensemble #><ext. coeff stash code>

I can send you one of the files if you need, let me know

[dadamsncsa]

@kgohil27 Sounds good. Please upload one of the files (are there more than 1?) to either this repo, or the shared c3 jupyter notebook space.