Dealing with malformed data

[orao]

I am looking at making modifications to the way the imported files are processed to support altimiters like AON2 and Dekunu.

AON2 doesn't have all the fields that are expected in the import, which can be derived from GPS and height data, while I've been suspicious of the accuracy of some values I've seen from the Dekunu and would like them to be either verified or have an option that allows them to be derived instead.

Would you be happy for me to simply implement this or would you prefer that we have some form of a design discussion ahead of time?

[crwper]

:-) So, first, I want to say thanks for all the work you've been doing. I've been a little swamped with other work, so haven't been able to merge anything yet, but I am planning to do so.

Let's chat a bit about the proposed changes.

At some point, I want to incorporate FlySight 2's additional sensor data into FlySight Viewer, but this will require a significant change to the data structure, since this additional sensor data comes in a a variety of sample rates. However, I'm not sure how soon I'll be able to make those changes (see comments on other work above).

For the moment, it might be best to implement AON2 and Dekunu data using a similar method to what's used for other optional data right now, e.g., using the hasGeodetic flag to allow processing data that doesn't have lat/lon. Can you put together a table showing what AON2 and Dekunu do provide?

[orao]

No worries, we all have day jobs. As you can see, my own turnaround time is equally lengthy... Also, I appreciate you making this open source and are actually actively participating in discussions here.

I was hoping to break down the DataPoint class completely and have many individual time series instead of one series of a set of known values. The core timeseries is one consisting of time,Lat,Lon,Altitude all other time series are either derived data from the core lat,lon,altitude (whether computed by the device or FlySightViewer later) and can be treated separately or they are simple additional details provided by the device (i.e. gps fix).

My thinking is inspired by the likes of Grafana, where any graph with 2 axes is just a time series of points. The data in FlySightViewer doesn't seem organized in a way that allows it to be easily unpacked or derived.

I did also notice some small performance issues when loading many different jumps but I'll raise those issues separately as I've not had a chance to profile the application yet.

In the end, there are very few operations that require the full set of the data available in the DataPoint and most operations simply pick and choose what they need.

Please see some sample data for the 2 altimeters below:

Dekunu

time,lat,lon,hMSL,velN,velE,velD,hAcc,vAcc,sAcc,heading,cAcc,gpsFix,numSV
,(deg),(deg),(m),(m/s),(m/s),(m/s),(m),(m),(m/s),(deg),(deg),,
2023-05-08T09:18:14.900Z,52.561399,-0.397338,3373.0,0.0,0.0,0.0,1,1,1,0.01,1,0,0
2023-05-08T09:18:15.000Z,52.561373,-0.397377,3373.0,0.0,0.0,0.0,1,1,1,0.01,1,0,0
2023-05-08T09:18:15.100Z,52.561346,-0.397416,3373.0,0.0,0.0,0.0,1,1,1,0.01,1,0,0
2023-05-08T09:18:15.200Z,52.56132,-0.397456,3373.0,0.0,0.0,0.0,1,1,1,0.01,1,0,0

I suspect that the velD values are nonsense and was hoping to get FlySightViewer to recompute them. Some of the jumps I looked at didn't make sense from a speed point of view.

AON2

Skydiving,2019-10-27T16:48:19.000,6338,726
2019-10-27T16:59:39.000,50.865498,-3.232356,4565
2019-10-27T16:59:40.000,50.865498,-3.232356,4535
2019-10-27T16:59:41.000,50.8657,-3.23188,4512

I couldn't get these into FlySightViewer at all just yet, not without code changes.

[crwper]

This is actually really close to what I had in mind. I've been doing a lot of work with Pandas, and it's made me think that it might be good to arrange data as a collection of "tables", where each table has as single "time" column and several data columns each with a string identifying it. Several of these tables could be stored for each track. For FlySight 2, we would presumably have separate tables for GNSS data, IMU data, barometric data, etc., since these are all asynchronous.

The interface to this data could be via the string identifiers. So, we might request the "velD" column, and the data (maybe along with the related time values) would be returned regardless of what table it's in (though we would have only one "velD" column amongst all the tables). This way, the requester doesn't need to know the structure of the data.

Additional values could be computed and stored in their own tables. These computations would involve requesting columns by their string identifier, so again the organization of the data is opaque to the requester.

Same thing for plotting.

I think this would also allow us to decouple plots from the data structure. We could have an array of plot types, each with its own default colour, line width, and string identifier for the data.

Does that align with what you had in mind?

[ColOfAbRiX]

Hi @orao! I indipendently started to work on a conversion tool from Dekunu to flysight but as an indipended tool rather than an import feature of flysight viewer. I haven't published mywork yet. I think your work is much better for end users!

Regardless, I'm still interested in devising a way to convert a dekunu file into flysight, would be lovely to see how you proceeded pr what ideas you have on how to improve the Dekunu data. I guess you also have plenty of problems converting the dekunu format with its bad data