Declared default CRS for formats that cannot carry one

[lcalisto]

The #116 extractors (I'm working on it) can read coordinates out of SEG-Y, XYZ, ASC and FLT files, but those formats have no place to say which CRS the coordinates are in, so we leave bbox_4326 NULL rather than guess. If we did assume one, TM06 would work for our surveys but be wrong for anyone else's data. The effect is that about 75k files in production have a perfectly good extent that the catalog can't use: currently it ends up as free text in the description, which nobody can search on.

extension	files	can the format declare a CRS?
.sgy	66,046	no, the headers give the units (metres) but never the system
.segy	3,153	no, same
.xyz	1,619	no, GDAL's XYZ driver doesn't even read a .prj sidecar
.flt	3,661	yes with a .prj, but the deliveries don't include one
.asc	233	yes with a .prj, same

So for about 70k of them (SEG-Y and XYZ) there's no way to fix this at the file level at all. XYZ is the clearest example: the two grids in owf-seism-2024 are certainly in EPSG:3763, they line up exactly with the twin GeoTIFF once reprojected, and we still can't map them.

What I'd propose is to declare the CRS ourselves for the formats that can't carry one, either per survey or as a single default for all of them:

{
  "survey": "owf-seism-2024",  ------> we remove this if its one for everything!
  "default_source_crs": "EPSG:3763",
  "applies_to": [".sgy", ".segy", ".xyz", ".asc", ".flt"]
}

The rule is simple: whatever the file or its sidecar says always wins, and the declaration only fills the gap when there's nothing at all. Records should also show it, something like CRS: EPSG:3763 (declared), so a stated extent is never mistaken for a measured one.

We can keep this as a versioned JSON file in the repo next to config/preview-folders.json, or anywhere else you @ricardogsilva sees fit. It's one line per survey or one line foe everything! We can always move it to the database later if someone outside the team ends up needing to set it.

Nothing changes for formats that do carry a CRS. GeoJSON is fine as it is (the spec says WGS84), and KMALL too, since its positions are already geographic.

This also affects #145 #154

Questions:

• Is per survey enough, or do we need it per folder in case a survey had contractors using different systems? Or do we start with a single default for all surveys? The simpler the better!
• What do you think? I'm open to discuss. @ricardogsilva @gioman

[ricardogsilva]

How about we add a single extra column in the surveymissions table named implicit_crs, which would be the assumed CRS for that mission?

This would be used as the fallback for all files found in the context of the mission. We could add this as additional field in the survey mission details page in the UI.

I understand this is not as expressive as being able to specify a CRS depending on the file format of the file, but it seems like a suitable middle ground solution, with the advantage that it could easily be exposed through the UI in a simple way

[lcalisto]

Yes, I like this a lot!

Mission is the right level. The UI point is great.

Three things I'd like to pin down:

• Precedence: implicit_crs only fills a gap, never overrides a CRS that the file or its sidecar declares, so GeoTIFFs and shapefiles in the mission are unaffected and it only kicks in for SEG-Y, XYZ and friends. What you think?
• Backfill: if someone sets or changes implicit_crs on a mission that already has records, what happens to those? Re-running discovery over tens of thousands of records hits the dramatiq 10 minute limit, so maybe an explicit "reapply" action on the mission page rather than doing it automatically?
• Column type: an integer EPSG code covers everything in the archive and lets us validate the value on input so nobody saves a typo. Free-text WKT would be more flexible but harder to trust. What you think?

I'd also keep a marker in the record description, something like CRS: EPSG:3763 (implicit), so it's always clear whether an extent was measured or assumed.

[ricardogsilva]

• Precedence: agree
• Backfill: not sure what is this dramatiq limit you are referring to, but ideally each individual survey-related record would get its own background task. So there would be many small tasks rather than a big one for the whole mission. Regardless, I think that discovery and generation of derived datasets should always be manually triggered, even if the trigger can be for a bulk action
• column type: I agree that an integer seems like the better option

WRT this marker denoting if the CRS is implicit or not I don't see much value in it - I don't think users will care how the CRS was computed

[lcalisto]

Agree with all your comments. And integer as the column type is indeed a better option.

I think we have a plan!