Recommended architecture for raster-aware spatial predictors in OpenZL

[csaybar]

Is your feature request related to a problem? Please describe.

OpenZL looks very promising for structured compression. I am getting very fast decompression results, and the graph-based design seems to open the door to a more unified compression framework, rather than relying on many independent codec-specific pipelines. I really like the direction of the project ❤️.

My use case is geospatial and scientific raster data, such as DEMs, Earth Observation imagery, climate grids, hyperspectral cubes, and tiled raster formats commonly handled by libraries such as GDAL.

GDAL is the main open-source library used to read, write, and process geospatial raster and vector data.

For raster data, one of the strongest sources of redundancy is spatial. A raster tile is not just a byte array. It usually has additional structure, such as:

width and height
number of bands
data type
interleave mode
nodata values
spatial neighborhood relationships

Traditional raster formats often exploit this structure using predictors before entropy coding. Examples include TIFF horizontal differencing, floating-point predictors, JPEG-LS/LOCO-I-style predictors, Paeth-like predictors, and simple spatial differencing schemes.

My question is about the best architectural fit for this kind of use case in OpenZL.

Describe the solution you'd like

I would like to understand what the OpenZL team considers the most appropriate approach for spatial raster predictors.

One possible direction is that the host format or library, for example GDAL, applies raster-specific predictors first. OpenZL would then receive residuals or transformed byte streams and compress them as generic data.

Another possible direction is that OpenZL provides standard shape-aware primitives for dense 2D numeric arrays. For example:

row or left predictor
column or up predictor
gradient predictor
Paeth-like predictor
simple 2D neighbor predictors

This would allow an OpenZL graph to describe the full compression pipeline in a portable way, while keeping the compressed stream decodable by a standard OpenZL decoder.

I am not sure which direction better matches OpenZL's design philosophy.

So the main question is:

Should spatial raster predictors live outside OpenZL, with OpenZL only compressing transformed streams, or would generic 2D spatial prediction primitives be considered in scope for standard OpenZL codecs?

[Cyan4973]

The general philosophy of OpenZL can be described as follows:

1. First, data must be parsed. We need to separate decoration (header, footer) from the actual substance (big arrays) and describe what they contain (pixels, fields, etc.).

   • That work can be done by a custom parser function, in this case dedicated to the GDAL format
   • or by an SDDL description of the target GDAL format, which will generate a parser on the fly at compression time.
   • SDDL is still in its early days, so its capability set is still limited. Yet, as long as you manipulate arrays of fixed-size fields, and a bit of decoration around, it should be already capable to handle that.
   • Let's be clear that even more complex formats, including variable-size fields and dynamic content, are within scope, so we expect that one day, any GDAL format can be expressed using SDDL.

2. Once we have separated the different components of the format, we can now tackle the compression of homogeneous slices, including those rebuilt after de-interleaving related fields that were previously standing apart. That's where the concept of array becomes interesting.

   • Note that OpenZL compression engine doesn't yet exploit the concept of 2D arrays. Funnily enough, SDDL can already express that, it's just unexploited by the engine afterwards. But it's within scope, and we'll add this feature in the coming year, at which point the topic of predictor will become more palatable

3. To optimally compress each stream (essentially each array), one can take ownership of the graph and decide what to do, or delegate that work to OpenZL's offline trainer, internally named ACE. The goal of ACE is to automatically generate graphs that are close to optimal, along both compression and speed axis, for the data to compress, and re-usable for other similar payloads. It presumes some "stability" or "homogeneity" between the samples used for training and the data to compress in the future.

   • For 2D arrays specifically, the concept of predictor is more important, since using neighboring information to predict the next cell offers substantial benefits for compression performance. We'll get there. Multiple ideas are still considered, including user-defined predictors. But there are also practical considerations, including performance and security, so the earlier implementations are likely going to be more limited.

4. All this work, including separating components, or preparing data with pre-filters, can be done outside of OpenZL. The problem is, in order to rebuild the original data, such compressed artifacts will have to go through OpenZL + custom decoding operations, which can be more or less difficult to deploy and maintain across all receivers over time. The benefit of doing these steps inside OpenZL is that all the reverse reconstruction stages become part of the decompression pipeline.

So now you have an idea of the general direction. That being said, I can understand that, while the capability set remains limited, it can be tempting to do the missing operations externally. And it can certainly done, especially if you need a result urgently, just be aware of the deployment issues for the decompression side. Alternatively, it can also become an opportunity to underline what's still missing within OpenZL to reach performance parity, and we would work towards closing this gap over time.

[csaybar]

Hi @Cyan4973, thanks so much for the detailed answer, and sorry for the slow reply.

We spent time reading through the engine to understand where 2D actually fits, and your framing makes a lot sense to us. You parse, separate the substance, then compress homogeneous slices, and the predictor question only really lands once the engine carries the 2D shape downstream. Going through the tests, we noticed estimateDimensionality already recovers the stride from LZ match offsets, and it doesn't seem wired into a graph yet.

https://github.com/facebook/openzl/blob/dev/tests/compress/test_estimate.cpp

On our side we've been building rumi, a GeoTIFF profile where each tile is a self-contained OpenZL frame (compression tag 60000, proposed).

The code is at https://github.com/asterisk-labs/rumi and a short overview is at https://asterisk.coop/rumi/deck/

So we're going to implement an experimental predictor custom codec in rumi, not as a candidate for the standard library, just to measure the gap you mentioned. Each rumi tile is a single-band block of known width, so we can hand OpenZL a clean 2D array and pass the stride explicitly rather than detecting it. That lets us isolate the predictor's benefit on data that behaves very differently, elevation (DEMs), temperature and precipitation grids, and satellite imagery, where continuous fields and categorical rasters should tell quite different stories.

We'll bring the numbers back here once we have them, ratio and decode speed both.