image addon serialization

[jerch]

Currently images printed to the terminal cannot be serialized. Previous tests regarding image serialization were done in jerch/xterm-addon-image#48. They revealed the following cornerstones:

• add QOI image format for performance reasons
• implement an image sequence format suitable for serialization
  • separate data load from placements
  • possible to shard data parts
• extend serialize addon with at least row-level events

Steps to be taken
As a possible cause of actions I suggest the following steps:

• add QOI decoder to IIP (maybe kitty too?) --> QOI support for IIP #5846
• implement serialization image sequence
  • add QOI encoder
  • learn from kitty protocol about sharding and placements
  • spec internal sequence with:
    • data load
    • placements
• fix resize handling (see image addon: proper resizing #5861)
• extend serialization
  • add events to serialize addon
  • expose events on serialize addon API
• register image serialization to serialize events

The reason for using QOI and not one of the standard formats is mostly performance. QOI yields comparable file sizes to PNG, but is in previous tests ~4 times faster to process.

For the custom sequence we can take bits from the kitty protocol, or extend it for encoding purposes with QOI. The tricky part will be to harmonize it with SIXEL and IIP states though. Currently I don't aim to write a full serialization for all protocol types, as this seems overbloated. Instead we should extract the minimal needed bits to restore images states correctly with one sequence and image format only.

Up for discussion first, as it will introduce quite some changes across the codebase.
FYI: @Tyriar, @anthonykim1, @PerBothner

[Tyriar]

expose events on serialize addon API

Interested in what you had in mind here with the events, have a small code snippet to communicate that?

[jerch]

In jerch/xterm-addon-image#48 I created a stub serialize (text-only for demo purpose, see ImageAddon.ts:L348-L361):

  public serialize(start: number, end: number): string[] {
    const st = Date.now();
    const lines: string[] = [];
    const buffer = this._terminal!._core.buffer;
    for (let row = start; row < end; ++row) {
      const line = buffer.lines.get(row);
      if (!line) break;
      // FIXME: hook into serialize addon instead of translateToString
      // lines.push(line.translateToString(true) + this.serializeLinePng(row));
      lines.push(line.translateToString(true) + this.serializeLineQoi(row));
    }
    console.log('duration', Date.now()-st);
    return lines;
  }

testing the differences for QOI and PNG based serialization. As you can see, I basically would need a callback at the end of every row to insert image data right there. In the demo I used line-wide canvas printing (would chunk an image into line images), for a real impl it would be better to feed the images as a whole at the beginning, and just ref them at the line stage (similar to kitty's placements).

So on serialize side I would need at least these callback possibilities:

/**
 * Announce a serialize attempt for a certain row range.
 * This is needed to get all images within row range and
 * serialize them for bulk load into `data`.
 */
onSerializeStart((data: SerializeData, firstRow: number, lastRow: number) => void);
/**
 * Event for every row at end of row serialization.
 * NOTE: This has to be row-level (not logical line level) to ensure,
 *       that the cursor did not drop out of reach yet (on a smaller output terminal).
 * The image addon callback places a custom sequence into `data`
 * to reference into the image tiles from images loaded above.
 */
onSerializeRowEnd((data: SerializeData, row: number) => void);

We would also need a onSerializeRowStart if we want to restore kitty's "render-below" behavior from serialization, but I think that we don't need that for a first prototype, as it makes things only more complicated.

Edit:
Rethinking it - the minimum requirement would be the onSerializeRowEnd hook, as we could also add the image data on first occurence instead of bulk loading them at the beginning (this is safer for proper eviction handling).

[jerch]

On a sidenote - the serialization prolly should be customizable, so embedders can set the appropriate image format for local vs. remote restores. Locally QOI would give the best performance and is lossless, so that is a no-brainer as default. For remote restores embedders might vote for JPEG on their platforms resulting in much smaller serialization data, but is only a lossy restore. (See throughput numbers in #5846)