Abysmal performance of the HEX record serialization

[Code7R]

Hello,

see attached example. Nothing spectacular, it basically writes out 128MB of raw data.

Expected result: be done in a couple of seconds.

Observed result: roughly 50 seconds for hexrec.SrecFile and 43 seconds for IhexFile. Also consuming over TREE Gig of RAM (on amd64 system with Python 3.13).

Stack trace on random locations shows it typically lurking around here:

cast, typing.py:2391
compute_checksum, ihex.py:187
validate, base.py:1386
validate, ihex.py:405
init, base.py:799
create_data, ihex.py:211
update_records, ihex.py:709
records, base.py:3554
serialize, base.py:3694
save, base.py:3603
, perf.py:7

perf.py

[TexZK]

Hi, yeah I have to admit that's exactly as you reported, and I think there can't be many changes, because of the architecture and the scope of this library.

By putting a few performance markers here and there, I think I figured out what's making such a gargantuan hit.
Basically, hexrec is a library for manipulating hexadecimal record files, so if you create say a SrecFile from memory, and then you decide to export as a records file, it internally chops the memory into individual records, which are collected and stored altogether within its SrecFile._records.
By default, the maxdatalen attribute is 16, so there's going to be a SrecRecord object for each 16-byte chunk, plus the metadata for each record (class object dictionary).
Each 16-byte record is going to cost like 20 times as much (I haven't measured it exactly), that's why 128 megabytes of chopped raw memory result in multi-gigabyte RAM usage. Setting maxdatalen to 128 bytes greatly reduced RAM usage.

That's by design, because the end goal of this library, which is the reason I started building it a dozen years ago, is to mix and match binary files for MCU embedded systems, which arguably don't host more than a single digit megabytes of memory.

Sooo yeah, sadly your test makes hexrec the wrong tool for your task as it is right now.
I don't know if you'd really need support for such "big" data, if you have a real world use case, or if that was just for benchmark.

Perhaps I could add some option for a lightweight conversion, for example a function argument to explicitly decide whether to hold the serialized records on conversion from memory (and vice-versa). This way, the update_records/apply_records functions would use volatile generators and memory views, instead of generating copies and holding them. This would help greatly with RAM consumption, and maybe a little bit with time.
This feature might be via function arg or as a dedicated function, which could host better optimization.
For the time performance side I don't have many hopes, in the end this library is pure Python, so I expect either no big optimizations, or some optimization which makes the code rather messy. I'd keep it plain and simple, considering the target.