Disclaimer: I am not sure if the following is compatible with the way yardl is designed.
Another way to potentially improve the efficiency of the python-based reader for "lists" of events with fixed (uncompressed) structure would be the following:
- assume we have an event with a fixed structure (e.g. 1 float, 2 ints, or e.g. a 32bit word that encodes 4x 6bit unints, and 2x 4bit ints, ...)
- assume that we stream and store many of those event (1e6) into a "TimeBlock"
If the structure of the events is fixed, numpy's memmap together with a custom dtype could be used to read all events efficiently in the TimeBlock into a "2D" array avoiding more expensive loops over events.
Obviously, we don't want to loose the compression feature, but instead of compressing individual events, the whole TimeBlock could be compressed / decompressed.
Disclaimer: I am not sure if the following is compatible with the way yardl is designed.
Another way to potentially improve the efficiency of the python-based reader for "lists" of events with fixed (uncompressed) structure would be the following:
If the structure of the events is fixed, numpy's memmap together with a custom dtype could be used to read all events efficiently in the TimeBlock into a "2D" array avoiding more expensive loops over events.
Obviously, we don't want to loose the compression feature, but instead of compressing individual events, the whole TimeBlock could be compressed / decompressed.