Conversation
efd6
added
enhancement
6 tasks
andrewkroh
reviewed
Jun 17, 2026
andrewkroh
left a comment
andrewkroh
left a comment
Member
There was a problem hiding this comment.
Is a decode_csv_stream_lazy feasible (not needed now)? I think that would make it possible to apply emit to the ti_recordedfuture integration.
Comment on lines
+36
to
+37
| // during CEL evaluation. The beats-side adapter wraps inputcursor.Publisher | ||
| // behind this interface. |
Member
There was a problem hiding this comment.
Suggested change
| // during CEL evaluation. The beats-side adapter wraps inputcursor.Publisher | |
| // behind this interface. | |
| // during CEL evaluation. |
I think we should not mention details about beats in the library side.
Comment on lines
+76
to
+77
| // If Emit returns an error, iteration stops. The result reflects the count | ||
| // and cursor at the point of failure. |
Member
There was a problem hiding this comment.
Could we return an error field in the result map (e.g. {"published": N, "incomplete": true, "error": "..."}) so the program/cursor logic can potentially react? IIUC a decode error is indistinguishable from a clean EOF without it.
Add three composable features for processing large payloads in CEL programs without materialising all records in memory: - Opaque stream type (streamVal) wrapping io.Reader, with stream_gzip and stream_zip producers that wrap decompression readers around in-memory bytes. - Lazy JSON stream decoder (decode_json_stream_lazy) that accepts stream, bytes, or string and returns a traits.Iterable decoding values on demand via json.Decoder. - Emit macro that iterates a list or iterable, calling an Emitter callback for each element. Supports optional per-element cursor tracking. In the mito CLI, emitted events are printed to stderr.
Collaborator
Author
|
Added CSV and line streams (for TSV and other failures to read prior art). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add three composable features for processing large payloads in CEL programs without materialising all records in memory:
Opaque stream type (streamVal) wrapping io.Reader, with stream_gzip and stream_zip producers that wrap decompression readers around in-memory bytes.
Lazy JSON stream decoder (decode_json_stream_lazy) that accepts stream, bytes, or string and returns a traits.Iterable decoding values on demand via json.Decoder.
Emit macro that iterates a list or iterable, calling an Emitter callback for each element. Supports optional per-element cursor tracking. In the mito CLI, emitted events are printed to stderr.