Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.
Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.