Add an option to reduce OPA agent memory pressure when loading large data.json files - #366
Draft
Pushpalanka wants to merge 1 commit into
Draft
Add an option to reduce OPA agent memory pressure when loading large data.json files#366Pushpalanka wants to merge 1 commit into
Pushpalanka wants to merge 1 commit into
Conversation
…y pressure When a system source has multiple HTTP/S3 datasources, OPA's bundle.Write collapses all data into a single root /data.json in the bundle tar. OPA agents must parse and hold the entire merged document in memory at once, causing memory pressure on systems with large datasources. Add a per-bundle flag options.nested_data_files (default: false). When enabled, the builder writes each datasource's data as a separate data.json at its namespaced path within the tar (e.g. /teams/data.json, /admins/data.json), matching the Styra DAS bundle layout. OPA's bundle reader merges nested files into the same data document at load time, so runtime semantics are unchanged. The flag is per-bundle because OCP builds bundles for many systems with different data characteristics — systems with many small datasources may not benefit and would carry extra tar-entry overhead.
srenatus
reviewed
Jun 25, 2026
Comment on lines
+511
to
514
| if b.nestedDataFiles { | ||
| return writeBundle(b.output, *result, bsFSMap) | ||
| } | ||
| return bundle.Write(b.output, *result) |
Contributor
There was a problem hiding this comment.
🏗️ I think this feature would make a lot of sense as an option on bundle. We wouldn't have to replicate the tarball creation logic here, and any OPA users would benefit from it, and also OCP.
What do you think? Would be you OK with pivoting into that direction? 😃
Author
There was a problem hiding this comment.
Thanks for the direction and taking time to check the draft @srenatus
That makes lot of sense, will try that path. 👍
Contributor
There was a problem hiding this comment.
Thank you! And thanks for contributing, this looks like such a nice tweak. ✨
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates: #365