Skip to content

feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166) - #222

Open
jagannalla wants to merge 2 commits into
Netflix:mainfrom
jagannalla:feature/local-memory-support
Open

feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166)#222
jagannalla wants to merge 2 commits into
Netflix:mainfrom
jagannalla:feature/local-memory-support

Conversation

@jagannalla

@jagannalla jagannalla commented Jun 14, 2026

Copy link
Copy Markdown

Description

This PR addresses three core enhancement and bug-fix issues across Maestro's runtime, engine, and database layers:

  1. Step Local Memory Support (Maestro add local-memory support #134):

    • Added transient, step-instance-scoped local memory with a strict capacity constraint of 64KB (serialized JSON bytes).
    • Integrated lifecycle hooks in MaestroTask to ensure local memory is cleared when the step terminates or is canceled.
    • Exposed getLocalMemory() in StepRuntimeSummary for child step execution contexts to read/write.
  2. Generation Guard Data Protection (Leverage generation_id to prevent reverting step data #128):

    • Added a generation_id column to the maestro_step_instance schema (migration included).
    • Enforced database-level protection checks on step updates/upserts, preventing stale flow executions/workers from overwriting updates from newer runs (generation_id <= EXCLUDED.generation_id and generation_id <= :flowGeneration).
    • Propagated flow generation context from the event loop down through database synchronization layer.
    • Preserved backwards-compatible signatures to prevent breaks in legacy tests and services.
  3. Human-in-the-Loop While Loop Breakpoint Support (Human in the loop support for while loop #166):

    • Added PAUSED to the StepRuntime.State state machine, allowing step execution to yield back a pause state.
    • Updated StepRuntimeManager to handle PAUSED step status by cleanly calling runtimeSummary.markPaused(...).
    • Updated WhileStepRuntime to check stepBreakpointDao.createPausedStepAttemptIfNeeded(...) between iterations. If a breakpoint exists, it yields a PAUSED state with timeline event logs.
    • Modified WhileStepRuntime.start(...) to return the existing WhileArtifact directly, allowing a seamless resume from the same attempt/iteration without resetting counters or the inner loop's run policy.
    • Wired database breakpoint DAO dependency in MaestroStepRuntimeConfiguration.

Verification

  • Unit and Integration Tests: Added extensive test coverage for all three features. Verified with ./gradlew :maestro-engine:test (all pass).
  • Compile/Build Checks: Verified compilation and formatting with ./gradlew build -x test and ./gradlew spotlessCheck.

@jagannalla jagannalla changed the title feat: Implement step local-memory (#134) and flow generation ID protection (#128) feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166) Jun 14, 2026
…data protection (Netflix#128)

- Added transient, step-instance-scoped local memory with 64KB limit and automated cleanup.
- Added generation_id database column to maestro_step_instance and enforced checks on UPSERT/UPDATE to prevent stale worker writes.
- Added unit and integration tests for local memory and generation ID protection.
- Configured SpotBugs exclusions filter and updated configuration.

Signed-off-by: Jagan Nalla <jagannalla1@gmail.com>
Signed-off-by: Jagan Nalla <jagannalla1@gmail.com>
@jagannalla
jagannalla force-pushed the feature/local-memory-support branch from ef2e098 to 97b11fa Compare June 15, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant