Skip to content

FIX only set max_action_concurrency on RunSpec when explicitly provided#1263

Closed
virchan wants to merge 1 commit into
flyteorg:mainfrom
virchan:RunSpec-has-no-max_action_concurrency-field
Closed

FIX only set max_action_concurrency on RunSpec when explicitly provided#1263
virchan wants to merge 1 commit into
flyteorg:mainfrom
virchan:RunSpec-has-no-max_action_concurrency-field

Conversation

@virchan

@virchan virchan commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Issue

Discovered when running the hello.py example from Quickstart with DevBox:

>>> flyte start devbox
>>> flyte run hello.py main
Execution failed: Protocol message RunSpec has no "max_action_concurrency" field.

The root cause is that my DevBox image was built before d5dd75c, so the older RunSpec does not recognise the max_action_concurrency field.

Currently in main, the SDK always passes max_action_concurrency=self._max_action_concurrency or 0 in RunSpec. When the user doesn't set it, it falls back to 0, and the old backend rejects it every time.

Fix

Remove max_action_concurrency from the RunSpec constructor and set it conditionally afterwards, consistent with the else branch:

flyte-sdk/src/flyte/_run.py

Lines 439 to 440 in 0066444

if self._max_action_concurrency:
run_spec.max_action_concurrency = self._max_action_concurrency

This makes the SDK backward-compatible with any Flyte backend that predates the field. Updating the DevBox image would also resolve the immediate error, but would leave the SDK incorrectly sending a default 0 for a field the user never set.

Updates:

The root cause turns out to be an outdated version of flyteidl2. Updating flyteidl2 should work.

…ovided

Signed-off-by: Virgil Chan <virgil@union.ai>
@virchan virchan force-pushed the RunSpec-has-no-max_action_concurrency-field branch from 77828e8 to 22db276 Compare June 26, 2026 23:03

@pingsutw pingsutw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work after you restart the devbox? I just pushed the new image today

@virchan

virchan commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

Does it work after you restart the devbox? I just pushed the new image today

I still get the Execution failed message even with the new image. I did a fresh start (deleted the DevBox container in Docker Desktop) and confirmed the image is from today:

>>> docker inspect cr.flyte.org/flyteorg/flyte-devbox:latest --format '{{.Created}}'
2026-06-26T20:03:57.84177642Z

Running hello.py on main:

>>> git log --oneline -5
00664448 (HEAD -> main, origin/main, origin/HEAD, dev-box-task-url-fix) test(integration): add clustered (JobSet) integration test (#1244)
45118356 fix(ci): build flyte with pretend version in integration tests (#1259)
2496abdd build(deps): bump the uv group across 12 directories with 5 updates (#1258)
1718f54e Filter runs by paused action (#1249)
22ffca39 add CLI JSON parsing for File, Dir, and DataFrame in collections (#1257)

So the fix in the SDK is still needed regardless of the image version.

@wild-endeavor

wild-endeavor commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

fixing this in the sdk is not the right thing to do though...
you're supposed to be able to add new fields to proto messages. i'm confused where the error is coming from. what devbox is this?

@virchan

virchan commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

fixing this in the sdk is not the right thing to do though... you're supposed to be able to add new fields to proto messages. i'm confused where the error is coming from. what devbox is this?

So I looked at flyteidl2 to check the proto bindings, and found that even 2.0.26 doesn't have max_action_concurrency in RunSpec:

>>> python -c "from flyteidl2.task import run_pb2; print(list(run_pb2.RunSpec.DESCRIPTOR.fields_by_name.keys()))"
['labels', 'annotations', 'envs', 'interruptible', 'overwrite_cache', 'cluster', 'raw_data_storage', 'security_context', 'cache_config', 'notification_rule_name', 'notification_rules']
>>> pip show flyteidl2                                                                                            
Name: flyteidl2
Version: 2.0.26
Summary: IDL for Flyte
Home-page: 
Author: 
Author-email: Union Eng <support@union.ai>
License: 
Location: /Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages
Requires: googleapis-common-protos, protobuf, protoc-gen-openapiv2, protovalidate
Required-by: flyte, flyteplugins-union

The DevBox image used is built on 2026-06-26T20:03:57.84177642Z.

So perhaps the proto bindings need to be updated first?

@pingsutw

Copy link
Copy Markdown
Member

I cannot repro. I do see the max_action_concurrency

(flyte) (flyte-sdk) ➜  flyte-sdk git:(main) uv pip list | grep flyte
flyte                     2.5.7b0               /Users/kevin/git/flyte-sdk
flyteidl2                 2.0.26
flyteplugins-spark        2.5.7.dev6+g00664448d /Users/kevin/git/flyte-sdk/plugins/spark
(flyte) (flyte-sdk) ➜  flyte-sdk git:(main) python -c "from flyteidl2.task import run_pb2; print(list(run_pb2.RunSpec.DESCRIPTOR.fields_by_name.keys()))"
['labels', 'annotations', 'envs', 'interruptible', 'overwrite_cache', 'cluster', 'raw_data_storage', 'security_context', 'cache_config', 'notification_rule_name', 'notification_rules', 'run_start_time', 'max_action_concurrency', 'run_base_dir']

I feel like you probably in the wrong python venv when you run flyte run

@virchan

virchan commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

I cannot repro. I do see the max_action_concurrency

(flyte) (flyte-sdk) ➜  flyte-sdk git:(main) uv pip list | grep flyte
flyte                     2.5.7b0               /Users/kevin/git/flyte-sdk
flyteidl2                 2.0.26
flyteplugins-spark        2.5.7.dev6+g00664448d /Users/kevin/git/flyte-sdk/plugins/spark
(flyte) (flyte-sdk) ➜  flyte-sdk git:(main) python -c "from flyteidl2.task import run_pb2; print(list(run_pb2.RunSpec.DESCRIPTOR.fields_by_name.keys()))"
['labels', 'annotations', 'envs', 'interruptible', 'overwrite_cache', 'cluster', 'raw_data_storage', 'security_context', 'cache_config', 'notification_rule_name', 'notification_rules', 'run_start_time', 'max_action_concurrency', 'run_base_dir']

I feel like you probably in the wrong python venv when you run flyte run

Thank you for sharing the snippet! It motivated me to take another look at flyteidl2, which turned out to be the root cause:

(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip install flyteidl2==2.0.20
Resolved 14 packages in 257ms
Uninstalled 1 package in 25ms
Installed 1 package in 9ms
 - flyteidl2==2.0.26
 + flyteidl2==2.0.20
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip list | grep flyte
flyte                     2.5.7b1         /Users/virchan/dev/github/flyte-sdk
flyteidl2                 2.0.20
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte start devbox                                                                                             
Resuming paused devbox cluster...
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte stop devbox
Devbox cluster stopped. Run flyte start devbox to resume.
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte start devbox
  Waiting for flyte cluster to be ready ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:58
╭────────────────────────────────────────────────────────────────────────── Flyte Devbox ──────────────────────────────────────────────────────────────────────────╮
│ Flyte devbox cluster is ready!                                                                                                                                   │
│                                                                                                                                                                  │
│   🚀 UI:             http://localhost:30080/v2                                                                                                                   │
│   🐳 Image Registry: localhost:30000                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte run z_testtest.py main                                                                                   
  > Launching remote execution...
  > Building 1 image...
    > Building image flyte for environment hello_env
    i Image ghcr.io/flyteorg/flyte:py3.12-v2.5.7b1 already exists, skipping build
  ✓ Built image for environment hello_env: ghcr.io/flyteorg/flyte:py3.12-v2.5.7b1
  ✓ Code bundle found in cache, skipping upload
╭───────────────────────────────────── Exception ─────────────────────────────────────╮
│ ✕ Execution failed: Protocol message RunSpec has no "max_action_concurrency" field. │
╰─────────────────────────────────────────────────────────────────────────────────────╯
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip list | grep flyte    
flyte                     2.5.7b1         /Users/virchan/dev/github/flyte-sdk
flyteidl2                 2.0.20

After updating flyteidl2 to 2.0.26, the error disappeared, and I could confirm that max_action_concurrency is present by running the snippet again. This also explains #1263 (comment), as I hadn't been testing carefully enough at the time.

With that said, I think we can close this PR. If other users run into the same issue, we can point them to updating flyteidl2 first.

Thank you everyone for your time! 🫡

@virchan virchan closed this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants