FIX only set max_action_concurrency on RunSpec when explicitly provided#1263
FIX only set max_action_concurrency on RunSpec when explicitly provided#1263virchan wants to merge 1 commit into
max_action_concurrency on RunSpec when explicitly provided#1263Conversation
…ovided Signed-off-by: Virgil Chan <virgil@union.ai>
77828e8 to
22db276
Compare
pingsutw
left a comment
There was a problem hiding this comment.
Does it work after you restart the devbox? I just pushed the new image today
I still get the Execution failed message even with the new image. I did a fresh start (deleted the DevBox container in Docker Desktop) and confirmed the image is from today: >>> docker inspect cr.flyte.org/flyteorg/flyte-devbox:latest --format '{{.Created}}'
2026-06-26T20:03:57.84177642ZRunning >>> git log --oneline -5
00664448 (HEAD -> main, origin/main, origin/HEAD, dev-box-task-url-fix) test(integration): add clustered (JobSet) integration test (#1244)
45118356 fix(ci): build flyte with pretend version in integration tests (#1259)
2496abdd build(deps): bump the uv group across 12 directories with 5 updates (#1258)
1718f54e Filter runs by paused action (#1249)
22ffca39 add CLI JSON parsing for File, Dir, and DataFrame in collections (#1257)So the fix in the SDK is still needed regardless of the image version. |
|
fixing this in the sdk is not the right thing to do though... |
So I looked at >>> python -c "from flyteidl2.task import run_pb2; print(list(run_pb2.RunSpec.DESCRIPTOR.fields_by_name.keys()))"
['labels', 'annotations', 'envs', 'interruptible', 'overwrite_cache', 'cluster', 'raw_data_storage', 'security_context', 'cache_config', 'notification_rule_name', 'notification_rules']>>> pip show flyteidl2
Name: flyteidl2
Version: 2.0.26
Summary: IDL for Flyte
Home-page:
Author:
Author-email: Union Eng <support@union.ai>
License:
Location: /Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages
Requires: googleapis-common-protos, protobuf, protoc-gen-openapiv2, protovalidate
Required-by: flyte, flyteplugins-unionThe DevBox image used is built on So perhaps the proto bindings need to be updated first? |
|
I cannot repro. I do see the max_action_concurrency I feel like you probably in the wrong python venv when you run |
Thank you for sharing the snippet! It motivated me to take another look at (flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip install flyteidl2==2.0.20
Resolved 14 packages in 257ms
Uninstalled 1 package in 25ms
Installed 1 package in 9ms
- flyteidl2==2.0.26
+ flyteidl2==2.0.20
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip list | grep flyte
flyte 2.5.7b1 /Users/virchan/dev/github/flyte-sdk
flyteidl2 2.0.20
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte start devbox
Resuming paused devbox cluster...
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte stop devbox
Devbox cluster stopped. Run flyte start devbox to resume.
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte start devbox
Waiting for flyte cluster to be ready ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:58
╭────────────────────────────────────────────────────────────────────────── Flyte Devbox ──────────────────────────────────────────────────────────────────────────╮
│ Flyte devbox cluster is ready! │
│ │
│ 🚀 UI: http://localhost:30080/v2 │
│ 🐳 Image Registry: localhost:30000 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % flyte run z_testtest.py main
> Launching remote execution...
> Building 1 image...
> Building image flyte for environment hello_env
i Image ghcr.io/flyteorg/flyte:py3.12-v2.5.7b1 already exists, skipping build
✓ Built image for environment hello_env: ghcr.io/flyteorg/flyte:py3.12-v2.5.7b1
✓ Code bundle found in cache, skipping upload
╭───────────────────────────────────── Exception ─────────────────────────────────────╮
│ ✕ Execution failed: Protocol message RunSpec has no "max_action_concurrency" field. │
╰─────────────────────────────────────────────────────────────────────────────────────╯
(flyte) virchan@Virgils-MacBook-Pro flyte-sdk % uv pip list | grep flyte
flyte 2.5.7b1 /Users/virchan/dev/github/flyte-sdk
flyteidl2 2.0.20After updating With that said, I think we can close this PR. If other users run into the same issue, we can point them to updating Thank you everyone for your time! 🫡 |
Issue
Discovered when running the
hello.pyexample from Quickstart with DevBox:>>> flyte start devbox >>> flyte run hello.py main Execution failed: Protocol message RunSpec has no "max_action_concurrency" field.The root cause is that my DevBox image was built before d5dd75c, so the older
RunSpecdoes not recognise themax_action_concurrencyfield.Currently in
main, the SDK always passesmax_action_concurrency=self._max_action_concurrency or 0inRunSpec. When the user doesn't set it, it falls back to0, and the old backend rejects it every time.Fix
Remove
max_action_concurrencyfrom theRunSpecconstructor and set it conditionally afterwards, consistent with theelsebranch:flyte-sdk/src/flyte/_run.py
Lines 439 to 440 in 0066444
This makes the SDK backward-compatible with any Flyte backend that predates the field. Updating the DevBox image would also resolve the immediate error, but would leave the SDK incorrectly sending a default
0for a field the user never set.Updates:
The root cause turns out to be an outdated version of
flyteidl2. Updatingflyteidl2should work.