Skip to content

Implement logical workers#236

Open
voegtlel wants to merge 2 commits into
developfrom
feature/logical_workers
Open

Implement logical workers#236
voegtlel wants to merge 2 commits into
developfrom
feature/logical_workers

Conversation

@voegtlel

Copy link
Copy Markdown
Collaborator
  • Allow setting logical_workers in the WorkerConfig. This number may be less than the number of physical workers (=world_size * num_workers), but physical workers must be divisible by logical workers.
  • Default: logical_workers = physical_workers (old behavior)
  • Example: Setting logical_workers = physical_workers / 4 would result in 4 physical workers iterating the same data, but each with a stride of 4
    • For optimization, one may set @skip_safe or @stateless(skip_safe=True) on functions in the TaskEncoder, resulting in not calling these functions when skipping samples (if the chain supports it).
    • Only functions without fanout and that do not raise errors (to be caught) may be marked as skip_safe.
    • Cannot go "through" packing or filtering (there are non-precomputable (i.e. without reading the sample) fanouts in these).

@voegtlel voegtlel requested a review from philipp-fischer May 27, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant