Skip to content

Broaden megatron-core packages.find to megatron* (namespaces), keep plugins#50

Open
Shi-Dong wants to merge 1 commit into
miles-mainfrom
shi/megatron-broaden-packaging
Open

Broaden megatron-core packages.find to megatron* (namespaces), keep plugins#50
Shi-Dong wants to merge 1 commit into
miles-mainfrom
shi/megatron-broaden-packaging

Conversation

@Shi-Dong
Copy link
Copy Markdown

Summary

Builds on the merged #49 (Package miles_megatron_plugins with Megatron Core) by broadening [tool.setuptools.packages.find].include from megatron.core* to megatron* with namespaces = true, while keeping miles_megatron_plugins in the include list.

This lets callers (e.g. the miles training runner) pip install -e . this checkout and import from megatron.training, megatron.legacy, megatron.rl, etc. without a PYTHONPATH override — megatron.training.initialize imports from megatron.legacy at module load, so a narrower include list breaks transitively. namespaces = true is required because megatron, megatron.legacy, and several sub-packages are PEP-420 namespace packages (no __init__.py).

Why

The miles submoduling stack pins this repo as third_party/Megatron-LM and bundles both megatron.* and miles_megatron_plugins into the miles-rl wheel directly from the submodule. That requires the source checkout to expose the full megatron* tree (not just megatron.core) and to keep packaging the plugins. This PR reconciles both needs in one place, on top of the current miles-main (post-#49).

Supersedes

Replaces the obsolete pair:

Test plan

  • pip install -e . on this checkout, then python -c "import megatron.training, megatron.legacy, miles_megatron_plugins" with no PYTHONPATH override
  • Build the miles-rl wheel from the miles submoduling stack and confirm megatron* + miles_megatron_plugins land at the top level of site-packages

Builds on PR #49 (miles_megatron_plugins packaged with megatron-core). Widen the find list to every megatron.* subpackage and enable namespace packages so an editable install exposes megatron.training/legacy/rl without a PYTHONPATH override, while preserving the miles_megatron_plugins entries PR #49 added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant