Add mpi4py parallel backend support for distributed optimization#13
Open
darshanmandge wants to merge 2 commits into
Open
Add mpi4py parallel backend support for distributed optimization#13darshanmandge wants to merge 2 commits into
darshanmandge wants to merge 2 commits into
Conversation
- Add mpi_map_function() using scatter/gather pattern in multiprocessing.py - Add mpi backend option to get_mapper() - Add use_mpi parameter to EModel_pipeline with mutual exclusivity check - Add mpi4py>=3.0 as optional dependency in pyproject.toml - Update examples (L5PC, nexus) to support --use_mpi CLI flag - Update nexus notebook with use_mpi parameter
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
🚀 New features to boost your workflow:
|
| # Need to reimport to pick up the mock | ||
| import importlib | ||
|
|
||
| import bluepyemodel.tools.multiprocessing as mp |
| from unittest.mock import MagicMock | ||
| from unittest.mock import patch | ||
|
|
||
| import pytest |
| from bluepyemodel.tools import multiprocessing as mp | ||
|
|
||
| # Temporarily replace MPI import | ||
| original_func = mp.mpi_map_function |
| rank = 1 | ||
|
|
||
| def mapper(func, it): | ||
| items = list(it) |
| forge_ontology_path = "./forge_ontology_path.yml" # this file also | ||
| # memodel_id = "<MEMODEL ID>" # replace with the id of the MEModel you want to update | ||
|
|
||
| mapper = map |
| morph_format = "swc" # assumes swc is always present and we do not care about small differences between format | ||
|
|
||
| # additional metadata we will need when saving me-model resource | ||
| subject_ontology = emodel_r.subject if hasattr(emodel_r, "subject") else None |
|
|
||
| # additional metadata we will need when saving me-model resource | ||
| subject_ontology = emodel_r.subject if hasattr(emodel_r, "subject") else None | ||
| brain_location_ontology = morph_r.brainLocation if hasattr(morph_r, "brainLocation") else None |
- Remove unused imports from examples/others/memodel/register_memodel.py (pathlib, Dataset, scores, compute_responses, get_evaluator_from_access_point, copy_emodel_pdf_dependencies_to_new_path, compute_scores) - Fix pycodestyle issues (blank lines, comment spacing, trailing whitespace) - Apply black formatting to multiprocessing.py and register_memodel.py - Add unit tests for mpi_map_function and get_mapper MPI backend (tests/unit_tests/test_multiprocessing.py)
darshanmandge
force-pushed
the
feature/add-mpi4py-support
branch
from
2893242 to
43fc450
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for
mpi4pyas a parallel backend option, enabling distributed optimization across multiple MPI processes. This complements the existingipyparallelandmultiprocessingbackends.Changes
Core Changes
use_mpiparameter with proper mutual exclusivity check againstuse_ipyparallelanduse_multiprocessingDependencies
mpioptional dependency (mpi4py>=3.0) and included it in theallgroupExamples
--use_mpiCLI argument--use_mpiCLI argumentuse_mpi=FalseparameterUsage
Install with MPI support:
pip install bluepyemodel[mpi] # or pip install mpi4py