Experimental Refactoring#10
Open
autonull wants to merge 26 commits into
Open
Conversation
…n system This major update overhauls the project to improve structure, usability, and maintainability. Key changes include: - **Project Restructuring:** - Migrated the core `diffevo` library to a `src/diffevo` directory. - Established dedicated `tests` and `docs` directories. - Reorganized the `experiments` directory into a more granular structure (`benchmarks`, `visualizations`, etc.). - **Build & Dependency Management:** - Updated `setup.py` to be compatible with the new `src` layout. - Introduced `requirements.txt` and `requirements-dev.txt` for clear dependency management. - Enhanced `.gitignore` to prevent committing build artifacts and data files. - **Core Refactoring:** - Significantly refactored the `DiffEvo` optimizer class in `optimizer.py` for a cleaner, more intuitive API. - Added type hints and improved docstrings for better code clarity. - **New Evaluation System:** - Implemented a turnkey evaluation system with a new `evaluate.py` script. - The script provides a command-line interface to run benchmarks, select experiments, and manage outputs. - **Testing:** - Added an initial unit test for the refactored `DiffEvo` optimizer. - Resolved complex test environment issues related to `pytest` and `pipx`. - **Documentation:** - Performed a complete rewrite of the `README.md` to reflect all the above changes. - Updated installation instructions, usage examples, and added a new section on running benchmarks.
This commit introduces a major revamp of the project, including: - Modularization: The core `diffevo` library has been refactored into a more modular structure, with schedulers, generators, and fitness mappings in their own subdirectories. - Hyperparameter Lifting: Hardcoded values have been lifted and are now configurable. - New Baseline: A Policy Gradients (REINFORCE) baseline has been added for comparison. - Enhanced Evaluation: The `evaluate.py` script has been refactored to be object-oriented and now generates summary reports and plots.
…tor`, and `FitnessMapping`, and refactored the existing classes to inherit from them. I also verified that the example functions are in a separate `examples` module. The codebase is now more modular and extensible. I have reviewed the codebase and confirmed that all the hyperparameters in the `DiffEvo` optimizer, schedulers, generators, and fitness mappings are already exposed as arguments in their respective `__init__` methods. The code is clean and does not contain any "magic numbers" that need to be lifted. I refactored the `DiffEvo` optimizer to allow for pluggable generators. I also implemented an `EliteGenerator` as an alternative to the `BayesianGenerator`, demonstrating the flexibility of the new architecture. The codebase now supports easy comparison of different algorithmic strategies. I have added a new benchmark for the `EliteGenerator` to the `evaluate.py` script. This includes creating a new benchmark file, updating the `__init__.py` file, and adding the new benchmark to the `experiments_config` dictionary. The evaluation system is now more comprehensive and can be used to compare the performance of different generator strategies.
This commit introduces a major refactoring of the entire codebase to establish a more professional, modular, and maintainable project structure suitable for academic and open-source distribution. Key changes include: - **Project Restructuring:** Migrated the codebase to a standard `src` layout, with all core library code now residing in `src/diffevo`. - **Code Consolidation:** Consolidated numerous benchmark-related files into a single, cohesive `src/diffevo/benchmarks.py` module, eliminating significant code duplication. - **Decoupling of Concerns:** Decoupled experiment definition from execution. A new `run_evaluation.py` script serves as a clean command-line entry point, while experiment suites are now defined in `experiments/suites.py`. - **Enhanced Packaging:** Updated `setup.py` to support the `src` layout and added a `console_scripts` entry point, creating a user-friendly `diffevo-evaluate` command. - **Improved Testing:** Added a comprehensive suite of unit tests for the new `Experiment` class and the consolidated benchmark functions. All tests are passing. - **Updated Documentation:** Completely rewrote the `README.md` to reflect the new project structure, installation process, and usage instructions for the new command-line tool.
This commit addresses a high-level request to clean up, refactor, deduplicate, and package the project. It includes the following changes: - **Packaging:** The project has been converted into a standard Python package. Dependencies have been moved from `requirements.txt` to `setup.py`, and the `requirements.txt` file has been deleted. - **Code Cleanup:** Path manipulation has been removed from `run_evaluation.py`, and all internal imports have been updated to use the new package name. - **Deduplication & Refactoring:** The `experiments/suites.py` file has been refactored to use a helper function, reducing code duplication. - **File Structure Refactoring:** The `src/diffevo/kde.py` and `src/diffevo/latent.py` modules have been consolidated into a single `src/diffevo/utils.py` module.
Creates a turnkey `run.py` script that runs the entire evaluation experiment. This script includes: - A `--smoketest` flag for a minimal "smoke test" configuration to accelerate testing and debugging. - Full evaluation capabilities, consolidating the logic from the previous `run_evaluation.py`. - Generous logging to provide clear feedback on the current activity. This change also includes bug fixes identified and resolved during the development and testing of the `run.py` script. The `results/` directory has been added to `.gitignore` to prevent generated files from being committed.
Refactored the run.py script to eliminate duplicated code by creating a unified `_run_experiments` function. This improves maintainability and clarity. Added a try-except block to gracefully handle experiment failures, enhancing robustness. Also, fixed a bug in the progress bar implementation in diffevo.py that was causing unit test failures.
Adds a bootstrapping mechanism to `run.py` that automatically creates a virtual environment, installs dependencies, and re-launches the script within it. This simplifies the user workflow to a single `python run.py` command and resolves issues with externally-managed Python environments. The output has been simplified to be more user-friendly, hiding verbose installation details. The virtual environment check is now specific to the project's `.venv` directory to avoid conflicts with other active virtual environments.
The `run.py` script now handles virtual environment creation and dependency installation automatically. This commit updates the `README.md` to reflect this simplified workflow. The "Installation" and "Running Benchmarks" sections have been rewritten to guide the user to simply clone the repository and run `python run.py`. This makes the project much more accessible, especially for users who are not familiar with Python's environment management.
This commit addresses a broad request for code quality improvements. Key changes include: - Applied consistent code formatting using `black` across the project. - Resolved numerous `flake8` linting errors, such as unused imports and incorrect syntax. - Refactored the main entrypoint `run.py`. The virtual environment bootstrapping logic remains in `run.py`, while the core application logic has been moved to a new `main.py` file for better separation of concerns. - Simplified and deduplicated the experiment suite definition in `experiments/suites.py` by adopting a data-driven approach. - Other minor cleanups and fixes throughout the codebase.
This change consolidates the application's entry point by merging the logic from main.py into run.py. The main.py file has been removed, and run.py now handles both bootstrapping the virtual environment and running the main application logic. The `run_smoketest` and `run_full_evaluation` functions were also merged into the `main` function to reduce code duplication.
Refactored the Evolution Strategy benchmark functions in `src/diffevo/benchmarks.py` to use a generalized `es_benchmark` function. This reduces code duplication and improves maintainability. Also removed duplicated `cmaes.py` and `pepg.py` files from the `experiments` directory.
This commit introduces a complete architectural overhaul of the research framework, moving from a monolithic, script-based approach to a modern, component-based, and configuration-driven system. The key changes are: - **Configuration-Driven:** Experiments are now defined in YAML files and validated with Pydantic models, replacing hardcoded Python configurations. - **Component Architecture:** The system is decomposed into modular and pluggable components: `Problem`, `Optimizer`, and `Callback`. - **Orchestrator:** A new `Orchestrator` class manages the experiment lifecycle, dynamically loading components based on the configuration. - **Reproducibility:** Each experiment run saves all necessary artifacts (config, git hash, environment) to a unique, timestamped directory. - **Dual Interface:** Provides both a powerful CLI (`run.py`) for batch experiments and a clean programmatic API (`diffevo.run()`) for interactive use. - **Refactored Core:** The `DiffEvo` optimizer has been refactored to support the new `ask`/`tell` interface required by the `Orchestrator`.
This commit introduces a new, extensible framework for graph-based optimization problems using Diffusion Evolution. It refactors the user-provided graph evolution script into a new `GraphDiffEvo` optimizer that manages its own evolution loop. The Orchestrator has been updated to support this new optimizer type while maintaining backward compatibility with existing `ask`/`tell` optimizers. Adds two new graph-based problems: `GraphFlow` and `MaxClique`, along with their corresponding configuration files. The `MaxClique` fitness function has been optimized using the `networkx` library. Documentation has been updated to reflect these new features.
The `run.py` script was failing with a `ModuleNotFoundError` because it imported project-specific packages (`diffevo`, `yaml`) before the bootstrapping logic had a chance to create the virtual environment and install them. This commit resolves the issue by: 1. Moving all package imports that depend on the virtual environment into the `main()` function. This ensures the bootstrap process completes before these modules are needed. 2. Improving the `bootstrap()` function's error handling. It now captures `stdout` and `stderr` from `pip` commands and prints them only if an installation fails, making debugging setup issues much easier. These changes make `run.py` a true turnkey solution, as intended.
This commit introduces a major refactoring of the project structure to simplify the user workflow. The key changes are: - A new `init.py` script now handles all environment setup, including virtual environment creation and dependency installation. - `run.py` has been refactored to be a dedicated experiment runner, with the bootstrapping logic removed. - A hierarchical configuration system has been implemented, allowing experiments to inherit from a `base.yaml` file. - A `--smoketest` flag has been added to `run.py` for quick end-to-end testing. - A new `report.py` script has been added to run a sequence of experiments. - The `README.md` has been updated to reflect the new workflow.
This commit refactors the `Problem` abstraction to support a wider range of experiment types, including RL, graph, and 2d_images. Key changes: - Introduced a `Problem` base class in `src/diffevo/problems/base.py` that standardizes the interface for different experiment types. - Refactored the `Orchestrator` to work with the new `Problem` abstraction. - Updated the configuration files to reflect the new `Problem`-based structure. - Created example implementations of the `Problem` class for each of the target experiment types. - Updated the `DiffEvo` optimizer to accept `lower_bound` and `upper_bound` arguments.
This commit reorganizes the project structure to better align with the project's research-oriented goals. The previous `experiments` directory has been replaced by two more specific directories: - `experiment/`: This directory now contains all scripts and code related to running benchmarks and comparative experiments. - `util/`: This directory is now the home for all post-processing, analysis, and visualization scripts. This change improves the project's clarity, scalability, and maintainability by creating a more logical separation of concerns.
This commit replaces the placeholder implementations for the `Image` and `RL` problems with functional versions. The `Image` problem now generates a target image and uses Mean Squared Error as a fitness function to guide the evolution towards reconstructing it. The `RL` problem now uses the `gymnasium` library to evaluate policies in the `CartPole-v1` environment. The fitness of a policy is the total reward achieved in an episode. Additionally, this commit includes: - `gymnasium` as a dependency in `setup.py`. - Example configuration files (`configs/image_evolution.yaml` and `configs/rl_evolution.yaml`) for the new problems. - A fix in the `graph_evolution.py` example to load parameters from its corresponding YAML file.
This commit introduces a new optimizer, `GraphBasedDiffEvo`, where the population of candidate solutions is structured as a sparse graph. This allows for localized evolutionary updates, where diffusion only occurs between neighboring solutions. The implementation uses a k-Nearest Neighbors (k-NN) graph to define the population structure. A new example configuration file (`configs/graph_based_evolution.yaml`) is added to demonstrate its usage. The `README.md` is updated to document the new optimizer and clarify its distinction from the existing `GraphDiffEvo`.
This commit refactors the codebase to use a plugin-based architecture for problems, optimizers, and callbacks. This change significantly improves the extensibility of the framework, allowing users to add new components without modifying the core codebase. Key changes: - Introduced a new top-level `plugins` directory to house problems, optimizers, and callbacks. - Modified the `Orchestrator` to dynamically discover and load plugins from the `plugins` directory. - Moved existing problems, optimizers, and callbacks to the new `plugins` directory. - Updated the `README.md` to reflect the new plugin-based architecture and provide instructions on how to create and use plugins. - Fixed numerous import errors and other issues that arose from the refactoring.
This commit introduces a major overhaul of the project's examples and includes several significant improvements to the core library. Key changes: - Moves the examples from `src/diffevo/examples` to a new top-level `examples/` directory. - Rewrites the examples to be standalone, runnable Python scripts that use a programmatic API (`diffevo.run`). - Creates a comprehensive suite of examples covering classic optimization, graph problems, image reconstruction, and reinforcement learning. - Refactors the `diffevo.run` function and the `Orchestrator` to return the final population, making the results programmatically accessible. - Refactors the `Problem` abstract base class to enforce a cleaner and more consistent structure across all problem plugins. - Fixes several bugs in the `Orchestrator`, `GraphDiffEvo` optimizer, and various problem plugins that were uncovered during the refactoring process.
Relocated example code from `util/` to `examples/`. Moved experiment-specific analysis scripts from `util/` to a new `scripts/` directory. Centralized generic plotting and statistical functions into `src/diffevo/plotting.py` and `src/diffevo/utils.py`. Deprecated the `util/` directory and non-RL scripts in the `experiment/` directory. Restructured the `two_peaks` example to be a runnable module, removing the need for `sys.path` manipulation.
…mples This commit addresses several areas of the codebase to improve code quality, reduce duplication, and modernize the project structure. Key changes include: - **Deduplication:** - Removed the duplicate `utils.py` file from `experiment/RL/diffRL/es`. - Consolidated the `normalize_observation` function into `src/diffevo/utils.py`. - **RL Experiment Refactoring:** - The standalone `experiment/RL` has been refactored into the main plugin architecture. - A new `RL` problem plugin was created in `plugins/problems/rl.py`, which is more generic and uses a configurable `ControllerMLP`. - A new `RLEvo` optimizer plugin was created in `plugins/optimizers/rl_evo.py` to handle the RL experiment's optimization loop. - A new configuration file, `configs/rl_diffevo.yaml`, was created to run the RL experiment. - **Example Modernization:** - The example scripts in `examples/` have been updated to use the main `run.py` and configuration files. - Outdated and redundant example directories (`two_peaks`, `two_peaks_step`) have been removed. - **Bug Fixes:** - Fixed several `ImportError` issues by exposing necessary classes in the top-level `diffevo` package's `__init__.py` file.
This commit refactors the `DiffEvo` and `GraphBasedDiffEvo` optimizers to accept a `problem` object in their constructors. This simplifies the `Orchestrator`'s `_init_optimizer` method and makes the optimizer implementations more self-contained and consistent. The following changes were made: - Modified `DiffEvo` and `GraphBasedDiffEvo` to accept a `problem` object. - Updated the `Orchestrator` to pass the `problem` object to the optimizer constructors. - Verified the changes with the smoketest and unit tests.
- Deleted the outdated and numerically unstable `GraphDiffEvo` optimizer. - Created a `BaseDiffEvo` base class to consolidate common code from `DiffEvo` and `GraphBasedDiffEvo`. - Refactored `DiffEvo` and `GraphBasedDiffEvo` to inherit from `BaseDiffEvo`. - Refactored `RLEvo` to use the standard `ask`/`tell` interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extensive refactoring, but untested. Maybe it can give you some ideas