1 - Modular : Sub-Packages

[MementoRC]

…iew of the ci-base->modular delta (sub-package wiring). CI state preserved so the PR surfaces gate failures. Regenerated each rebuild; not a hand-merge target.

Before submitting this PR, please make sure:

• Your code builds clean without any errors or warnings
• Tests all pass
• You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

Tests performed by the developer:

Tips for QA testing:

Summary by Sourcery

Introduce modular sub-package wiring and compatibility checks while updating MQTT integration and maintained external event handling behavior.

New Features:

• Add sub-package compatibility CI job with import, contract, and integration checks across all modular sub-packages.
• Provide a compat shim for the legacy hummingbot.core.api_throttler import path backed by the web_assistant throttler implementation.

Enhancements:

• Refine MQTT bridge creation to use the new remote_iface gateway configuration, with safer teardown on failures.
• Update strategy and controller external event/topic listeners to consume MQTT instances from the main application and handle missing MQTT gracefully.
• Extend pixi tasks to install all sub-packages in editable mode and run comprehensive compatibility tooling.
• Adjust Backpack connectors typing signatures to use built-in dict type hints for in-flight and lent balances.

CI:

• Add a dedicated sub-packages compatibility workflow to enforce modular package import and contract correctness and gate CI success on its outcome.

Tests:

• Introduce a ci/check_subpackage_compat.py script that validates sub-package imports and candles-feed API contracts, and wire it into new Pixi test tasks.

Chores:

• Configure sub-packages as git submodules to support the modular repository layout.

[sourcery-ai]

Reviewer's Guide

This PR modularizes MQTT and sub-package integration, adds a compatibility shim for api throttling, and introduces CI + pixi tasks to validate compatibility of new sub-packages while preserving existing interfaces.

Sequence diagram for updated MQTT gateway startup

sequenceDiagram
    actor User
    participant HummingbotApplication
    participant mqtt_command
    participant RemoteIface
    participant Broker

    User->>HummingbotApplication: start_mqtt_bridge
    HummingbotApplication->>mqtt_command: start_mqtt_async(timeout)
    mqtt_command->>mqtt_command: build BrokerConfig
    mqtt_command->>mqtt_command: build GatewayConfig
    mqtt_command->>RemoteIface: create_gateway(self, config, broker_config)
    RemoteIface-->>mqtt_command: gateway

    alt [connection successful]
        mqtt_command->>gateway: start
        mqtt_command->>HummingbotApplication: set _mqtt = gateway
    else [connection failed]
        mqtt_command->>gateway: stop
        mqtt_command->>HummingbotApplication: set _mqtt = None
    end

Loading

File-Level Changes

Change	Details	Files
Refactor MQTT gateway usage and external events to go through the new remote_iface-based gateway and HummingbotApplication._mqtt instance, with graceful handling when MQTT is unavailable.	Replace imports from hummingbot.remote_iface.mqtt with remote_iface factories and protocols. Initialize ExternalEventFactory and ExternalTopicFactory using HummingbotApplication.main_application()._mqtt and store queues/listeners on the strategy/controller instance. Guard queue processing and listener removal when MQTT is not configured or None. Adjust logging strings to consistent double-quote style.	scripts/external_events_example.py hummingbot/client/command/mqtt_command.py controllers/directional_trading/ai_livestream.py
Add a compat shim module mapping hummingbot.core.api_throttler to web_assistant.throttler to keep legacy import paths working under the new modular layout.	Create hummingbot.core.api_throttler package that re-exports async throttler classes and data types from web_assistant.throttler.*. Register submodules in sys.modules so old dotted imports (async_throttler, data_types, etc.) resolve to web_assistant implementations. Define all to match the expected public API.	hummingbot/core/api_throttler/__init__.py
Introduce CI workflows and pixi tasks to install all sub-packages in editable mode and run compatibility checks (import, contract, integration) against the modularized sub-packages.	Add pixi tasks to install all 14 sub-packages in editable mode, then install the main package, and run compat-import, compat-contract, compat-integration, and aggregate compat-check. Add a sub-packages-compat GitHub Actions job that runs compat-import and compat-check, gated by detect-changes and included in CI success conditions. Implement ci/check_subpackage_compat.py to smoke-test imports for all sub-packages and validate candles-feed contracts and DataFrame output. Update ci-success job to fail when the sub-packages-compat job fails.	.github/workflows/workflow.yml pyproject.toml ci/check_subpackage_compat.py
Adjust type hints in Backpack connectors to use built-in dict syntax instead of typing.Dict and remove unused typing imports.	Change function signatures and local variable typings from Dict[str, Decimal] to dict[str, Decimal]. Remove unused Dict import from backpack_perpetual_derivative and backpack_exchange modules.	hummingbot/connector/derivative/backpack_perpetual/backpack_perpetual_derivative.py hummingbot/connector/exchange/backpack/backpack_exchange.py
Add sub-packages as git submodules/directories to support modularized code layout used by pixi and compat checks.	Introduce .gitmodules and placeholder entries for each sub-package directory. Add sub-packages/* directories (async-utils, candles-feed, connector-utils, etc.) to the repository structure.	.gitmodules sub-packages/async-utils sub-packages/candles-feed sub-packages/connector-utils sub-packages/data-type-primitives sub-packages/event-bus sub-packages/liquidations-feed sub-packages/logger sub-packages/market-connector sub-packages/market-data sub-packages/market-simulator sub-packages/rate-oracle sub-packages/remote-iface sub-packages/strategy-framework sub-packages/web-assistant

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In the new MQTT wiring (e.g., external_events_example and ai_livestream), consider centralizing the HummingbotApplication.main_application()/_mqtt lookup in a helper to avoid repeated inline imports and reduce the risk of inconsistent null handling across different components.
• The compatibility shim in hummingbot/core/api_throttler/__init__.py relies on sys.modules mutation; it would be safer to add a brief comment or guard explaining when this is expected to run to avoid surprising behavior if the module is imported before web_assistant is available.
• In ci/check_subpackage_compat.py, the mode handling via sys.argv[1] could be made more robust (e.g., using argparse and validating allowed modes) to prevent confusing failures when the script is invoked with unexpected or missing arguments in CI.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In the new MQTT wiring (e.g., external_events_example and ai_livestream), consider centralizing the `HummingbotApplication.main_application()`/`_mqtt` lookup in a helper to avoid repeated inline imports and reduce the risk of inconsistent null handling across different components.
- The compatibility shim in `hummingbot/core/api_throttler/__init__.py` relies on `sys.modules` mutation; it would be safer to add a brief comment or guard explaining when this is expected to run to avoid surprising behavior if the module is imported before `web_assistant` is available.
- In `ci/check_subpackage_compat.py`, the mode handling via `sys.argv[1]` could be made more robust (e.g., using argparse and validating allowed modes) to prevent confusing failures when the script is invoked with unexpected or missing arguments in CI.

## Individual Comments

### Comment 1
<location path="ci/check_subpackage_compat.py" line_range="93-98" />
<code_context>
+    return failures
+
+
+def check_contracts() -> list[str]:
+    """Verify expected public API classes have required methods/attributes."""
+    failures = []
+
+    # --- CandlesFeed contract ---
+    from candles_feed.core.candles_feed import CandlesFeed
+
+    required_methods = {
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Direct imports in `check_contracts` will raise unhandled ImportErrors instead of producing structured failure messages.

These direct imports in `check_contracts` aren’t guarded, so a missing or misinstalled `candles_feed` will crash the script instead of being reported as a structured failure. Consider matching `check_imports`: wrap the imports in `try/except ImportError`, record a clear failure message, and return normally so the summary and exit code capture the issue without a stack trace.

Suggested implementation:

```python
    # --- CandlesFeed contract ---
    try:
        from candles_feed.core.candles_feed import CandlesFeed
    except ImportError as e:
        failures.append(
            f"FAIL: Cannot import CandlesFeed from candles_feed.core.candles_feed: {e}"
        )
        return failures

```

If more contract checks are added later that also perform imports inside `check_contracts`, they should follow the same pattern: wrap each import in `try/except ImportError`, append a structured failure message, and either `return failures` immediately or continue depending on whether a missing import should block subsequent checks.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

suggestion (bug_risk): Direct imports in check_contracts will raise unhandled ImportErrors instead of producing structured failure messages.

These direct imports in check_contracts aren’t guarded, so a missing or misinstalled candles_feed will crash the script instead of being reported as a structured failure. Consider matching check_imports: wrap the imports in try/except ImportError, record a clear failure message, and return normally so the summary and exit code capture the issue without a stack trace.

Suggested implementation:

    # --- CandlesFeed contract ---
    try:
        from candles_feed.core.candles_feed import CandlesFeed
    except ImportError as e:
        failures.append(
            f"FAIL: Cannot import CandlesFeed from candles_feed.core.candles_feed: {e}"
        )
        return failures

If more contract checks are added later that also perform imports inside check_contracts, they should follow the same pattern: wrap each import in try/except ImportError, append a structured failure message, and either return failures immediately or continue depending on whether a missing import should block subsequent checks.