Skip to content

fix: eliminate Linux Docker crash and restore MLX on native macOS - #3

Open
king-dopey wants to merge 1 commit into
olafgeibig:mainfrom
king-dopey:feat/linux
Open

fix: eliminate Linux Docker crash and restore MLX on native macOS#3
king-dopey wants to merge 1 commit into
olafgeibig:mainfrom
king-dopey:feat/linux

Conversation

@king-dopey

Copy link
Copy Markdown
Contributor

Problem

Linux Docker containers crashed immediately on startup with:
ImportError: libmlx.so: cannot open shared object file: No such file or directory

Root cause: __init__.py eagerly imported JinaMLXReranker at module level, which triggered a top-level import mlx.core as mx before backend selection could occur. On Linux, MLX shared libraries do not exist.

A secondary regression caused MLX to no longer auto-install on native macOS after dependencies were restructured as optional extras.

Changes

src/local_reranker/init.py

  • Replaced eager from .jina_mlx_reranker import JinaMLXReranker with a __getattr__ lazy-loading implementation so MLX is only imported when actually accessed, making package import safe on Linux/Docker.

src/local_reranker/config.py

  • Added get_mlx_platform_error(): returns an error string when platform is not macOS arm64, None otherwise.
  • Added validate_backend_type(): raises ValueError with an actionable message if mlx backend is selected on an unsupported platform.

src/local_reranker/api.py

  • Moved MLX import inside the mlx branch of the lifespan startup function, after validate_backend_type() is called, so it is never reached on Linux.

src/local_reranker/cli.py

  • Added validate_backend_type() call before server startup so CLI fails fast with a clear error message if mlx is requested on a non-macOS platform.

src/local_reranker/reranker_mlx.py

  • Added init-time platform check via get_mlx_platform_error(); raises RuntimeError with an actionable fallback suggestion if not on macOS arm64.
  • Moved JinaMLXReranker import inside _load_mlx_reranker() with try/except for ImportError/OSError.

src/local_reranker/reranker_pytorch.py

  • Added warnings.filterwarnings to suppress the harmless torch_dtype is deprecated, use dtype instead warning emitted by Jina model files downloaded from HuggingFace.

pyproject.toml

  • Moved MLX dependencies from [project.optional-dependencies] back to main [project.dependencies] with platform markers: mlx>=0.30.0; sys_platform == 'darwin' and platform_machine == 'arm64' This restores auto-installation on native macOS Apple Silicon without requiring a separate install step or extra flag.
  • Pinned transformers>=4.57.0,<5 to avoid breaking API changes in 5.x that affect Jina model loading inside Docker.

Dockerfile

  • Added RERANKER_BACKEND_TYPE=pytorch environment variable.
  • CMD already hardcodes --backend pytorch ensuring the container never attempts to load MLX regardless of any env var override.

tests/test_cli.py

  • Added test_main_unsupported_mlx_backend: verifies mlx backend fails with SystemExit on unsupported platforms.
  • Added test_package_import_keeps_mlx_lazy: confirms jina_mlx_reranker is not present in sys.modules after a bare package import.

README.md

  • Clarified that MLX auto-installs only on native macOS Apple Silicon.
  • Updated Backend Support table to distinguish native macOS (supported) from Docker Desktop on Apple Silicon (not supported, Linux runtime).
  • Removed outdated uv pip install -e ".[dev,mlx]" install instructions.
  • Updated Troubleshooting section to reflect new dependency structure.

## Problem
Linux Docker containers crashed immediately on startup with:
  ImportError: libmlx.so: cannot open shared object file: No such file or directory

Root cause: `__init__.py` eagerly imported `JinaMLXReranker` at module level,
which triggered a top-level `import mlx.core as mx` before backend selection
could occur. On Linux, MLX shared libraries do not exist.

A secondary regression caused MLX to no longer auto-install on native macOS
after dependencies were restructured as optional extras.

## Changes

### src/local_reranker/__init__.py
- Replaced eager `from .jina_mlx_reranker import JinaMLXReranker` with a
  `__getattr__` lazy-loading implementation so MLX is only imported when
  actually accessed, making package import safe on Linux/Docker.

### src/local_reranker/config.py
- Added `get_mlx_platform_error()`: returns an error string when platform is
  not macOS arm64, None otherwise.
- Added `validate_backend_type()`: raises ValueError with an actionable message
  if mlx backend is selected on an unsupported platform.

### src/local_reranker/api.py
- Moved MLX import inside the `mlx` branch of the lifespan startup function,
  after `validate_backend_type()` is called, so it is never reached on Linux.

### src/local_reranker/cli.py
- Added `validate_backend_type()` call before server startup so CLI fails fast
  with a clear error message if mlx is requested on a non-macOS platform.

### src/local_reranker/reranker_mlx.py
- Added init-time platform check via `get_mlx_platform_error()`; raises
  RuntimeError with an actionable fallback suggestion if not on macOS arm64.
- Moved `JinaMLXReranker` import inside `_load_mlx_reranker()` with
  try/except for ImportError/OSError.

### src/local_reranker/reranker_pytorch.py
- Added `warnings.filterwarnings` to suppress the harmless
  `torch_dtype is deprecated, use dtype instead` warning emitted by Jina
  model files downloaded from HuggingFace.

### pyproject.toml
- Moved MLX dependencies from `[project.optional-dependencies]` back to main
  `[project.dependencies]` with platform markers:
    mlx>=0.30.0; sys_platform == 'darwin' and platform_machine == 'arm64'
  This restores auto-installation on native macOS Apple Silicon without
  requiring a separate install step or extra flag.
- Pinned `transformers>=4.57.0,<5` to avoid breaking API changes in 5.x
  that affect Jina model loading inside Docker.

### Dockerfile
- Added `RERANKER_BACKEND_TYPE=pytorch` environment variable.
- CMD already hardcodes `--backend pytorch` ensuring the container never
  attempts to load MLX regardless of any env var override.

### tests/test_cli.py
- Added `test_main_unsupported_mlx_backend`: verifies mlx backend fails with
  SystemExit on unsupported platforms.
- Added `test_package_import_keeps_mlx_lazy`: confirms jina_mlx_reranker is
  not present in sys.modules after a bare package import.

### README.md
- Clarified that MLX auto-installs only on native macOS Apple Silicon.
- Updated Backend Support table to distinguish native macOS (supported) from
  Docker Desktop on Apple Silicon (not supported, Linux runtime).
- Removed outdated `uv pip install -e ".[dev,mlx]"` install instructions.
- Updated Troubleshooting section to reflect new dependency structure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant