Thanks for your work on this.
I'm using an M5 Max / 128GB RAM.
Attached is my log. no matter how low I set the memory policy (have gone as low as 0.5), any model I attempt to run that doesn't fit into physical RAM will exhaust memory and crash during the warm-up phase. I've tried manually patching out the warm-up, and of course then the crash happens at the first query. Zero tokens generated when it happens. Appreciate your looking at this. Happy to experiment. Fresh python venv just for this. Followed your steps on building from GH.
Thank you!
Currently attempting to run lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit
mlx-flash --model $MODEL_PATH
--port 8131
--host 0.0.0.0
--kv-bits=4
--cache-budget-pct=0.8
--safety-margin-gb=8.0
--preload
ProductName: macOS
ProductVersion: 26.5
BuildVersion: 25F71
(.venv) zombies@Mac ~ % flash
Password:
2026-05-24T20:27:31.fZ INFO python-worker :8131: MLX-Flash inference server starting model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit port=8131
2026-05-24T20:27:31.fZ INFO python-worker :8131: Hardware detected pressure=normal memory_gb=116.9
2026-05-24T20:27:31.fZ INFO python-worker :8131: Memory before load pressure=normal memory_gb=128.0 action=pre_load_memory
2026-05-24T20:27:31.fZ INFO python-worker :8131: Loading model model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit action=model_load_start
2026-05-24T20:29:25.fZ INFO python-worker :8131: Model loaded model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit load_time_s=114.5 action=model_load_complete
2026-05-24T20:29:25.fZ INFO python-worker :8131: Warming up (compiling Metal shaders) action=warmup_start
[WARNING] Generating with a model that requires 182165 MB which is close to the maximum recommended size of 120832 MB. This can be slow. See the documentation for possible work-arounds: https://github.com/ml-explore/mlx-lm/tree/main#large-models
libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
zsh: abort mlx-flash --model --port 8131 --host 0.0.0.0 --kv-bits=4 --preload
Thanks for your work on this.
I'm using an M5 Max / 128GB RAM.
Attached is my log. no matter how low I set the memory policy (have gone as low as 0.5), any model I attempt to run that doesn't fit into physical RAM will exhaust memory and crash during the warm-up phase. I've tried manually patching out the warm-up, and of course then the crash happens at the first query. Zero tokens generated when it happens. Appreciate your looking at this. Happy to experiment. Fresh python venv just for this. Followed your steps on building from GH.
Thank you!
Currently attempting to run lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit
mlx-flash --model $MODEL_PATH
--port 8131
--host 0.0.0.0
--kv-bits=4
--cache-budget-pct=0.8
--safety-margin-gb=8.0
--preload
ProductName: macOS
ProductVersion: 26.5
BuildVersion: 25F71
(.venv) zombies@Mac ~ % flash
Password:
2026-05-24T20:27:31.fZ INFO python-worker :8131: MLX-Flash inference server starting model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit port=8131
2026-05-24T20:27:31.fZ INFO python-worker :8131: Hardware detected pressure=normal memory_gb=116.9
2026-05-24T20:27:31.fZ INFO python-worker :8131: Memory before load pressure=normal memory_gb=128.0 action=pre_load_memory
2026-05-24T20:27:31.fZ INFO python-worker :8131: Loading model model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit action=model_load_start
2026-05-24T20:29:25.fZ INFO python-worker :8131: Model loaded model=/Users/zombies/Models/LMStudio/models/lmstudio-community/Qwen3-235B-A22B-Thinking-2507-MLX-6bit load_time_s=114.5 action=model_load_complete
2026-05-24T20:29:25.fZ INFO python-worker :8131: Warming up (compiling Metal shaders) action=warmup_start
[WARNING] Generating with a model that requires 182165 MB which is close to the maximum recommended size of 120832 MB. This can be slow. See the documentation for possible work-arounds: https://github.com/ml-explore/mlx-lm/tree/main#large-models
libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
zsh: abort mlx-flash --model --port 8131 --host 0.0.0.0 --kv-bits=4 --preload