Unmanaged KV cache and wired memory
I watched a 26B MoE model crash an M4 Max with 64GB of memory during an agentic coding session. The mlx_lm.server wires 75% of the system RAM at startup via mx.set_wired_limit(), which locks 48GB of memory on that 64GB machine. Because the server lacks a –max-kv-size parameter in version 0.31.2, the KV cache grows without an upper bound. When using a model like Gemma 4 26B, which has a hybrid attention architecture with 50 sliding-window layers and 10 global attention layers, the KV cache can exceed 20GB. Since the memory is wired, macOS cannot reclaim it through the Jetsam OOM killer, and the GPU driver hits a reference count underflow in IOGPUMemory.cpp that triggers a kernel panic. You should cap Metal memory allocation before the server starts to prevent these reboots. Using a model with standard grouped-query attention like Qwen3 instead of Gemma 4’s hybrid design provides a more predictable KV cache footprint. This failure mode creates a data integrity risk for developers running 30B+ models for agentic workloads on 64GB machines. I saw the system report no memory pressure before the crash because the wired memory cannot be swapped to disk. This issue is particularly common when using models like Gemma 4 26B, which is an MoE variant with 3.8B active parameters per token.
Formatting errors and training limits
Fine-tuning fails when developers attempt to use GGUF models with MLX. The framework requires models in Hugging Face format. I also found that editing tokenizer_config.json directly in the Hugging Face cache folder causes the huggingface-cli to re-download the entire model because the hash no longer matches. For example, adding a tool_parser_type to a 17.5GB model will trigger a full re-download if you work within the ~/.cache/huggingface/hub/ directory. Copy the model to a local directory first to avoid this. To fix tool parsing issues for Qwen3-Coder, you must add the tool_parser_type entry to the tokenizer_config.json file. Adding this key to the Hugging Face cache directory results in a hash mismatch, so you must copy the model to a local path first. For training, gradient checkpointing trades processing speed for memory usage by storing only specific layer activations. This allows for larger batch sizes. Using LoRA, developers can reduce trainable parameters by 99% or more. QLoRA allows fine-tuning 7B parameter models with as little as 16GB of unified memory by using 4-bit NormalFloat4 quantization. On an M4 Max with 36GB of memory, Llama 3.2 3B hits a throughput of 1100 tokens per second during prefill. MLX also supports many training methods such as DPO, ORPO, and GRPO. Developers can target Llama, Mistral, and Qwen model families. A 128B Mistral Medium 3.5 model takes 158GB of RAM at a 65,536-token context. A 27B Qwen 3.8 model requires 68GB of RAM at the same context size.
| Model | Parameters | Precision | Memory (65k context) |
|---|---|---|---|
| Muse Spark Open | 816B | NVFP4 | 493GB |
| Mistral Medium 3.5 | 128B | FP8 | 158GB |
| Qwen 3.8 | 27B | BF16 | 68GB |
| Gemma 4 26B | 26B | Q4_K_M | 17GB |
Hardware limits and parallelization
The M5 Ultra delivers 1.2 TB/s of unified memory bandwidth, which is a 48.7% increase over the 819 GB/s provided by the M3 Ultra. This bandwidth dictates generation speed. The GPU Neural Accelerator in the M5 architecture provides dedicated matrix-multiplication operations that speed up the time to first token by up to 4.6x compared to the M3 Ultra. For large models like Muse Spark Open, which requires 493GB in NVFP4 precision, you must use parallelization across multiple Mac Studios. Tensor parallelization distributes the model width and requires low-latency communication. Using two Thunderbolt cables per link in a mesh configuration instead of one cut the time to first token for Muse Spark Open from 50.5s to 40.6s at a 65,536-token context. The M5 delivers a 19% to 27% performance boost in token generation compared to the M4 because of its 153 GB/s memory bandwidth compared to 120 GB/s. For comparison, a 26B model on an M4 Max uses 17GB of memory at 4-bit quantization. A 27B Qwen 3.8 model needs 68GB of RAM at a 65,536-token context. The M5 generation includes the GPU Neural Accelerator, which provides dedicated matrix-multiplication operations needed for many machine learning workloads. Why do some developers still ignore the impact of attention architecture on memory headroom?
