Apple hardware provides the necessary compute for modern AI workloads. The M6 chip is the first 2nm chip from Apple and it contains a 12-core CPU complex alongside a 12-core GPU. This chip includes two super cores, four performance cores, and six efficiency cores. It also contains a Dual 16-core Neural Engine that provides twice the peak compute of previous generations. The M6 provides 170GB/s of unified memory bandwidth, which is a 10 percent increase over the M5.
The M5 Ultra is the most powerful chip in the M-series lineup. It uses UltraFusion technology to connect two dual-die M5 Max chips into a quad-die architecture. This architecture provides an inter-die bandwidth of over 4.4TB/s and increases connection density by over 6x. The M5 Ultra contains an up-to-36-core CPU and an up-to-80-core GPU. It provides 1.2TB/s of unified memory bandwidth, which is 50 percent more than the M3 Ultra. The M5 Ultra also contains a 32-core Neural Engine for complex AI tasks.
The M5 chip contains Neural Accelerators in its GPU cores. These accelerators provide dedicated matrix-multiplication operations that are critical for machine learning. The M5 provides a 19-27% performance increase in generation speed compared to the M4 because it has greater memory bandwidth. The M4 provides 120GB/s of memory bandwidth, while the M5 provides 153GB/s. For time-to-first-token (TTFT) generation, the M5 provides a 3.52x to 4.06x speedup for various Qwen models compared to the M4.
| Model | TTFT Speedup | Generation Speedup | Memory (GB) |
|---|---|---|---|
| Qwen3-1.7B-MLX-bf16 | 3.57 | 1.27 | 4.40 |
| Qwen3-8B-MLX-bf16 | 3.62 | 1.24 | 17.46 |
| Qwen3-8B-MLX-4bit | 3.97 | 1.24 | 5.61 |
| Qwen3-14B-MLX-4bit | 4.06 | 1.19 | 9.16 |
| gpt-oss-20b-MXFP4-Q4 | 3.33 | 1.24 | 12.08 |
| Qwen3-30B-A3B-MLX-4bit | 3.52 | 1.25 | 17.31 |
Modular MAX architecture utilizes three layers
The Modular MAX engine is a graph-compiled inference engine. It targets CUDA, ROCm, and Apple Metal from a single Mojo kernel codebase. The engine consists of three specific layers that work together to handle AI workloads.
The first layer is the graph compiler. It takes a PyTorch or ONNX model and processes it through MLIR to produce fused, hardware-specific IR. This layer operates at the computation graph level. It fuses operations that span the Python and CUDA boundary. For example, it can fold RoPE positional embeddings, attention, and output projection into a single kernel pass. This prevents the materialization of intermediate tensors.
The second layer is MAX Serve. This is the OpenAI-compatible serving layer. It exposes the /v1/chat/completions and /v1/completions endpoints. It handles continuous batching and manages the KV cache with a paging mechanism equivalent to PagedAttention. Users can swap MAX Serve in front of any OpenAI client without changing their code.
The third layer is Mojo. Mojo is a Python-superset language that compiles via MLIR to Metal shaders on Apple hardware. Mojo handles SIMD vectorization, memory layout, and target-specific codegen automatically. A Mojo function signature is different from a CUDA C++ signature. It does not require explicit template instantiation or raw pointer arithmetic.
Running LLMs and diffusion models on Mac
The MAX engine supports M1 through M5 Apple silicon GPUs. Appropriately-sized text LLMs, vision models, and image diffusion models run across these devices. You can try a simple LLM on your Mac by setting up MAX or cloning the modular repo. Users can run an invocation to do direct text generation or start serving an endpoint on their machine.
The system uses shared memory between the CPU and GPU. Users can use the --device-memory-utilization and --max-batch-size flags to limit the amount of memory MAX allocates. If a system has at least 15 GB of RAM available after other applications run, it can use the FLUX.2 [klein] 4-billion-parameter image generation model. Users can use the simple_offline_generation example from the modular repo to generate an image.
| Model | Input ($/1K @ 1MP) | Output ($/1K @ 1 MP) |
|---|---|---|
| FLUX.2-dev | 10 | 10 |
| FLUX.2-klein-9B | 6 | 6 |
| FLUX.2-klein-4B | 1 | 1 |
The M5 pushes the time-to-first-token generation under 10 seconds for a dense 14B architecture. It also achieves under 3 seconds for a 30B MoE architecture. Generating subsequent tokens depends on memory bandwidth. The M5 provides 153GB/s of bandwidth, while the M4 provides 120GB/s. The 24GB unified memory in a MacBook Pro with M5 can hold an 8B model in BF16 precision or a 30B MoE 4-bit quantized model. The inference workload stays under 18GB for both architectures.
Mojo enables custom kernel development
Mojo allows developers to write custom attention kernels that fuse RoPE and attention in a single pass. Developers can then register these kernels with MAX Serve. This process does not require CUDA C++, Triton, or pybind11. Mojo provides several tools to simplify this work.
Mojo handles SIMD vectorization and memory layout. The language also manages target-specific codegen. Developers can use the @parameter decorator to signal that a value is known at compile time. This instruction helps the compiler unroll loops and select the correct SIMD width. You can register a custom kernel with MAX Serve by compiling it and passing the path at startup. MAX Serve validates the kernel signature against the model’s attention shape before it loads. If the shapes do not match, the system falls back to the compiled default kernel and issues a warning.
Mojo provides a typed pointer called DTypePointer[dtype] to tell the compiler what memory layout to expect. It does not use raw void* pointers. The @parameter decorator helps the compiler pick the right SIMD width. The vectorize[fn, SIMD_WIDTH] command automatically generates SIMD instructions for the given width. On H100 hardware, this maps to PTX instructions. Advanced users can use mlir_attr to annotate memory access patterns to control cache behavior without using inline CUDA.
Performance benchmarks for M5 and M4
The GPU Neural Accelerators on the M5 chip work with MLX to improve workloads involving large matrix multiplications. This results in up to a 4x speedup for time-to-first-token in language model inference compared to an M4 baseline. Generating a 1024×1024 image with FLUX-dev-4bit is more than 3.8x faster on an M5 than on an M4.
The performance of the M5 varies depending on the architecture of the model. The M5 provides a performance boost for subsequent token generation because of its memory bandwidth. The M5 has 153GB/s of bandwidth, which is 28% higher than the 120GB/s found in the M4. The M5 is better for tasks that are bounded by memory bandwidth rather than compute ability.
| Model | Input ($/1M) | Output ($/1M) | Cache Hit ($/1M) |
|---|---|---|---|
| DeepSeek V4 | 1.74 | 3.48 | 0.145 |
| DeepSeek V4 Flash – Standard | 0.14 | 0.28 | 0.028 |
| Gemma 4 31B | 0.25 | 0.65 | 0.08 |
| Gemma 4 26B A4B | 0.15 | 0.6 | 0.07 |
| GLM 5 | 0.95 | 3.15 | 0.2 |
| GLM 5.1 | 1.3 | 4.3 | 0.26 |
| GLM 5.2 | 1.4 | 4.4 | 0.26 |
| GLM 5.3 | 1.4 | 4.4 | 0.26 |
| GPT OSS 120B | 0.1 | 0.5 | N/A |
| Kimi K2.5 | 0.6 | 3 | 0.12 |
| Kimi K2.6 | 0.85 | 3.5 | 0.16 |
| Llama Guard 4 12B | 0.2 | N/A | N/A |
| MiniMax M2.5 | 0.3 | 1.2 | 0.06 |
| MiniMax M3 | 0.3 | 1.2 | 0.06 |
| NVIDIA Nemotron 3 Super | 0.3 | 0.75 | 0.06 |
| NVIDIA Nemotron 3 Ultra | 0.6 | 3.6 | 0.2 |
| Qwen 3 235B A22B FP8 | 0.2 | 0.6 | N/A |
| Qwen 3.5 9B | 0.17 | 0.25 | N/A |
| Qwen 3.6 Plus | 0.5 | 3 | 0.1 |
| Qwen 3.7-Max | 1.25 | 3.75 | 0.13 |
Deployment and scalability options
Modular provides three different deployment paths for the MAX engine. Users can choose the path that best fits their specific infrastructure and budget requirements.
The first path is Self-Hosted. Users can deploy MAX and Mojo anywhere they want and build with open source. This option is free to use. It allows for full control over the deployment location and the compute hardware. Users can scale on their own using the MAX container. This version provides an active community with fast responses in Discord, Discourse, and Github.
The second path is Modular Cloud. This option allows users to access the Modular Platform through a console for deploying, scaling, and managing AI endpoints. It provides auto-scaling with the ability to scale to zero and provides burst capacity. Pricing for the cloud option is based on per token for shared usage or per minute for dedicated usage. It offers diverse GPU providers optimized for speed and cost.
The third path is Your Cloud. This option allows users to deploy MAX and Mojo in their own environment. It also allows users to use their existing AWS, GCP, or Azure credits and commits. This path provides dedicated support and an engineering team to help train teams and migrate workloads. It also allows for the use of NVIDIA, AMD, Trainium, TPU, Qualcomm GPUs, Intel, AMD, and ARM CPUs.
| Deployment Type | Support | Models | AI Skills | Platform Access | Scalability |
|---|---|---|---|---|---|
| Self-Hosted | Active community | Hundreds of models | Use open AI skills | Deploy anywhere | Scale with MAX container |
| Our Cloud | Dedicated support | Top performers | Modular engineers help | Access via console | Auto-scaling |
| Your Cloud | Dedicated support | Top performers | Modular engineers help | Access via console | Proven at Fortune 500 scale |
Model pricing and availability
The cost of running models through the Modular platform depends on the specific model and the deployment type. The pricing for dense models is often calculated per million tokens.
For DeepSeek V4, the input price is $1.74 per 1M tokens and the output price is $3.48 per 1M tokens. The cache hit price for this model is $0.145 per 1M tokens. The Gemma 4 31B model has an input price of $0.25 per 1M tokens and an output price of $0.65 per 1M tokens. The cache hit price for Gemma 4 31B is $0.08 per 1M tokens.
The Qwen 3.7-Max model has an input price of $1.25 per 1M tokens and an output price of $3.75 per 1M tokens. The cache hit price is $0.13 per 1M tokens. For the MiniMax M3 model, the input price is $0.3 per 1M tokens and the output price is $1.2 per 1M tokens. The cache hit price for MiniMax M3 is $0.06 per 1M tokens.
| Model | Input ($/1M) | Output ($/1M) | Cache Hit ($/1M) |
|---|---|---|---|
| DeepSeek V4 | 1.74 | 3.48 | 0.145 |
| Gemma 4 31B | 0.25 | 0.65 | 0.08 |
| MiniMax M3 | 0.3 | 1.2 | 0.06 |
| Qwen 3.7-Max | 1.25 | 3.75 | 0.13 |
The FLUX.2 models follow a different pricing structure based on the number of tokens. The FLUX.2-dev model costs $10 for input and $10 for output per 1K tokens. The FLUX.2-klein-4B model costs $1 for input and $1 for output per 1K tokens.
Compilation requirements and limitations
The MAX engine requires a compilation step before it can serve requests. The graph compiler processes the model to produce hardware-specific IR. This process takes time and consumes compute resources.
The compilation time varies based on the size of the model. For 8B models, the compilation takes between 5 and 10 minutes. Larger models, such as those with 70B parameters or more, require even more time to compile. Users running on spot instances should mount /opt/max-cache on a persistent volume to avoid paying the compilation cost on every restart. If persistent storage is not available, users must switch to an on-demand instance for production serving.
The current support for M-series chips is progressive. M1 through M5 GPUs are supported in MAX. However, some models might not work well on older M-series SoCs due to a lack of testing. Engineers are working to improve model coverage and performance. Will the integration between NemoClaw and Docker Model Runner arrive soon?
