Mojo and the MAX engine bring AI performance to Apple M4 Max

Mojo addresses the performance gap

Mojo achieves speeds 35,000 times faster than Python on Mandelbrot set benchmarks. Chris Lattner designed Mojo to bridge the performance gap in AI infrastructure, specifically targeting the requirements of ML engineers and researchers. The language uses MLIR to compile code to machine code, whereas Python relies on an interpreter. You can run Python code inside Mojo programs because it acts as a superset. While Python uses the Global Interpreter Lock (GIL) to limit threading, Mojo provides developers with direct control over memory allocation and SIMD operations. This control allows Mojo to outperform NumPy on vector addition tasks. For a 1000-dimensional vector addition, a SIMD-optimized Mojo implementation reaches 13.33 GFLOP/s, beating the 1.46 GFLOP/s of NumPy. Python developers often rely on C extensions like NumPy to bypass the GIL, but Mojo removes this dependency by offering native, hardware-level SIMD operations and fine-grained control over memory layout. This capability addresses the "two-language tax" in quantitative finance, where researchers must rewrite Python models in C++ to avoid the performance overhead of Python-to-C boundaries. Such rewrites consume multiple engineering quarters and often introduce numerical discrepancies in production because Python and C++ handle floating-point operations differently.

Apple Silicon GPU integration

Mojo nightly releases include support for Apple Silicon GPUs, covering the M1 through M4 series. To use these capabilities, developers require macOS 15 or newer and Xcode 16 or newer. The Metal Shading Language version 3.2 used by the compiler requires the macOS 15 SDK. You can test GPU functions by cloning the Modular repository and accessing the examples/mojo/gpu-functions directory. Most examples work on Apple Silicon today, though reduction.mojo and complex matrix multiplication examples do not. Mojo handles hardware interaction through the DeviceContext type. For Apple Silicon, the specialized MetalDeviceContext uses the Metal-cpp API to compile the AIR representation into a .metallib for execution on device, while it also manages a Metal CommandQueue and buffers operations for moving data and running GPU functions. Modular also works with hardware vendors through the Modular alliance program to build LLVM backends. This allows for high-quality register allocation and scheduling across various architectures. Technical blockers currently prevent the use of certain features, including MAX_THREADS_PER_BLOCK_METADATA, GridDim, and lane_id. Developers must also wait for support for bfloat16 on ARM devices, atomic operations, and the full implementation of MetalDeviceContext::synchronize. This support enables new paths of local-to-cloud development for AI models.

Kernel fusion and M4 Max hardware

The Apple M4 Max chip provides a CPU and GPU core-count boost and delivers a 16-core neural engine with 38 TOPS. This surpasses the 18 TOPS of the M3 Max. The M4 Max also delivers a 75% boost in memory bandwidth. AI models use kernel fusion to combine multiple layers into a single GPU kernel to reduce DRAM usage. This technique aims to eliminate the latency overhead from launching many discrete kernels. However, large fused kernels struggle with dynamic input shapes and synchronization. Because CUDA only supports synchronization within a single SM, splitting a fused computation across multiple SMs can lead to deadlocks. In real-time, low-batch inference, the amount of available work per SM is too small to hide memory latency. On an Nvidia L40S, an optimized Triton kernel might use only 128 blocks, leaving 14 SMs idle. This results in a throughput of approximately 150 TFLOPS, which is less than half of the chip’s 362 TFLOPS rating. This performance gap occurs because there are not enough active warps to hide memory latency. What remaining hardware features will the next Mojo release prioritize for Apple Silicon?