Liquid AI Architecture and LFM Models
Liquid AI released its first series of Liquid Foundation Models (LFMs) to handle sequential multimodal data like text, audio, and time series. These models, including the 1.3B, 3.1B, and 40.3B versions, utilize computational units based on dynamical systems, signal processing, and numerical linear algebra. The 40.3B Mixture of Experts (MoE) model uses 12B activated parameters. Unlike transformer architectures, LFMs maintain a minimal memory footprint as input context length grows because they compress inputs efficiently. The LFM-3B model positions itself as first place among 3B parameter transformers, hybrids, and RNN models, and it also outperforms the previous generation of 7B and 13B models and remains 18.4% smaller than the Phi-3.5-mini model.
Ramin Hasani and his colleagues at MIT proposed the Liquid Neural Network (LNN) architecture to address the overparameterization curve in current neural networks. This architecture uses linear first-order differential equations modulated via nonlinear interlinked gates to improve time-series prediction. LNNs utilize the Adjoint method, introduced by Pontryagin in 2018, to optimize the training process. This method introduces an adjoint state on top of the hidden state to carry the sensitivity of the loss. To handle stiff ODEs, researchers use a fused Euler solver. The decay rate of the hidden state $x(t)$ is the reciprocal of the time constant tau, and the parameter $A$ is the learnable weight that updates during training.
| Model | Architecture | Context Length |
|---|---|---|
| LFM-1B | Dense | 32k |
| LFM-3B | Dense | 32k |
| LFM-40B | Mixture of Experts | 32k |
Deployment for BMW and Roche
BMW and Roche deploy Liquid AI technology to manage complex, time-continuous data. BMW works toward a scalable autonomous platform and previously partnered with Intel and Mobileye to test 40 autonomous vehicles. Klaus Frohlich, a member of the BMW board of management, called for other manufacturers and technology companies to join the autonomous platform. Intel CEO Brian Krzanich noted that sharing costs and pooling resources provides speed in development. The Snapdragon Ride Pilot system in BMW’s electric vehicles provides 20 times the computing power of its predecessor. This system uses high-definition cameras and radar sensors to provide a 360-degree view of the surroundings. The technology enables automatic lane changes and hands-free highway journeys through the use of satellite data and maps.
Roche uses AI to manage the drug discovery process, a journey that takes 12 years to move a molecule from research to medicine. The global AI in drug discovery market reaches $1.81 billion this year and grows at a 25% CAGR. Roche researchers screen thousands of synthetic compounds and evaluate candidates through computer programs to find specific biological activities. Clinical testing follows four phases, starting with safety studies in healthy volunteers before moving to large-scale trials with thousands of patients. During Phase II studies, physician investigators observe patients to identify potential risks as early as possible. Once a drug candidate passes these phases, regulatory agencies like Swissmedic, the EMA, or the FDA must approve it for marketing.
Will the LFM architecture eventually replace the transformer as the standard for edge computing?
Performance and Technical Faults
LFM-1B achieves the highest scores in the 1B category on benchmarks like GSM8K, where it scores 55.34 compared to the 31.92 of Danube 2. LFM-3B scores 70.28 on GSM8K, outperforming the 44.28 of Gemma 2 2.6B. LFM-3B also achieves a 78.48 on Hellaswag, which exceeds the 71.31 of Gemma 2 2.6B. LFM-40B achieves an MMLU score of 78.76, which exceeds the 75.75 of Qwen 2. You can observe the memory efficiency when comparing these models to traditional transformers that experience linear memory growth in the KV cache. The LFM models provide a 32k token context length, a capability confirmed by the RULER benchmark.
The $2.5B liquid neural network is a system that delivers superior efficiency for time-series tasks but fails at zero-shot code tasks and precise numerical calculations. Users have also reported technical failures during model deployment. Specifically, JohnLoveJoy reported on the LM Studio bug tracker that LiquidAI models fail to load on Windows 10. This error produces exit code 18446744072635810000.
