LLM
-
Memory Exhaustion and Communication Bottlenecks in Llama 3.1 405B
Llama 3.1 405B requires 810 GB of memory for FP16 precision, necessitating distribution across multiple SambaNova SN40L RDUs. Improper sharding in feedforward…
-
Wasted Infinity Fabric bandwidth in Mixtral 8x22B clusters
Engineers often stall Mixtral 8x22B performance by ignoring ROCm 7 kernel optimizations and incorrect NUMA modes. Proper configuration of AITER kernels and…
-
Llama 4 deployment priorities for local inference
Meta’s Llama 4 models offer efficient local enterprise deployment through mixture-of-experts architectures. Llama 4 Maverick outperforms GPT-4o and Gemini 2.0 Flash on…
-
Deploying Mistral AI Codestral 22B on Azure AI Foundry
Azure AI Foundry provides Serverless API and managed compute paths for deploying Mistral AI’s Codestral 22B. This 22B parameter model achieves an…
-
The Cerebras CS-3 delivers 2,500 tokens per second for Llama 4 Maverick
Cerebras CS-3 delivers Llama 4 Maverick inference at 2,500 tokens per second per user, significantly outperforming the NVIDIA DGX B200. The WSE-3…
-
The hardware economics of Tenstorrent Blackhole AI chips
Tenstorrent Galaxy Blackhole systems achieve $6 per million tokens for DeepSeek R1 671B, significantly undercutting NVIDIA’s implied $30 per million token cost.…
-
Baidu’s research foundation and the rise of Chinese AI models
Baidu established its Institute of Deep Learning in 2013, later hiring Andrew Ng to lead research. This foundation helped develop ERNIE 4.5,…
-
Sakana AI optimizes model performance through evolutionary merging
Sakana AI utilizes evolutionary techniques to combine open-source models, creating specialized systems like the EvoLLM-JP-v1-7B which outperforms larger 70B parameter models. The…
-
Databricks’ $1.3B MosaicML acquisition shifts LLM training costs
Databricks leverages its $1.3 billion MosaicML acquisition to enable enterprises to train custom large language models up to 7x faster. The platform…
-
DeepSeek V3 architecture and enterprise LLM deployment economics
DeepSeek-V3 utilizes a 671B parameter Mixture-of-Experts architecture to drive inference efficiency. DeepSeek-V4-Flash offers significant cost advantages for enterprises, with input costs at…
-
Deploying open-source LLMs with Tenstorrent Wormhole and TT-Metalium
Tenstorrent Wormhole n150 hardware and the TT-Metalium SDK enable efficient local LLM inference. The hardware features Tensix cores and GDDR6 memory, allowing…
-
Common mistakes in Databricks DBRX 132B MoE fine-tuning
Engineering teams lose their DBRX 132B MoE advantage when neglecting Spark-based data preparation or ignoring the 32,000 token context limit. Proper management…