Databricks’ $1.3B MosaicML acquisition shifts LLM training costs

Databricks acquired MosaicML for $1.3 billion to lower the cost of training custom large language models. This move allows enterprises to use proprietary data to build models without relying on third-party APIs. The MosaicML platform trained MPT-7B in 9.5 days on 440 GPUs for approximately $200,000. During this run, the platform detected and addressed four hardware failures automatically. Training a 30B parameter model on 600 billion tokens costs about $500,000. The integration of MosaicML into the Databricks platform gives businesses the tools to train models up to 7x faster than standard approaches. While Snowflake focuses on fine-tuning models on customer data and deploying them through Streamlit, Databricks intends to own the entire lifecycle from the initial training phase to the final deployment of the model. MosaicML provides an end-to-end toolchain where businesses can capture, customize, store, label, and train data at a low cost. This platform manages infrastructure scaling via Kubernetes. It also supports multiple clouds including AWS, GCP, and Azure. The MPT series includes the MPT-7B Base, a decoder-style transformer with 6.7B parameters. MosaicML also trained MPT-30B on 256 H100 GPUs using 50 billion tokens.

DBRX performance and technical specifications

The DBRX model uses a fine-grained mixture-of-experts architecture to provide high performance. This model has 132 billion total parameters, but only 36 billion parameters stay active during any given input. Databricks used 3072 NVIDIA H100 GPUs to train DBRX. DBRX beats Llama 2 70B and Mixtral 8x7B on benchmarks for programming, math, and logic. It also outperforms GPT-3.5 on several benchmarks. DBRX uses Rotary Position Encodings, Gated Linear Units, and Grouped Query Attention. It also uses the GPT-4 tokenizer. DBRX uses the MegaBlocks open source project and is twice as compute-efficient as other available leading LLMs.

You already know that proprietary data is the main advantage for enterprise AI. The MPT-30B model follows a decoder-only, transformer-based architecture and outperforms GPT-3 with only 30 billion parameters. This model uses an 8,000-token context length. MPT-30B-Instruct costs $0.005 per 1,000 tokens for inference. Training a 30B parameter model on 600 billion tokens requires 3,000 H100-days. The MPT-7B-StoryWriter-65k+ variant can handle context lengths up to 65,000 tokens. It even demonstrated generations as long as 84,000 tokens on a single node of A100-80GB GPUs. This variant can read The Great Gatsby in about 20 seconds. MPT-7B-Instruct was built by finetuning MPT-7B on a dataset derived from Databricks Dolly-15k and Anthropic’s Helpful and Harmless datasets. MPT-7B-Chat was finetuned using ShareGPT-Vicuna, HC3, Alpaca, Helpful and Harmless, and Evol-Instruct datasets.

Snowflake vs Databricks strategy

Snowflake and Databricks approach the AI market with different philosophies. Snowflake focuses on fine-tuning models on customer data and deploying them through Streamlit. It also launched Snowpark Container Services to run applications directly within its compute clusters. Snowflake acquired Streamlit for $800 million and Neeva for $150 million to build its machine learning stack. Databricks focuses on the end-to-end machine learning process, including training from scratch.

One data engineer noted that Snowflake compute costs can exceed Databricks costs. That same professional stated Snowflake is dramatically cheaper to develop, manage, and operate than Spark-based solutions. Snowflake holds approximately 21% of the market share, while Databricks holds about 11%. Databricks uses an open source data format, which behaves like Android, while Snowflake operates a more closed system similar to Apple. Databricks uses Unity Catalog to manage data and derived embeddings. It also uses the Mosaic AI Agent Framework to coordinate planning, tool use, evaluation, and continuous learning. Databricks’ Mosaic AI stack handles the lifecycle gaps like evaluation, governance, and orchestration. Snowflake uses semantic views based on the Open Semantic Interchange standard. The Cortex Analyst tool uses these semantic views to allow users to talk to data with natural questions via YAML metadata. Will enterprises eventually choose the flexibility of open formats over the ease of managed services?