The technical evolution of Stable Diffusion

Stable Diffusion technology evolved from Latent Diffusion Models to advanced SDXL and Stable Diffusion 3 versions. Stability AI faced significant financial instability, reporting a $153 million loss in 2023, which preceded CEO Emad Mostaque’s departure in March 2024.

The technical evolution of Stable Diffusion

Stability AI released Stable Diffusion in August 2022. The technology uses a Latent Diffusion Model (LDM) framework. This model uses a Variational Autoencoder (VAE) to compress images into a low-dimensional latent space. It operates on this compressed space rather than raw pixels. This method reduces the computational load compared to traditional diffusion models like DDPM. The system uses a U-Net structure to predict noise during the generation process. Cross-attention layers connect the latent representation to text embeddings from OpenAI’s CLIP. This connection allows the model to translate text prompts into visual features. The developers trained the model on the LAION-5B dataset. This dataset contains 5 billion image-text pairs.

The company released several versions to improve performance. Stable Diffusion 1.1 used 237,000 steps for 256×256 images and 197,000 steps for 512×512 images. Version 1.2 used 515,000 steps for 512×512 images and the laion-improved-aesthetics dataset. Version 1.3 used 195,000 steps for 512×512 images and reduced text adjustment by 10%. Version 1.4 used 225,000 steps for 512×512 images and reduced text adjustment by 10%. Stable Diffusion 2.0 used 550,000 steps for 256×256 images and 850,000 steps for 512×512 images. Version 2.1 is a fine-tuned version of 2.0 that uses an OpenCLIP text encoder. This version allows for 768×768 resolution. In July 2023, the company released SDXL. SDXL uses a two-stage approach that generates a rough structure with a base model and then refines the details with a refiner model to produce high-quality images. In November 2023, the company released SDXL Turbo. This model uses Adversarial Diffusion Distillation (ADD) technology to reduce the generation steps from 30 or 40 down to 1 or 4. The company released Stable Video Diffusion. One version generates 14 frames at 576×1024, while the XT version generates 25 frames at 576×1024. In February 2024, the company released Stable Diffusion Cascade. This model uses an ABC three-stage generation process to produce different styles from a single model. In April 2024, the company released Cos Stable Diffusion XL. This model uses a Cosine-Continuous EDM VPred schedule to generate images from pure black to pure white. In February 2024, the company released Stable Diffusion 3. This model outperforms DALL-E 3 and Midjourney v6 in typography and prompt adherence.

Financial instability and Mostaque’s exit

Emad Mostaque stepped down as CEO and board member in March 2024. He stated he wants to pursue "decentralized AI" to avoid the consolidation of power among big players. His resignation followed months of tension between the leadership and investors over company direction and spending. Stability AI reported a $153 million loss in 2023, while revenue reached only $11 million. As of October 2022, the company burned $8 million per month. By October 2023, the company held less than $4 million in its bank accounts. Mostaque faced accusations of exaggerating company achievements and making misleading claims about partnerships. One co-founder also accused Mostaque of acquiring a 15% stake for only $100. Investors including Lightspeed Venture Partners, Sound Ventures, Coatue, and Intel provided $101 million in October 2022. In late June, reports indicated a May 1st funding event of just under $25 million. The company raised just under $50 million led by Intel. You likely know Stable Diffusion for its image quality, but the company’s financial management was a disaster. In April 2022, Fortune reported that the company sought a $4 billion valuation. Stability AI faced issues with unpaid wages and payroll taxes. Eight former employees reported these delays. The UK tax agency threatened to seize company assets. To generate revenue, the company explored reselling computing power leased from providers like AWS and CoreWeave.

Leadership changes and technical brain drain

High staff turnover characterizes the firm. Robin Rombach and his research team left in March 2024. David Ha, the head of research, also resigned. Shan Shan Wong and Christian Laforte now lead the company as interim co-CEOs. The company implemented layoffs to reduce the workforce by approximately 10%. Stability AI had nearly 200 employees globally in October 2023. In mid-2023, the company saw 13 employees and executives leave. Mostaque’s departure leaves the firm to manage legal challenges from Getty Images in the UK regarding training data. The lawsuit includes a group of artists who dispute the use of copyrighted material. Mostaque’s resignation left the company to decide whether to commercialize SDXL-3 or stay open-source. Will the company find a permanent leader capable of satisfying both developers and investors?

airtrain.ai
airtrain.ai

The airtrain.ai newsroom covers AI research, models and the tools built on them.

More on this topic

Stay ahead of AI

Get the week's most important AI stories delivered to your inbox every Monday.

No spam. Unsubscribe anytime.

More Stories