Mixture-of-experts (MoE) architectures increase model capacity by activating only specialized submodels, or experts, for each token. Instead of running every parameter during every step, a learned routing mechanism selects specific subnetworks to participate. This selective activation reduces computational overhead and lowers inference costs compared to dense models. Most top-performing models today use MoE to deliver higher quality while maintaining manageable computation. This architecture allows researchers to build models with trillions of parameters without increasing training or inference time proportionally.
The gating network examines attention outputs and routes each token to a targeted subset of experts. During training, experts specialize in areas like different languages, code, or sentiment analysis. This process requires careful load balancing to ensure all experts in the network receive comparable training exposure. In production, a model can spread experts across many GPUs, but this distribution places strain on memory bandwidth. Together AI provides the infrastructure for these workloads, utilizing data centers across the U.S. and Europe to provide more than 20 exaflops of compute power.
The company closed a $106 million funding round led by Salesforce Ventures, which brought its valuation to $1.25 billion. It also maintains a cumulative funding total of approximately $1.58 billion following a Series C round in April 2026. The involvement of Nvidia drives the success of the platform. CEO Vipul Prakash leads the company, which began in 2022. Together AI hosts these models for over 450,000 registered users.
Catalog Breadth and Inference Economics
Together AI competes on catalog breadth, hosting over 200 open-source models. I find its catalog much larger than Fireworks AI, which hosts around 20 models. When a team evaluates the decision between specialized providers, they must look past the headline per-token rate to consider how latency stability, custom model deployment, and specific model families impact the total cost of ownership for high-volume production workloads. For users needing specific models for code, math, or distinct languages, Together AI includes more options. The platform includes a batch inference API at 50% of the standard serverless pricing, making it the cheapest option for workloads that tolerate multi-hour latency.
| Model | Input (per 1M) | Output (per 1M) |
|---|---|---|
| Llama 3.1 8B Instruct | $0.18 | $0.18 |
| Llama 3.1 70B Instruct | $0.88 | $0.88 |
| Llama 3.1 405B | $3.50 | $3.50 |
| Llama 4 Scout | $0.11 | $0.34 |
I would skip Together AI for latency-sensitive applications because its P50 Time to First Token (TTFT) on Llama 3.3 70B measures 220ms, whereas Fireworks AI hits 150ms. If your production requirement demands the absolute lowest p95 latency for open-source models, Fireworks AI remains the better choice. Together AI remains 5% to 15% cheaper than Fireworks on most comparable models. Groq provides even faster speeds, reaching 200 to 400 tokens per second on Llama 3 70B. Groq’s Llama 3.1 8B pricing sits at $0.05 for input and $0.08 for output. Do you need the cheapest token rate or the fastest response?
Customizing Models for Specialized Workloads
The Together AI Fine-tuning API allows teams to customize open models using their own datasets. This API supports conversational and instruction data formats, which eliminates the need for manual reformatting of examples. The platform supports up to 32k context length for Llama 3.1 8B and 70B fine-tuning. In one experiment, a fine-tuned Llama 3.1-8B outperformed its 70B base counterpart by over 10% in ROUGE score on a synthetic repetition task. This capability helps small, efficient models reach the performance levels of much larger models.
Users can use the API to feed conversation histories or instruction datasets directly. The platform includes validation dataset support to monitor model loss on unseen data during training. You can customize training jobs with parameters like weight decay and gradient clipping. The API also includes an automated batch size setting to improve training efficiency. For teams iterating on fine-tuned models, the platform includes dedicated GPU endpoints, though these cost approximately $6.49 per hour for an H100.
Deployment complexity remains a factor when using MoE models on distributed hardware. Coordinating many experts across GPUs adds strain to memory bandwidth. Users must manage routing, load balancing, and communication between GPUs to prevent hotspots. You should compare active parameters, latency, and networking requirements before choosing a provider.
