Together AI Launches Polymorphic Inference and US-Only Endpoints

Together AI announced Polymorphic Inference on November 4, introducing Together Compiler as its first implementation. The system uses agents to continuously retune inference for changing models, traffic, and hardware across production workloads. It delivered 1.3x to 3x the tokens per second of open-source baselines in under 24 hours during tests with Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. This approach treats optimization as an ongoing search process tied to a deployment rather than a round of manual tuning frozen at launch.

Agent-Driven Optimization

The compiler takes traffic shape, model, and hardware as inputs, then uses agents to generate and measure candidate optimizations. A persistent engine loads the model once and hot-reloads candidate forward passes without restarting the full engine. Parallel agents communicate through a message board to keep experiments queued while others reason or use tools. Two measurement tiers balance experiment volume against realism: fast engines load a subset of layers for rapid iteration, while slower engines load more layers to test candidates on realistic data distributions. A separate cost model combines hardware specifications and microbenchmarks to prune the parallelism-configuration search before agents spend time optimizing candidates. Winning candidates undergo verification tests for KV-cache contamination and output quality before they can be served. The startup comparison reports 467 seconds to serving for the status-quo sequence versus 9.0 seconds with the compiler’s setup. These figures cover engine startup during optimization rather than individual request latency, which explains why hot-reloading is central to the system’s experiment throughput.

Production observations supply concrete examples of why static configurations fail. Prompts during one country’s working hours reached about 97,000 tokens compared with 70,000 elsewhere, and prompt lengths rose 80% in the two weeks after one model release as usage shifted toward agentic workloads. Customer bursts can double prompt lengths and reduce cache hit rates, changing the conditions under which an existing configuration was tuned. The announcement revises the company’s earlier emphasis on megakernels, concluding that fusion alone is not a universal design goal because mixture-of-experts architectures and dynamic workloads require different techniques. Kernels developed around fixed or random traffic can be miscalibrated for production traffic whose data-dependent routing affects which optimizations win. The paper attributes results partly to 252-fold higher experiment throughput than baseline agentic recursive-self-improvement systems. Beyond language-model inference, the system reports 1.5x to 6.4x speedups on tested weather, biology, time-series, and partial-differential-equation models while maintaining bit-exact outputs.

Data Residency Options

Data residency is now a self-serve, published option on the serverless API. A team that must keep inference inside the United States points its client at a separate US endpoint, keeps its API key and model IDs, and pays a quarter more per token. The option covers seven models: Kimi K3, GLM-5.3, GLM-5.3 Flash, DeepSeek V4 Pro 0813, both DeepSeek V4.1 Flash versions, and MiniMax M3.1 Flash. The premium applies evenly to input, cached input, and output tokens, so the cheapest route stays the cheapest when you switch. If you only need US processing for part of your traffic, send just that part to the US endpoint because the premium is charged only on requests sent there. Requests to the standard endpoint keep standard rates. The US endpoint is https://us-api.together.ai/v1.

End customers can request early access to the compiler through a limited beta. The immediate question is whether the limited beta can reproduce the reported gains on their own models, traffic, and hardware while passing the same quality checks.