Deploying and orchestrating with Fireworks AI

Fireworks AI enables efficient model deployment through prompt caching that reduces time to first token by up to 80%. Users can manage compound AI systems using Nexus orchestration and optimize costs with LoRA fine-tuning or reserved pricing on…

Deploying and orchestrating with Fireworks AI

Transition from serverless to production

Create an API key and send a chat request to a serverless model to test prompts and answer quality. Prompt caching works by default. This feature cuts time to first token by as much as 80% as chat history grows. You must pin the session by setting the user field, the x-session-affinity header, or the prompt_cache_key. By setting the user field, the x-session-affinity header, or the prompt_cache_key, you ensure that the Fireworks AI platform sends that session’s specific requests to the same replica to maintain the benefits of prompt caching. Without this affinity, turns on a busy deployment miss the cache. Once you move past prototyping, deploy on the Fast shape with –min-replica-count 1 to skip cold starts. A breach at a third-party server rental company previously exposed billing records, datasets, and incomplete model weights. This incident led Notion to stop using the platform for its open-weight models. You can also use reserved pricing to reduce costs by 30-50% once your chat volume grows. The platform achieves 94-98% GPU utilization on H100 GPUs.

Orchestrating workloads with Nexus and LoRA

Manage compound AI systems with Nexus to decide which model answers each request and hold a budget. You can send intent classification to a small model like Qwen2.5 7B and pass hard reasoning turns to a larger one. This orchestration allows for effective task distribution across different model sizes and speeds. For specialized performance, fine-tune smaller models with LoRA. This method uses a small set of extra weights on top of a frozen base model. It costs up to 100x less than GPT-4 and allows Cresta to serve many adapters on one base model. You can also use Direct preference optimization (DPO) to teach a model to lean toward preferred answers. Reinforcement fine-tuning (RFT) uses a grader to push the model toward higher-scoring answers, which often costs more due to grader tokens. Supervised fine-tuning (SFT) remains the default method, where you show the model example prompts with the answers you want. Distillation allows a large teacher model to write answers for a smaller student model. You must pay for the teacher’s calls and the training. How will model orchestration change as latency requirements for agentic workflows tighten?

Optimizing performance and costs

Use the Responses API to keep state on the server. This API continues a chat from a previous_response_id, so you skip resending the full history. You can also set store=false on Responses API calls to opt out of server-side storage. For tool calling, define tools in JSON Schema and set tool_choice to auto, none, required, or a specific function. Setting stream to true allows tokens to reach your UI as server-sent events while the model generates. Dedicated deployments ship with default speculative decoding drafters for most models. For high-performance workloads, Llama 4 Scout (17B) on H100 80GB hardware achieves 20,622 tokens per second prefill throughput and 24ms time to first token. You should measure TTFT on your own traffic before you tune.

Model Parameter Size SFT Price (per 1M tokens) LoRA DPO Price (per 1M tokens)
Up to 16B $0.50 $1.00 – $2.00
16.1B – 80B $3.00 $6.00 – $12.00
80B – 300B $6.00 $12.00 – $24.00
Over 300B $10.00 $20.00 – $40.00
airtrain.ai
airtrain.ai

The airtrain.ai newsroom covers AI research, models and the tools built on them.

More on this topic

Stay ahead of AI

Get the week's most important AI stories delivered to your inbox every Monday.

No spam. Unsubscribe anytime.

More Stories