Anthropic’s Claude Opus 4 carries a list price of $75 per million output tokens. A customer service system handling 10 million interactions monthly with 500 output tokens per interaction generates 5 billion output tokens every month. At the Opus 4 price, that workload costs $375,000 per month, while a commodity-tier model at $5 per million tokens costs $25,000 per month. This $350,000 monthly delta makes model selection an operating cost decision that compounds every quarter.
Architectural waste in reasoning loops
Most enterprise AI budgets spiral because agent architectures are structurally wasteful. ReAct agents follow an observe-think-act cycle that consumes tokens without producing extractable output through reasoning traces. These agents reload the full context, including instructions and prior results, at every single step. This repeated context loading turns architectural waste into a recurring operating expense.
A ReAct agent processing a commercial loan package must extract borrower information, financial data, and compliance certifications from multiple document types. Each extraction starts a new context load and reasoning chain with no shared state between independent steps. When extraction fails, the agent reloads context and adds error information to the prompt to attempt a retry. These repeated retries consume the full context window plus diagnostic tokens.
| Model Tier | Input (per 1M) | Output (per 1M) | Context Window |
|---|---|---|---|
| GPT-5.4 nano | $0.20 | $1.25 | 400K |
| GPT-5.4 mini | $0.75 | $4.50 | 400K |
| GPT-5.4 | $2.50 | $15.00 | 1M |
| Claude Haiku 4.5 | $1.00 | $5.00 | 200K |
| Claude Sonnet 4.6 | $3.00 | $15.00 | 1M |
| Claude Opus 4.6 | $5.00 | $25.00 | 1M |
You should not jump to fine-tuning if you have not iterated a prompt at least 10 times against a 100-case evaluation set. Engineering teams often prefer fine-tuning because it feels like engineering, while prompt engineering feels like fiddling. This bias leads to massive spending on training runs that could have been solved by better context management.
The fine-tuning math and the prompt fallacy
Fine-tuning updates model weights through gradient descent, while prompt engineering modifies the input via system instructions or few-shot examples. A major mistake involves using fine-tuning for facts that change. Facts that change belong in retrieval, not in weights.
Fine-tuning provides value when a task is narrow, stable, and high-volume. A financial services firm processing 8 million classification calls per month on loan documents found that fine-tuning a smaller model for $95,000 yielded $1.53 million in annual token savings. This project achieved 94% accuracy compared to 91% with prompt engineering on a frontier model. However, at 50,000 calls per month, that same $95,000 investment would have wasted money because prompt engineering on a frontier model is cheaper.
The data preparation phase for a fine-tune project consumes 60% of the budget. Curating 1,000 to 10,000 high-quality pairs requires significant domain expert time for cleaning, labeling, and balancing classes. Training runs consume 20%, and evaluation consumes the remaining 20%. If you skip the evaluation step, you deploy a model you cannot trust.
Prompt engineering remains a powerful lever for 70% of behavior problems. Techniques like DSPy and MIPROv2 allow for Bayesian search over instructions, which can lift end-to-end accuracy by 10 to 30 points without touching weights. GEPA uses natural language to provide a higher-bandwidth feedback signal than scalar rewards, outperforming GRPO by 6 percentage points on average.
Scaling errors and the convergence of methods
Companies face extreme difficulty budgeting for tokens because usage scales with complexity. Uber exhausted its 2026 AI budget by April after deploying Claude Code across thousands of engineers. Microsoft also reduced its internal Claude Code licenses after token-based usage costs climbed sharply.
One significant risk involves the brittleness of large prompts. A 4,000-token prompt containing fifteen rules can behave like a Jenga tower where adding a sixteenth rule causes the model to stop obeying the first. This instability drives engineers toward fine-tuning, but modern adapters like LoRA or QLoRA offer a middle path. LoRA trains only 0.1 to 1% of the parameter count, meaning the trained artifact is a small file of tens or hundreds of megabytes rather than a multi-gigabyte model.
The distinction between form and facts remains the primary decision driver. Form refers to behavior like tone, structure, or output schema. Facts refer to knowledge like product details or policies. If your problem concerns form that is still in flux, writing an unstable preference into model weights is just an expensive way to lock in a decision you have not made yet.
Which specific threshold of usage will finally force companies to abandon expensive frontier models for locally hosted, fine-tuned alternatives?
