A practical playbook for CTOs, CIOs and finance leaders: the cost anatomy of an AI agent, a step-based forecasting model with a worked example, decision frameworks for model routing and scaling, and the guardrails that keep agent spend predictable.
AI agents have changed the shape of the technology bill. A chatbot answers one question with one model call. An agent plans, calls tools, reads documents, checks its own work and sometimes tries again, and every one of those steps consumes tokens. Usage that looked trivial in a pilot can become a material line item once an agent runs thousands of tasks a day.
That is why AI FinOps has moved onto the executive agenda. In the FinOps Foundation’s State of FinOps 2026 survey, 98% of respondents said they now manage AI spend, up from 63% in 2025 and 31% in 2024, and FinOps for AI emerged as practitioners’ top forward-looking priority (FinOps Foundation, State of FinOps 2026). Gartner goes further for one of the fastest-growing agent workloads, predicting that by 2028, AI coding costs will surpass the average developer’s salary as token consumption surges (Gartner, June 24, 2026).
This AI FinOps guide sets out how enterprises can forecast, optimize and control AI agent costs: the cost anatomy of an agent, a step-based forecasting model with a worked example, a model-routing decision framework, the optimization levers that matter, the guardrails that prevent runaway spend and a 90-day plan to put AI FinOps into practice.
AI FinOps is the practice of making AI spend visible, attributable and accountable to business value. It extends cloud FinOps to the cost drivers that are specific to AI: model tokens, inference capacity, vector and retrieval services, agent tool calls, evaluation runs and the people who review AI output.
Classic FinOps asks, “What did this workload cost, and who owns it?” AI FinOps adds two harder questions: “What did each outcome cost?” and “Is that outcome worth more than its cost compared with the alternative?” For agents, that alternative is usually a human workflow, a simpler automation or a smaller model.
That variability is the core AI FinOps problem. Cloud costs scale mostly with provisioned capacity and traffic. Agent costs scale with behavior: how many steps an agent takes, how much context it carries and how often it fails and retries. The same request can cost very different amounts depending on the path the agent chooses.
| Cost driver | Traditional cloud workload | AI agent workload |
|---|---|---|
| Primary unit | Instance hours, storage, requests | Tokens per step × steps per task |
| Variability per request | Low: similar requests cost about the same | High: reasoning path, tool calls and retries vary |
| Growth pattern | Roughly linear with traffic | Can grow faster than traffic as context and steps accumulate |
| Pricing model | Mature commitments and reservations | Consumption-based list prices that change with each model generation |
| Allocation | Tags on resources | Requires per-request metadata (agent, team, customer, use case) |
| Hidden costs | Data transfer, idle capacity | Tool and search fees, evaluation runs, human escalation |
Practitioners report the allocation problem directly: the State of FinOps 2026 findings describe allocating AI costs to business units as harder than allocating traditional infrastructure, and list granular monitoring of tokens, LLM requests and GPU use as the most requested tool feature (FinOps Foundation).
NineHertz perspective: In agent programs we review, cost surprises rarely come from list prices. They come from design choices nobody priced: a planning loop without a step limit, a full document re-sent on every step, or a frontier model doing classification work a small model handles well. Price the architecture, not just the model.
Every AI FinOps program starts by breaking an agent’s cost into layers. This is the cost stack our AI engineering team uses when we instrument agents in production.
| Layer | What drives it | How it is billed | Who usually owns it |
|---|---|---|---|
| 1. Model inference | Input and output tokens per step, model tier, number of steps | Per million tokens, by model | Engineering |
| 2. Context and retrieval | Documents retrieved, embeddings, vector database queries, cached context | Per token, per query or per GB stored | Platform / data |
| 3. Tools and actions | Web search, third-party APIs, code execution, internal system calls | Per call or per 1,000 calls | Engineering / integration owner |
| 4. Orchestration and runtime | Agent framework hosting, queues, logging, observability | Compute hours, log volume | Platform |
| 5. Quality and safety | Evaluation suites, guardrail models, red-team runs | Tokens and compute for test traffic | AI governance / QA |
| 6. Human in the loop | Escalations, approvals, rework of failed tasks | People time | Business operations |
Tool fees are easy to miss. Anthropic, for example, lists web search on the Claude API at $10 per 1,000 searches plus standard token costs, and notes that tool definitions add system-prompt tokens to every request that uses tools (Claude API pricing).
Token costs tell you what you spent. They do not tell you whether the agent was worth it. In our AI FinOps work, we recommend every agent report one primary unit metric:
Cost per Successful Task (CPST) = (Model + Retrieval + Tools + Runtime + Quality + Human review costs) ÷ Number of tasks completed successfully
Dividing by successful tasks, not all tasks, matters. An agent that is cheap per run but fails 30% of the time, and pushes those failures to people, can cost more per outcome than a slower, more expensive design that succeeds reliably. The FinOps Foundation’s guidance on FinOps for AI also recommends unit metrics such as cost per inference and cost per token, alongside ROI and time to business value (FinOps Foundation, FinOps for AI). CPST connects those technical units to the business result.
Forecasting is the first AI FinOps capability to build. Forecasting agents by “users × price per user” fails because cost lives in the steps. Use a step-based model instead.
Monthly model cost = Tasks per month × Model calls per task × Retry factor × [(Input tokens per call × Input price) + (Output tokens per call × Output price)]
Then add the other layers of the cost stack and divide by successful tasks to get CPST.
To show how much architecture moves the number, and why AI FinOps teams should compare designs before launch, here is an illustrative support-operations agent. The assumptions are ours and chosen for round numbers; the prices are Anthropic’s published Claude API list prices on October 28, 2026 (Claude API pricing).
Assumptions (illustrative): 50,000 tasks per month; 6 model calls per task; a 1.15 retry factor; 4,000 input tokens and 500 output tokens per call on average. That is 345,000 calls, about 1,380 million input tokens and 172.5 million output tokens a month.
Published list prices used (per million tokens): Claude Opus 5.5 at $4 input and $20 output; Claude Sonnet 5.5 at $2 input and $10 output; Claude Haiku 5.5 at $0.10 input and $0.50 output for prompts up to 100,000 tokens. Cache reads are billed at 0.05x the base input price on Opus 5.5 and Sonnet 5.5 and 0.1x on other models, and the Batch API carries a 50% discount on input and output tokens.
| Scenario | Architecture | Model cost per month | Model cost per task | Change vs. Scenario A |
|---|---|---|---|---|
| A | Every step on the frontier model (Opus 5.5) | $8,970 | $0.179 | Baseline |
| B | Every step on the mid-tier model (Sonnet 5.5) | $4,485 | $0.090 | About 50% lower |
| C | Routed: 70% of steps on Haiku 5.5, 30% on Sonnet 5.5 | $1,502 | $0.030 | About 83% lower |
| D | Routed, with 60% of input tokens served from cache | $978 | $0.020 | About 89% lower |
| E | Scenario D, with 30% of tasks moved to the Batch API | $832 | $0.017 | About 91% lower |
Model token costs only. Scenario D ignores cache-write premiums for simplicity; Scenario E assumes those tasks can wait for asynchronous results. Quality must be validated at each step: routing only works where the smaller model meets the evaluation threshold.
Now add the layer most forecasts leave out. Suppose 5% of tasks escalate to a person for 4 minutes each, at a loaded cost of $60 an hour (again, illustrative). That is 2,500 escalations, about 167 hours and roughly $10,000 a month, which is more than ten times the optimized model cost in Scenario D.
NineHertz perspective: Once routing and caching are in place, the escalation rate usually becomes the biggest cost lever an agent has. A two-point improvement in task success can be worth more than switching models. That is why we put evaluation and failure analysis inside the FinOps loop, not beside it.
Instrument cost per successful task, compare architectures before launch, and cut model spend without hurting quality.
Gartner’s guidance on AI coding costs makes the same point for developer tooling: route simple, frequent tasks to smaller models and reserve frontier models for complex, high-value work (Gartner, June 24, 2026). The matrix below is the AI FinOps routing framework we use to decide the default tier for each agent step, before evaluation confirms or overrides it.
| Step complexity ↓ / Cost of a wrong answer → | Low (easily reversed, internal) | Medium (customer-visible, correctable) | High (financial, legal, safety or irreversible actions) |
|---|---|---|---|
| Simple (classify, extract, route, format) | Small model; no review | Small model with validation rules | Small model plus deterministic checks and human approval |
| Moderate (summarize, draft, multi-source answer) | Small or mid-tier model | Mid-tier model with automated evaluation | Mid-tier model plus human approval |
| Complex (multi-step planning, ambiguous reasoning, code changes) | Mid-tier model; escalate on low confidence | Frontier model for planning, smaller models for execution steps | Frontier model with human approval and full audit trail |
How to use it: assign each step a cell, start with the cheapest tier in that cell, and move up a tier only when the evaluation set shows the cheaper tier misses the quality threshold. Record the decision so it can be revisited when new models launch.
Not every AI FinOps lever suits every agent. The table shows what each one does, the documented effect where providers publish one, and the effort and quality risk we typically see.
| # | Lever | What it does | Documented effect | Effort | Quality risk |
|---|---|---|---|---|---|
| 1 | Model routing | Sends each step to the smallest model that passes evaluation | Price gaps between tiers can exceed 10x (see the worked example) | Medium | Medium: requires evaluation per step |
| 2 | Prompt caching | Reuses repeated system prompts, tools and documents | Claude cache reads bill at 0.1x base input price, or 0.05x on Opus 5.5 and Sonnet 5.5 (Anthropic) | Low | Low |
| 3 | Batch processing | Runs non-urgent work asynchronously | 50% discount on Anthropic’s Batch API and OpenAI’s Batch API, with results within 24 hours on OpenAI (Anthropic, OpenAI) | Low | Low |
| 4 | Context trimming | Sends only the relevant documents and history to each step | Varies by agent; measure tokens per step | Medium | Medium: too little context hurts accuracy |
| 5 | Conversation summarization | Replaces long histories with running summaries | Varies; measure | Medium | Medium |
| 6 | Step and loop limits | Caps planning iterations and tool calls per task | Prevents worst-case runs rather than reducing the average | Low | Low |
| 7 | Output length control | Sets maximum output tokens and structured output formats | Varies; output tokens are priced 5x input on the models above | Low | Low |
| 8 | Deterministic pre-processing | Uses code, rules or search before calling a model | Removes model calls entirely for some steps | Medium | Low |
| 9 | Response reuse | Serves known answers to repeated questions | Varies with repeat rate | Medium | Medium: stale answers |
| 10 | Tool call hygiene | Avoids redundant searches and failing API calls that trigger retries | Reduces retry factor and tool fees | Medium | Low |
| 11 | Commitments and platform choice | Uses committed spend or cloud marketplace agreements for predictable volumes | Depends on negotiated terms | Low | None |
| 12 | Retire low-value agents | Stops agents whose CPST exceeds the value they create | Removes the full cost | Low | None |
NineHertz perspective: Sequence matters. Apply levers 6, 7 and 2 first: they are cheap, low-risk and make every later measurement more accurate. Routing (lever 1) delivers the largest savings but only once an evaluation set exists, so build the evaluation set during the first month.
In AI FinOps, optimization lowers the average. Guardrails protect against the tail: the agent that loops overnight, the prompt that suddenly carries a 300-page document, the integration that fails and triggers endless retries. Build them into the platform, not into a monthly report.
| Guardrail | Example setting | Action when breached |
|---|---|---|
| Per-run budget | Maximum tokens or dollars per task | Stop the run, return a safe fallback, log for review |
| Step limit | Maximum planning iterations and tool calls per task | Stop and escalate to a person |
| Per-agent daily budget | Daily spend ceiling per agent and environment | Throttle new runs, alert the owner |
| Context size cap | Maximum input tokens per call | Trim or summarize before sending |
| Anomaly detection | Spend or tokens per task above a rolling baseline | Alert engineering and FinOps owners |
| Kill switch | Feature flag per agent | Disable the agent without a deployment |
| Environment separation | Separate keys and budgets for development, test and production | Prevent test traffic from consuming production budget |
| Attribution metadata | Agent, team, use case and customer on every request | Enables showback and chargeback |
Gartner recommends setting token thresholds, escalation policies and automated monitoring within engineering workflows, noting that token discipline will not emerge through developer choice alone (Gartner, June 24, 2026). The same applies to every agent in production.
Cost control is only half of AI FinOps. The other half is deciding whether an agent should exist at all, and whether it should scale. Gartner predicts that by 2030, the cost per resolution for generative AI in customer service will exceed $3, more than many B2C offshore human agents, as vendors move toward profitability and use cases consume more tokens (Gartner, January 26, 2026). In other words, “AI is cheaper than people” cannot be assumed; it has to be measured.
We use a simple ratio and a decision grid:
Agent Value Ratio = Value per successful task ÷ Cost per successful task
Value per task is the cost of the current way of doing the work, plus any revenue, speed or quality gain the business agrees to count.
| CPST falling month over month | CPST flat or rising | |
|---|---|---|
| Value Ratio above 3 | Scale: expand volume and use cases | Optimize, then scale: apply levers before growth multiplies the cost |
| Value Ratio 1 to 3 | Optimize: focus on success rate and routing | Redesign: revisit the workflow, model tiers and human hand-offs |
| Value Ratio below 1 | Fix or pause: set a deadline for improvement | Retire: stop the agent and reallocate budget |
The thresholds are our starting defaults; finance and product owners should set their own based on risk appetite and strategic value.
AI spend crosses team boundaries, so AI FinOps ownership must be explicit. This is the responsibility split we recommend.
| Activity | Engineering | Platform / MLOps | Finance / FinOps | Product owner | AI governance |
|---|---|---|---|---|---|
| Instrument tokens, tools and outcomes per request | Responsible | Accountable | Informed | Informed | Consulted |
| Forecast and budget per agent | Consulted | Consulted | Accountable | Responsible | Informed |
| Model routing and optimization | Responsible | Accountable | Informed | Consulted | Consulted |
| Guardrails and kill switches | Responsible | Accountable | Informed | Consulted | Consulted |
| Value measurement and scale decisions | Consulted | Informed | Consulted | Accountable | Consulted |
| Quality and risk thresholds | Consulted | Consulted | Informed | Consulted | Accountable |
The FinOps Foundation describes managing AI costs in Crawl, Walk and Run phases, from pilots with preset cost and time limits to core processes with advanced anomaly tracking and ROI metrics (FinOps Foundation, FinOps for AI). Our 90-day plan puts AI FinOps into practice for an organization with enterprise AI agents in, or close to, production.
Score each dimension of your AI FinOps practice to see where to focus next.
| Dimension | Crawl | Walk | Run |
|---|---|---|---|
| Visibility | Monthly invoice per provider | Spend per agent and team | Real-time spend per step, customer and use case |
| Unit metrics | Cost per token | Cost per task | Cost per successful task and Value Ratio |
| Forecasting | Annual estimate | Monthly step-based forecast per agent | Scenario forecasts tied to business volume drivers |
| Optimization | Ad hoc prompt fixes | Caching, limits and batch in place | Evaluation-driven routing with continuous tuning |
| Control | Provider-level spend alerts | Per-agent budgets and step limits | Automated anomaly response and kill switches |
| Governance | No formal owner | Named owners per agent | Shared operating model across engineering, finance, product and governance |
At The NineHertz, AI FinOps is designed in, not audited afterwards. Our AI Delivery Readiness Diagnostic is a fixed-fee, two-week assessment that sizes use cases, data readiness, risk and the expected unit economics of each agent before a full build. In delivery, we instrument every agent for cost per successful task, build the evaluation sets that make model routing safe, and implement budgets, guardrails and showback as part of the platform behind production AI.
AI FinOps controls are part of the ContinuumAI™ governance framework, alongside quality, security and AI risk, so finance and risk teams see the same evidence as engineering. As a member of Anthropic’s Claude Partner Network, we also help teams choose the right Claude model tier and platform for each workload. For build costs rather than running costs, read what it costs to build an agentic AI system.
Get a clear view of cost per successful task for your priority use cases.
AI FinOps is the practice of making AI spend visible, attributable and accountable to business value. It extends cloud FinOps to AI-specific cost drivers such as model tokens, inference capacity, retrieval, agent tool calls, evaluation runs and human review, and measures cost per outcome rather than cost per resource.
Use a step-based model: tasks per month multiplied by model calls per task, a retry factor and the tokens and price for each step’s model tier. Add tool, retrieval, runtime and human review costs, build expected, growth and peak scenarios, and re-forecast monthly against actuals. This step-based approach is the foundation of any AI FinOps forecast.
Cost per successful task: the total cost of running the agent, including model, retrieval, tools, runtime, quality checks and human review, divided by the number of tasks it completed successfully. It links technical spend to the business outcome and exposes the cost of failures and escalations.
Start with low-risk levers such as step limits, output caps and prompt caching, then build an evaluation set and route each agent step to the smallest model that passes it. Move non-urgent work to batch processing, trim context and fix tool errors that cause retries.
Put guardrails in code: per-run token or dollar budgets, step and tool-call limits, per-agent daily budgets, context size caps, anomaly alerts against a rolling baseline, separate keys per environment and a kill switch that can disable an agent without a deployment.
AI FinOps ownership is shared. Engineering instruments and optimizes agents, the platform team owns guardrails, finance owns forecasts and budgets, product owners are accountable for value and scale decisions, and AI governance sets quality and risk thresholds.
No. Gartner predicts that by 2030 the generative AI cost per resolution in customer service will exceed $3, more than many B2C offshore human agents. Compare the cost per successful task with the full cost of the current process before scaling an agent.
Key Takeaways Multi-agent systems can improve task specialization, coordination, and scalability by distributing complex workflows to specialized AI agents. Sequential,…
Key Takeaways According to statistics, multi-agent systems have a 41% to 86.7% failure rate. Some of the biggest reasons behind…
Imagine a million-dollar deal being finalized without any human intervention! It sounds hypothetical, but it’s quickly becoming a reality. Agentic…
