Most enterprise AI projects cost far more than the forecast, and the overrun is usually architectural rather than a model-pricing problem: ReAct-style agents retry failed actions, reload context at every step, and burn tokens on redundant reasoning loops. This post breaks down where the spend actually goes and what compiled execution changes.
AI budgets are spiraling. Models are not the only reason. Most agent architectures are structurally wasteful. Every reasoning loop, every context reload, and every retry burns tokens that produce no business value. The enterprises that control AI costs will eliminate architectural waste from their execution pipelines.
The AI Cost Crisis
Gartner forecasts continued growth in worldwide AI spending. A Gartner update identifies agentic AI workloads as a primary demand driver. That spending includes training, inference, infrastructure, and integration costs that often arrive after the initial business case. A Forbes analysis found that enterprise agent bills can keep climbing even when model prices fall because token consumption per completed task grows with workflow complexity. A Gartner release also highlights the role of allocation and usage efficiency in controlling AI spend.
Integration and maintenance can account for a substantial share of total AI deployment costs. The model API bill is the visible expense. The invisible expenses are the engineering hours spent debugging stochastic failures, rebuilding prompts when outputs drift, and maintaining fragile tool-calling chains that break when upstream APIs change. These costs compound over time. The model bill stays visible while everything around it grows.
Aaron Levie, CEO of Box, noted on LinkedIn that enterprise leaders are asking how to budget for tokens across use cases and teams. This is a C-level conversation about operational expenditure that scales with usage, has no natural ceiling, and resists traditional forecasting methods.
Gartner projects that a growing share of enterprise software applications will include agentic AI. Every deployment faces the same question: how do we control inference costs at scale? The platforms that answer that question architecturally, not just commercially, will win.
Where the Tokens Go
To understand why AI agent costs blow budgets, you need to trace where tokens are actually consumed. Consider a ReAct agent processing a commercial loan package. The agent needs to extract borrower information, financial data, collateral descriptions, covenant terms, and compliance certifications from multiple document types within the package.
- Context loading: At each step, the ReAct agent reloads the full context. The document content, the instructions, prior extraction results, and conversation history all go back into the prompt. A long document is loaded repeatedly, so much of the context is redundant.
- Reasoning loops: The ReAct pattern follows an observe-think-act cycle. Each cycle produces a reasoning trace explaining what the agent sees, what it plans to do, and why. These traces can help with debugging, but they consume tokens without producing extractable output.
- Retries: When extraction fails or the output does not match the expected format, the agent reloads context, adds error information to the prompt, and tries another approach. Repeated retries consume the full context window plus diagnostic tokens. A McKinsey report describes iterative checking, correcting, and rerunning as a broad source of agentic spend.
- Sequential processing: ReAct agents process documents one section at a time. Each extraction starts another context load and reasoning chain, with no shared state between independent steps.
The total for a document-heavy workflow can become large enough to affect the business case. At scale, repeated context loading, reasoning traces, retries, and sequential tool calls turn architectural waste into a recurring operating expense.
How Compiled Execution Eliminates Waste
Compiled execution takes a fundamentally different approach. Instead of giving an agent a goal and letting it figure out the steps at inference time, a policy engine compiles plain English rules into an execution plan before any document is processed. The plan specifies exactly which fields to extract, from which document types, using which methods, and in what order.
Context loads once. The compiled plan ingests the document content once. Extracted data is stored in a shared context that all subsequent steps can reference. There is no repeated context loading or redundant token consumption.
Deterministic operations compile to code. Calculations, comparisons, format validations, and cross-field consistency checks do not require an LLM. A compiled plan identifies which steps require genuine language understanding and which can be executed as deterministic code without inference-token cost.
Structured LLM calls replace reasoning chains. When the plan invokes an LLM, it uses constrained outputs against a pre-built schema. The model receives a specific extraction target with a typed output schema. There is no need for a free-form observe-think-act cycle.
No retries. The plan is deterministic. If an extraction fails, it fails for a structural reason, such as a missing or unreadable field, rather than because the agent guessed wrong about which tool to call. Structural failures route to human review instead of burning tokens on retry loops that will not succeed.
The Parallelism Dividend
ReAct agents process sequentially by design. Each step depends on the reasoning output of the previous step, so the agent cannot start an independent extraction until the current reasoning chain finishes. This sequential dependency is architectural, not incidental.
Compiled execution plans identify independent steps at compile time. Borrower extraction and collateral extraction can run independently. Financial data extraction and covenant identification can run independently. The compiled plan runs independent steps in parallel, reducing wall-clock time and the compute needed to handle a queue.
Speed and cost improve together. You do not choose between fast and cheap. Compiled execution delivers both by reducing sequential work and making the execution path predictable.
For high-volume deployments, the parallelism dividend compounds. A lender using parallel execution needs less compute capacity to handle the same throughput because infrastructure scales with the workflow's parallelizable surface area rather than with every sequential step.
Token Budgeting for the Enterprise
Levie's observation deserves a closer look: token costs probably do not fit neatly inside a traditional IT budget. Traditional IT budgets assume predictable, fixed costs. AI inference costs scale with usage in ways that resist forecasting. A workflow change can increase token consumption quickly, and a new document type can introduce retry patterns across an entire pipeline. Uber's experience and Tesla's response both illustrate why organizations are adding more explicit controls around AI tool spending. A McKinsey survey likewise frames budget discipline as an operating-model problem.
The answer is to treat AI compute like cloud compute. Allocate by workflow. Measure per transaction. Optimize the highest-volume paths first. Use the AI agent ROI calculator to model cost per completed workflow. This requires a platform that makes per-transaction costs visible and predictable. ReAct agents cannot provide this because their token consumption varies with document complexity, retry rates, and reasoning depth. You can measure average costs, but you cannot predict them for a specific transaction.
Compiled execution makes per-transaction costs more predictable. The execution plan is fixed. The same document type, processed by the same compiled plan, follows the same execution path every time, with only minor variation in model output length. Finance teams can forecast AI costs the same way they forecast cloud compute: by multiplying transaction volume by per-transaction cost. No surprise spikes from stochastic retry storms.
The practical implication for budget planning is to start with the highest-volume workflow. Measure its current per-transaction token cost. Identify the waste, calculate the cost under a compiled execution model, and use the difference to prioritize the migration.
Token budgeting checklist
- Allocate AI compute by workflow.
- Measure cost per completed transaction.
- Start with the highest-volume path.
- Identify context reloads, reasoning traces, and retries.
- Calculate the compiled-execution baseline.
- Re-measure production cost as workflows change.
The Math: ReAct vs. Compiled for a Real Workflow
Let's make this concrete with a construction draw review. The workflow processes a multi-document draw package and applies policy rules covering budget compliance, lien waiver completeness, insurance coverage, retainage calculations, and change order authorization.
| Cost component | ReAct agent | Compiled execution |
|---|---|---|
| Context loading | Reloads context through runtime reasoning loops, which drives repeated input usage. | Loads context once and shares extracted data across steps. |
| Reasoning and extraction | Uses free-form reasoning traces, sequential tool calls, and retries. | Uses structured LLM calls and deterministic checks. |
| Measured evaluation reference | $4.64 per evaluation cycle and 3.6x input replay for a growing-context agentic run. | $1.34 per evaluation cycle for a single structured pass. |
The linked measured study provides a reference point for the architectural difference. The important budgeting distinction is that compiled execution makes the workflow and its token path explicit before runtime.
Beyond Tokens: Total Cost of Ownership
Token cost is the most visible expense. It is not the largest. The total cost of ownership for an AI agent deployment includes workflow maintenance, debugging, and compliance costs that compiled execution reduces simultaneously.
Workflow maintenance. ReAct agents are governed by prompts. When a business rule changes, someone must update the prompt, test it across edge cases, verify that the change did not break adjacent behaviors, and deploy it. This is engineering work. For a policy-driven platform, a rule change is a policy edit in plain English. The platform recompiles the execution plan automatically. The compliance officer who understands the regulation can make the change directly. No engineering ticket or extended sprint cycle.
Debugging costs. When a ReAct agent produces an incorrect output, debugging requires reconstructing the reasoning chain to find where it went wrong. Was it a hallucinated tool call, a retry that corrupted state, or a reasoning trace that drifted off-task? These failures are stochastic. The same input might produce the correct output on a rerun, making the bug unreproducible. Compiled execution produces structural failures that can be traced to the policy rule that governs them.
Compliance costs. In regulated industries, every automated decision must be explainable. ReAct agents produce reasoning traces that read like stream-of-consciousness notes: "I think the coverage amount is material based on what I see in the document, but let me check again..." This is not auditable evidence. Compiled execution produces structured why-trails: policy version applied, data extracted with source pointers, and each condition evaluated with pass or fail results. Auditors and examiners can review decisions without interpreting an AI's thought process.
When maintenance, debugging, and compliance costs are added to token costs, the total cost advantage of compiled execution over traditional workflow approaches extends beyond token savings across a multi-year deployment horizon. The operational savings justify the platform.
MightyBot compiles plain English policies into hybrid execution plans that eliminate architectural waste. Learn how compiled execution can reduce your AI agent costs.
