Inference costs for private equity portfolios

When a portfolio company ships an AI feature, the marginal cost of serving each request is not zero. Tokens, embedding lookups, guardrail models, logging, and GPU or reserved capacity all show up somewhere — either as COGS for a productized feature or as opex for internal tools. Private equity owners who underwrote margin expansion from “AI efficiency” without modeling that marginal cost are discovering the gap in monthly closes.

The spread between frontier and efficient models makes this urgent. A support assistant that is delightful on an expensive frontier model may be economically unacceptable at scale, while an open-weight or hosted efficient model may clear the same eval bar for a fraction of the unit price. That is not a religious debate; it is a routing and measurement problem. Start with the pricing index and stress scenarios in the inference cost calculator.

Build the model bottom-up. List workloads: customer chat, document extraction, agentic multi-step tools, batch reprocessing, coding assistants, and anything else burning tokens. For each, estimate monthly request volume, average input and output tokens, cache hit rates if relevant, and the fraction of traffic that needs the highest-quality model. Apply list or negotiated rates. Add overhead for eval runs, moderation, and failed retries — teams underestimate retries.

Separate product inference (touches customers, belongs in unit economics) from internal inference (employee tools, belongs in functional budgets). The governance and approval paths differ. A CTO can often approve an internal pilot; a COGS line that moves gross margin needs CFO and board visibility.

Refresh assumptions when vendors change prices or when product usage steps up after a launch. An as-of date on every chart is non-negotiable editorial practice here for the same reason it is on the Deal Wire: stale unit costs produce false confidence.

Cost cutting without quality floors is how companies train employees and customers to ignore AI features. Define eval sets per workload — accuracy, groundedness, toxicity, latency, and task success. A cheaper model may serve traffic only if it clears the floor. Borderline traffic can escalate to a frontier model. This ladder is the heart of open-weight vs frontier decisioning.

Routing also encodes risk. A regulated advice-like answer may require human review regardless of model. An internal summarization of a public webpage can be aggressive on cost. Write the policy down; do not leave it in an engineer’s head.

Meter at the grain you manage: use case, application, team, and legal entity. Tag tokens with these dimensions at the gateway. Weekly FinOps reviews should show top movers, anomaly spikes, and whether growth is “good usage” (adoption) or “bad usage” (loops, unbounded agents, prompt bloat). Unbounded agents deserve special fear — a poorly bounded tool loop can spend a month’s budget in a night.

Portfolio AI operations platforms — shared gateways, budgets, and kill switches — amortize this instrumentation across companies. See portfolio AI operations. Firm-level participation in lab vehicles may help commercial terms; it does not replace metering. Watch peers on the league table and Deal Wire.

Beyond model choice, reduce tokens: tighter prompts, retrieval that sends only relevant chunks, caching repeated contexts, batching where latency allows, and smaller models for classification stages. Architecturally, prefer pipelines that extract structure once and reuse it rather than re-sending entire PDFs on every question.

Commercially, negotiate committed-use discounts only after you understand organic demand. Buying a huge commitment to a frontier stack before routing discipline exists is how you lock in the wrong cost curve. Include data-handling terms, uptime, and rate-limit clarity in the same negotiation as price.

Boards need a short story: what we spend, why it is growing, whether quality floors hold, and what decisions are needed. Avoid drowning directors in token trivia. Tie spend to value — deflected tickets, hours saved, conversion lift — or honestly admit a strategic investment period with a sunset review date.

In new deals, underwrite inference like any other variable cost. Diligence should capture current spend, vendor concentration, and roadmap features that would multiply tokens. See AI due diligence and put baselines into the 100-day plan. For the parent playbook, return to AI in private equity.

Translate tokens into the language of the P&L. For a customer-facing feature, compute cost per session, cost per successful task, and cost as a percent of revenue for that feature. For internal tools, compute cost per employee per month and cost per ticket deflected. Put those next to labor benchmarks. If the AI path is more expensive than the human path at current volumes, either improve routing, raise prices, or stop calling it efficiency.

Build a simple sensitivity: volume ±30%, output tokens ±20%, frontier mix ±15 points. Many “surprises” are just sensitivities nobody ran. Require every AI investment case in the portfolio to include this table before capital committee — the same way you would require a headcount plan for a shared service center.

Distinguish experiments from production. Sandbox spend should have a monthly cap and an expiration date. Production spend should have an owner and an alert threshold. Mixing the two in one cloud account is how experiments bankrupt a cost center without anyone noticing until close.

Agents that loop tools can multiply tokens nonlinearly. A single user request may trigger dozens of model calls, each with growing context. Set maximum steps, maximum dollars per run, and human approval for tool classes that mutate state. Log step counts as a first-class metric. If median steps creep upward after a prompt change, treat it like a performance regression.

Retrieval quality is a cost lever. Bad retrieval sends huge irrelevant chunks, inflating input tokens and lowering answer quality — you pay more for worse outcomes. Invest in chunking, metadata filters, and freshness. The cheapest token is the one you never send.

Coding assistants deserve a separate policy. They can be enormous productivity wins and enormous silent costs if every repository is indexed carelessly or if idle tabs stream context. Meter by team, publish normals, and coach outliers. Engineers will optimize what you show them.

Operating partners love benchmarks. Use them carefully. Cost per ticket in a regulated insurer should not be compared naively to a consumer marketplace. Normalize by workload class and by whether the feature is customer-critical. Publish ranges, not fake medians with n=2.

Where you do benchmark, share playbooks from top-quartile portcos: how they structured gateways, which evals caught regressions, how they negotiated. The league table and Deal Wire show capital formation; your internal FinOps council shows operating reality. Both matter.

Revisit contracts when usage steps through tiers. A pilot rate that looked fine at 20m tokens/month can be wrong at 200m. Build calendar reminders around expected launch ramps so procurement is not surprised by success.

Across funds, the teams that make durable progress share a few habits. They write decisions down with dates. They refuse to expand scope before metering exists. They pair every automation claim with a quality floor and a named executive owner. They bring CFOs into model-routing debates early, before unit costs become a surprise in the monthly pack. And they treat vendor press releases as inputs to diligence, not as substitutes for operating proof.

The teams that struggle also rhyme. They launch too many pilots. They staff AI as a side project for already overloaded engineering managers. They buy enterprise agreements to “get started” without workload maps. They hide failures instead of killing them. In a five-year hold, those habits compound into wasted calendar time — the scarcest resource in a portfolio company fighting day-to-day fires.

On InferenceCosts, use the rest of this site as a toolkit, not as dogma. The Deal Wire tells you where capital is forming. The league table shows who is participating. The pricing index and calculator quantify unit economics. The spoke guides dig into sourcing, diligence, costs, value creation, ops, governance, model choice, and the first hundred days. Your job is to assemble the pieces into a plan your board can govern and your operators can run on a Monday morning.

Finally, remember the asset-class basics still bind. Returns still come from buying well, improving companies, and selling better. IRR and MOIC still disagree usefully. Leverage still amplifies both directions. AI changes the operating toolkit and the cost stack inside that timeless loop. If you keep that proportion straight, you will ask better questions than peers who think a model alone is a strategy.