Why is inference so expensive?

Because you are renting scarce machine time rather than buying a commodity, because the unit of work has grown, and because the part that costs the most is usually quality rather than compute. Those three stack up. First, on supply: Baseten CEO Tuhin Srivastava says the capacity market is far tighter than founders assume, that securing capacity takes multi-month or quarter-long planning, and that you have to be at a sufficient scale to be able to get it at all. Second, the unit of work grew. A single user request in an agent product is no longer one model call: the agent loops, calls tools, retries, and carries context across steps, so spend per task rises even while published per-token prices fall. Third, the expensive part is usually quality rather than compute. Most teams that could cut the bill by dropping to a weaker model decide not to, because the revenue at risk is bigger than the saving. That is a deliberate purchase, not waste, and it is worth separating from the waste in your bill before you go hunting for savings.

Is AI inference getting cheaper?

Token prices keep falling while the cost of a finished task often rises at the same time. The two move apart because an agent spends many model calls on one unit of work where a single prompt and response used to spend one, and because reasoning models spend tokens thinking before they answer. Databricks' internal coding benchmark shows how far apart the two numbers can drift: it found that Anthropic's Sonnet 5, around 1.7 times cheaper per token than Opus 4.8, cost more per completed task, $2.09 against $1.94, because it used more tokens and finished 81 percent of tasks against 87 percent. The practical reading for a founder is that falling token prices are real but they are not a plan. Your bill is set by how much work your product does per request and how efficiently it does it, and both of those are yours to control.

Is AI inference actually profitable?

It can be, and the levers are not the ones founders reach for first. Margin on an AI product is the gap between what you charge per unit of work and what that unit of work costs you to run, so it improves in three places: doing less work per task, running the work on a machine you keep busy, and only paying frontier prices where a customer can tell the difference. The lever that gets reached for first, downgrading the model in the customer-facing path, is usually the worst one, because Baseten CEO Tuhin Srivastava's observation is that buyers will pay a large premium for a modestly better experience. Note that inference economics differ sharply by layer. A provider serving many tenants can amortize a GPU across customers, while a single application company on a dedicated deployment pays for the idle time itself.

AI Inference Cost: What Actually Moves It

Q: How much do 1000 tokens cost?

A thousand tokens costs a fraction of a cent on small models and a few cents on frontier models, and those list prices move often enough that any figure printed here would be stale within months. The more useful answer is that a thousand tokens is the wrong unit to budget in. Databricks put the conclusion plainly in its benchmark report: the token price of a model is a poor indicator of actual costs incurred on end-to-end tasks, and larger models can be far more token efficient and have lower overall costs. Databricks CTO Matei Zaharia's version is shorter: cheaper per-token does not imply cheaper per-task. Price the thing your customer actually buys, which is a resolved ticket, a merged pull request, or a completed filing, and measure your cost against that same unit.

Your per-token prices keep falling and your inference bill keeps rising. That is not a billing error, and it is not only that you are serving more users. Your AI inference cost is not the number on a model's pricing page: the unit of work changed underneath you. One thing a user asks for now costs many model calls, and the meter you have been watching stopped matching the thing you are buying.

This article is about the cost side of inference, not the price side. If you are working out what to charge, read AI pricing. If you are working out why your bill looks the way it does and which levers genuinely move it, start here.

The clearest recent account of how inference is bought and sold comes from Tuhin Srivastava, CEO and co-founder of Baseten, interviewed by Greylock partner Corinne Riley on the firm's Change Agents series. Baseten sells production inference to fast growing AI companies, which makes it a useful vantage point: its customers are the ones whose bills are already large enough to hurt.

Your AI inference cost is not a per-token price

The number on a model's pricing page is a unit price for an input you do not control the quantity of. Your customer does not buy tokens. They buy a resolved ticket, a merged pull request, a completed filing. Between those two units sits everything your product does, and that is where the cost lives.

Databricks put a number on the gap. It built an internal benchmark from real engineering tasks against its own multi-million line codebase and scored models and harnesses on quality and on cost. Its third headline conclusion: "The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks. Larger models can be far more token efficient and have lower overall costs."

The example is the part worth keeping. Anthropic's Sonnet 5 is roughly 1.7 times cheaper per token than Opus 4.8. On Databricks' tasks it was more expensive anyway: $2.09 per task against $1.94, because it consumed more tokens and completed 81 percent of tasks against Opus 4.8's 87 percent. Databricks CTO Matei Zaharia compressed it to a sentence: "Cheaper per-token does not imply cheaper per-task".

The same benchmark cuts the other way too, which is the honest half. An open weight model, GLM 5.2, landed in the top capability tier statistically tied with Opus 4.8 on quality while costing $1.28 per task against Opus's $1.94. So the lesson is not that you should reach for the expensive model. It is that the ranking flips depending on which unit you measure, and only one of those units is the one you pay for.

A second finding from the same report gets less attention and deserves more: the harness a model is called from dramatically affects both cost and quality, and in many cases the simpler harnesses performed best. The scaffolding around the model, how much context it stuffs in, how many times it retries, how many tools it offers, is a cost decision your team makes and can change this week. The model vendor is not the one making it.

Inference is sold three ways, and they are not the same product

Founders talk about "the inference bill" as though it were one line item bought one way. Srivastava describes three distinct products, and the difference between them is mostly a difference in what you control.

Shared endpoints. You call a model that is serving many tenants at once. This is the easy on-ramp, and Srivastava is candid about the trade: on a multi-tenant endpoint "you don't really have a ton of control around your SLAs". You are buying tokens at a list price and inheriting someone else's queue. For a prototype, or for a feature nobody's business depends on, that is the correct choice and the cheapest way to find out whether the thing works at all.

Dedicated deployments. Most of Baseten's customers run this way, according to Srivastava. You get a deployment of your own once you have a point of view on how the model should run: its performance characteristics, its reliability target, where it physically sits. This is the step where your cost structure changes shape, which the next section is about.

Owned models. The third mode is post-training a model on your own data so that it is very good at the specific thing you need, then serving it yourself. Srivastava calls the shift from the first mode to the third "rented to owned intelligence", and his framing of the payoff is precise: customers who own their models "will not only own the quality of the model, but they'll own the performance of the model", along with utilization, cost, and where the thing runs.

Worth noting for anyone in a regulated market: the deployment mode is also your data-residency and isolation posture, so the cost decision and the security decision are the same decision made once. Baseten's multi-cloud architecture exists because of exactly this. Srivastava says enterprises in its early years required everything to be deployed in their own VPC, so the company built for separate VPCs from the start and got good at stitching deployments together across environments. A constraint the compliance team hands you tends to arrive as a cost, and occasionally turns out to be an architecture worth having.

On a dedicated deployment you stop buying tokens and start buying time

This is the shift most teams are unprepared for, and it is the single most useful thing to understand about inference cost.

On a shared endpoint you pay per token, so an idle minute costs you nothing. On a dedicated deployment you are paying for a GPU whether or not you are using it. Your effective cost per token becomes a function of utilization: the same hardware serving twice the traffic halves your unit cost, and the same hardware sitting idle overnight doubles it. Nobody sends you an invoice line called "idle time". You find it by dividing your bill by the work you actually completed.

That means the dedicated step only pays off above a traffic floor, and the floor is higher than most founders guess. Below it you are buying control you cannot yet afford. The test is not "are we serious about AI". It is whether your traffic is steady enough, and large enough, to keep the machine busy.

Two operational realities make the arithmetic harder. The first is that the hardware is unreliable in ways the abstraction hides. Srivastava's line, from building for healthcare customers where downtime is not acceptable, is that "GPUs are flaky", which is why Baseten built primitives for two active deployments running at once rather than one. Redundancy is not free, and if your uptime commitment is real then the redundant capacity is part of your cost of serving, not an infrastructure nicety.

The second is geography. Baseten's explicit strategy, in Srivastava's words, is that "you want to be distributed for a number of reasons, both for reliability, failover" and also so models run closer to the end customer. Latency and resilience both push toward more locations, and more locations means more capacity commitments, each of which needs to be kept busy.

If you are weighing whether to rent compute directly rather than buy inference as a service, our guide to the GPU cluster decision covers what to check before you sign, and inference chips covers why the silicon itself is still built for a workload shape agents abandoned.

Do not save money where the customer can feel it

Here is the finding that should reorder most cost-cutting plans. Describing the fast growing companies Baseten serves, Srivastava puts a number on how lopsided the trade is. Customers will "overpay", in his words, by "5 to 10x". He restates the same point as a "500% premium" paid for what he describes as "a 10% better experience for your customers". Asked a moment later what the most discerning customers optimize for, his answer was not cost at all. It was customer value, and specifically the quality of the model's output.

Take that seriously before you swap a model in the path your users see. Shaving a third off the inference bill is a rounding error against the revenue of a product people prefer. Srivastava's summary of what he sees across a large customer base is blunt: "for the most part, people want the best model for that task." The teams doing this well are not picking one cheap model, they are choosing deliberately per task, which is the discipline we covered in how to choose an LLM.

He does allow a sliding scale, and it points exactly where cost work belongs. For some back-office and batch tasks, customers tell him "We don't need the best model". That is the honest map: the customer-facing path is where you buy quality, and the internal, asynchronous, nobody-is-waiting work is where you cut. Most teams apply the same model policy to both and then feel the savings in the wrong place.

The corollary is that you need to know which of your calls are which. If you cannot currently split your inference spend into customer-facing and internal, that is the first instrumentation job, and it is usually a day of work. Our guide to LLM observability covers the traces to capture.

Capacity is a cost you pay in lead time

Compute scarcity does not show up as a higher price so much as a longer lead time and a higher minimum commitment, and founders consistently underestimate both.

Srivastava is emphatic about the state of the market: whatever you think the available capacity is, "it is 10x less than whatever you think it is". Getting it means "multi-month-long or quarter-long planning", and, he adds, you have to be at a sufficient scale to be able to get capacity at all, while the largest companies in the world are happy to absorb whatever supply exists.

Two consequences for planning. First, capacity is a procurement problem with a lead time, so if a launch six months out needs dedicated capacity, that conversation starts now rather than in month five. The commitment terms themselves, and what to check before signing one, are covered in the GPU cluster decision. Second, a commitment made months ahead is a bet on a demand curve you cannot see yet, which is why Srivastava says capacity planning four to six months out is genuinely hard and why he is willing to say out loud that "none of us really know anything about this market".

One piece of good news sits inside the constraint, and it comes with a caveat from the source himself. Srivastava's read is that hardware demand is converging a bit: "no one wants like decade-old Nvidia hardware", but people still want H100s that are three years old. He adds immediately that customers mostly want the best thing for their business, which a lot of the time is the fastest thing they can run, so this is not him telling you to settle for older silicon. Our own read is that the previous generation is cheaper and easier to get, and sufficient for a great many production workloads, which makes it a question worth putting to your provider rather than a default you should assume either way.

When owning the model starts to pay

Post-training your own model is the most-discussed cost lever and the one with the highest bar to clear, so it is worth being precise about when it earns its place.

Srivastava describes the trigger he sees in customers as two things arriving together: companies reach a scale where costs have risen with the business, and they have also lost control of the thing they are trying to do. Cost alone does not do it. Control alone does not do it. It is the pair, and if only one of them is true for you, this is probably not yet your move.

He credits a combination of things for making it viable now: open source models got genuinely good, there are enough public examples of post-training working to serve as blueprints, and the infrastructure to train and serve a model in the same place finally exists. Baseten bought into that last piece directly, acquiring Parsed, a post-training team, in a deal announced in December 2025, explicitly to close the loop between training and inference.

That loop is the strategic point. Srivastava's view is that the inference stack itself is changing shape as agentic work grows, from a service that runs a model into a set of tools that run agents: "Inference goes from I need to run this model to a set of tools to run these agents." Routing between models, spinning sandboxes up and down to execute code, running tools alongside the model, all of that is now inside the thing you are paying for. When you own the model and the loop, the efficiency gains compound instead of being priced back to you.

If you are earlier than that, the cheaper first step is usually retrieval rather than training, which we compared in RAG vs fine-tuning, and the question of which tasks should stop calling a frontier model at all is covered in where open source LLMs belong.

The cheapest token is the one you never send

Every lever so far has been about buying inference better. The largest one is usually about needing less of it.

The mechanism is context. Most agent systems re-send enormous amounts of the same material on every pass: sprawling system prompts, re-researched facts, documents the model has already read. Speaking to Sequoia about continual learning, the team behind Engram argued that removing that re-reading is not a marginal saving but a step change, and that "it can be 100x fewer tokens" once a model has learned the things it currently has to be told on every call. Treat the specific multiple as one team's estimate rather than a benchmark, but the direction is not controversial, and it is the same instinct behind Databricks' finding that a simpler harness often costs less and performs better.

Routing is the other half. Mike Mignano, a general partner at Union Square Ventures, expects that if capability gains flatten into a linear curve rather than continuing exponentially, buyers will start optimizing hard on spend: leaning on open weight models and on a routing layer that sends each call to "the model that maybe gives you the most bang for your buck", as he put it, "rather than always just token maxing and picking the most powerful model". That world is a forecast rather than today, and Mignano is explicit that his advice for right now runs the other way: as a startup CEO he would "actually still be pounding the table to maximize token spend", though he qualifies it immediately, "on the right things", reaching for a cheaper model "for simple tasks like summarization". Which is the same split this article keeps arriving at. Build the seam that lets you route per task now, because a product that cannot change models per task has no cost lever at all when it needs one.

None of this is separate from how you run the company. Compute is a strategic input, and the teams that treat it as one budget it, instrument it, and review it the way they review payroll. That wider frame is our pillar on AI for startups.

What to do this week

Compute your cost per completed task. Divide last month's inference spend by the number of finished units of work your customer would recognize. That single number is your real meter, and most teams have never calculated it.
Split the bill into customer-facing and internal. Tag calls at the call site. You cannot cut safely until you know which spend a user would notice.
Audit one harness. Take your most expensive agent path and count the tokens it re-sends every pass. Trim the system prompt, cut a retry, drop a tool the model never picks, then re-measure cost per task.
Check your utilization before you go dedicated. If you are already on a dedicated deployment, divide the bill by the work completed and find your idle time. If you are considering the move, work out the traffic floor that makes it cheaper than per-token pricing.
Run a per-task bake-off, not a per-token comparison. Take twenty real tasks, run two or three models and harnesses against them, and score completion rate alongside cost. Expect the ranking to differ from the price list.
Put a date on any capacity you will need in six months. If the answer involves a dedicated commitment, start that conversation now, and price the older hardware generation as the default rather than the exception.
Write down the two conditions for owning a model. Costs rising with scale, and losing control of quality. Revisit quarterly, and do not start post-training until both are true.

Founders who can name their cost per task make better calls on pricing, on hiring, and on what to build next. Working out that number, and the operating habits around it, is part of what we teach in AI Operating System for Startups.

Sources

The Infrastructure Behind AI Agents with Baseten (Greylock Change Agents, May 2026), the conversation this article is built on. Greylock partner Corinne Riley interviews Baseten CEO and co-founder Tuhin Srivastava on rented versus owned intelligence, shared and dedicated deployments, GPU capacity, and how the inference stack changes for agents.
Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase (Databricks), the primary source for cost per task versus token price, the GLM 5.2 and Opus 4.8 comparison, and the finding on harnesses. Corroborated, with the Sonnet 5 figures and Matei Zaharia's quote, by The Register.
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin (Sequoia Capital), for the argument that not re-sending context is a step-change reduction in token consumption rather than a marginal one.
Mike Mignano on 20VC, for the forecast that a flatter capability curve pushes buyers toward routing and open weight models. Profile: Union Square Ventures.
Baseten company background and the Parsed acquisition: Baseten's announcement and the BusinessWire release.