What is AI agent infrastructure?

AI agent infrastructure is everything that sits between a model call and a finished task. When a product just sent a prompt and rendered a response, there was almost nothing in that gap. An agent loops, so the gap fills up: a loop that decides what to do next, a layer that actually executes tools, an environment to run code in, somewhere to keep the state of a run that outlives a request, a seam that lets you change models, a place that enforces spending limits, and telemetry that reconstructs what happened. Baseten CEO Tuhin Srivastava, talking to Greylock, described the shift as models needing an ecosystem of tools "for running tools, for running other models, for routing between different models". Most teams end up owning all seven pieces without ever deciding to.

What is an agent harness and how do you know if yours is good?

A harness is the loop around the model: it assembles context, decides which tool to call, feeds the result back, and decides when the task is done. The useful thing is that it has measurable properties rather than a vibe. Factory co-founder and CEO Matan Grinberg names several on Sequoia's Training Data podcast: your rate of token caching, since he puts cached tokens at about a tenth the price of uncached ones, how well the harness compresses a session once it exceeds the model's context limit, and how well it uses the environment to validate its own work. On compaction he suggests a needle in a haystack test, asking "How often will your harness preserve that through compaction?" Pick two of those, instrument them, and you can tell whether a change helped.

Should you build your own agent harness or use a provider's?

Build the parts your product's correctness depends on, rent the parts that are somebody else's core competency. In practice that means building the thin seam that lets you swap models, the policy on what a tool call is allowed to do, and the definition of a finished task. It means renting sandboxes, durable execution and observability, because those are hard, undifferentiated, and already sold by people who work on nothing else. The complication is that the layer you build is also the layer model providers keep absorbing, so anything you maintain to compensate for a current model weakness should be treated as temporary and reviewed on a schedule.

How do you set token budgets per class of work instead of per person?

Not as one blanket cap. Factory co-founder and CEO Matan Grinberg's point is that a uniform per person limit puts the same ceiling on people doing very different work: "the PMs who are like vibe coding dashboards get the same token limits as like the engineers who are building like critical infrastructure". He also says enterprises are currently setting these numbers by shooting from the hip. The workable version is to budget per class of work rather than per person: name three or four classes, give each one a model tier and a spend ceiling, enforce both in the runtime so an overrun stops a run instead of appearing on a bill, and revisit the numbers monthly against what each class actually produced.

AI Agent Infrastructure: Build or Buy

Nobody sits down and decides to build AI agent infrastructure. It accumulates. A retry somebody added after a bad demo, a queue for the jobs that started timing out, a container that runs the generated code, an environment variable holding the model name. Six months later that pile is load bearing, nobody owns it, and the first serious outage is an education.

This article is the component list for AI agent infrastructure. Not the cost of running it, which is covered in AI inference cost, and not the silicon underneath it, which is inference chips. Not the internal stack a company runs on itself either, which is internal AI infrastructure for startups. This is the layer you assemble yourself: what is in it, what each piece has to do once work starts running while nobody is watching, and which pieces are worth your own engineers.

The short answer on build or buy, before the detail: build the components your product's correctness and your security posture depend on, and rent the rest. In practice that is three things, the model seam, the tool execution layer, and the budget control. Sandboxes, durable run state and telemetry are somebody else's full time job.

AI agent infrastructure is everything between a model call and a finished task

When a product sent a prompt and rendered a response, there was nothing in that gap worth naming. The model was the system. An agent loops instead, so the gap fills up with machinery.

Tuhin Srivastava, CEO and co-founder of Baseten, put the change plainly in a Greylock clip on agentic inference. Models went from taking an input and producing an output to thinking, doing work in place, and needing an ecosystem of tools around them: tools "for running tools, for running other models, for routing between different models", plus sandboxes to execute code and a way to spin all of it up and down.

Read as an architecture note rather than a market observation, that is a component list, and it is longer than most teams realize they own:

The loop, or harness. Assembles context, picks the next tool call, feeds the result back, decides when the task is done.
Tool execution. The thing that actually calls the third party API, holds the credential, enforces the timeout, and truncates a result that came back too large.
The execution environment. A sandbox for generated code, with a filesystem, a network policy, and a lifetime.
Run state. Where a task that outlives a single HTTP request keeps its history, so it can be paused, resumed, inspected, or cancelled.
The model seam. The one place that decides which model gets a given call.
Budget enforcement. The control that stops a run before it spends more than the work is worth.
Telemetry. Enough of a trace to reconstruct a run you were not watching.

Each of those is a build or buy decision. Most teams make all seven implicitly, by accident, in the order the incidents arrive.

Most of your tokens are about to be asynchronous

The reason this stops being a tidy-up job and becomes an architecture problem is a change in who is waiting.

Matan Grinberg, co-founder and CEO of Factory, which sells enterprise software development agents it calls Droids, made the point on Sequoia's Training Data podcast. A lot of AI consumption today is synchronous, he says, and the test is brutal: "if everyone woke up sick tomorrow, a lot of Claude Code usage would be zero." A person types, an agent answers. His forecast is that this flips: "I think in 12 to 24 months, like 90% of tokens will be asynchronous tokens." Agents acting on a signal they found themselves, without anyone kicking them off. Today, he says, "right now we're still kind of in like Copilot mode." Even the agentic version is still human-triggered: "It is more agentic because it's not going to come back and ask you a ton, but it's still like you are kicking it off."

Treat the 12 to 24 months as one founder's estimate rather than a date. The direction is the part that changes your architecture, and four things invert when nobody is watching a run:

Failures stop being visible. A synchronous failure has a person looking at a spinner. An asynchronous one has nothing, until a customer notices a week of work that never happened. This is where the failure design in AI reliability stops being optional.
Cost accrues unattended. A loop that would have been killed by an impatient human at second 30 now runs for an hour.
Results need somewhere to land. Synchronous output goes to the screen the request came from. Asynchronous output needs a destination, an owner, and a rule for what happens when the destination rejects it.
Cancellation becomes a real feature. Not a nice one. When a user abandons a run, or a newer run supersedes it, something has to propagate that decision to a sandbox, a queue, and an in-flight model call. Nothing does this for you by default, and orphaned work is expensive precisely because nobody is looking at it.

The long-running execution machinery itself, queues, idempotency, pause and resume, is a solved and purchasable problem, which is the argument in AI workflow automation. What is not purchasable is your decision about which work is allowed to run unattended in the first place.

The harness has measurable properties, so measure yours

"Harness quality" sounds like taste. Grinberg's version is a list of numbers, which is much more useful, and it is the part of this article most likely to change what your team does on Monday.

Asked what keeps a harness at the frontier, he names caching first: cached tokens "end up being like a tenth as expensive", so a harness's rate of token caching is a direct performance property, not an accounting detail. Second is compaction. Any long session eventually exceeds the model's context limit, so the harness summarizes and compresses, and "the way that you perform during that compaction is a big determining factor of how good your harness is." The test he describes is a needle in a haystack: put one critical fact in a long thread and ask "How often will your harness preserve that through compaction?" Third is tool use, and specifically "how does it use the environment to validate whatever work that it's doing", which is the difference between an agent that claims it fixed something and one that ran the test.

Those unpack into four instrumentable numbers: cache hit rate, needle survival rate through compaction, tool call success rate, and the share of completed tasks that were validated against the environment rather than self-reported. Most teams track none of them and argue about prompts instead.

There is a real tension here worth naming rather than hiding. The case in vertical AI agents is that the model eats the harness: scaffolding you maintain today gets absorbed into the model tomorrow, and Sequoia's summary of Google DeepMind's Logan Kilpatrick on that episode puts a rough 12 month clock on it. Both things are true, and measuring is how you live with them. If you know your compaction score, you will know when a new model's longer context makes your compaction code dead weight, and you can delete it on evidence rather than keep it out of habit.

Model agnostic is a quality property, not just a hedge

The usual argument for keeping a thin seam between your product and a model provider is procurement: avoid lock-in, keep a price lever. That argument is covered in how to choose an LLM, and it is correct as far as it goes.

Grinberg makes a different and more interesting claim: a harness exposed to many models is a better harness. His analogy is to the early machine learning instinct to train a personal model only on your own data, which turned out to be worse than training on the whole internet. The analog that emerges, he says, is "what data is to a model, models are to a harness", because "the more models you expose to a harness, you avoid over fitting that harness to the nuances of that model in particular".

If that holds, a single-model harness carries a quality cost on top of the commercial one. It is quietly overfitted to one model's habits, and some of what you think is your agent logic is compensation for quirks that the next model will not have. He is describing his own product, so weigh it accordingly, but the implication is testable on your own stack in an afternoon: run your agent's evaluation suite against a second model without changing anything else. Whatever breaks is not agent logic. It is model-specific scaffolding you have been maintaining without knowing it.

That test is also the cheapest audit of component five on the list. If swapping the model touches more than one file, you do not have a model seam, you have a model dependency.

Token budgets belong in the runtime, not the quarterly review

Here is the component founders consistently leave out, and the one that gets expensive fastest once work runs unattended.

Grinberg describes how enterprises arrived at today's spending. Boards asked CEOs about AI strategy, CTOs pushed adoption into performance reviews and public usage rankings, and employees responded rationally by using the most expensive model for everything. The result, in his account of banks Factory works with, is that "they are spending literally hundreds of thousands of dollars a month on people asking things like literally what is the weather." He is not scolding anyone for it. He thinks adoption by any means necessary was probably the right first move, and that curbing usage afterwards is probably faster than limiting it from the start.

The correction he expects is not a smaller cap. It is a cap that knows what work it is capping. "Having a blanket kind of token cap for every individual in some large bank, let's say, makes no sense," he says. "So, every CIO is going to need to answer for every incremental token, where do we put it? And right now it is super not obvious how you would do that." His example lands close to home for anyone running a small engineering org: "the PMs who are like vibe coding dashboards get the same token limits as like the engineers who are building like critical infrastructure. That's probably not the best thing to do." And today, he says, "people are determining token budgets just by shooting from the hip."

For a startup, the practical translation is that a spending limit is a runtime control, not a finance report. A number you discover at the end of the month is not a budget, it is a receipt. A budget is something the runtime can enforce mid-run: this class of work gets this model tier and this ceiling, and a run that exceeds it stops and says so. Which means the budget has to be attached to a class of work, and somebody has to name the classes. Three or four is usually enough to start.

None of this is an argument for spending less overall. The case for spending hard on the best model where a customer can feel the difference is made in AI inference cost, and the reliability side of routing is in AI reliability. A per-class ceiling is what lets you spend aggressively on the classes that earn it without the weather questions quietly funding themselves. The point here is narrower: whatever routing you have needs a spend ceiling sitting next to it, or the routing is advisory.

The execution environment is a security boundary

Generated code has to run somewhere, and that somewhere is the part of your agent infrastructure a security reviewer will ask about first.

Anthropic's platform team has described the mature version of this from the vendor side, covered in how Anthropic's platform team handles sandbox governance: spawning sandboxes with the right governance, self-hosted sandboxes so execution can happen on infrastructure the customer already owns, and tunnels so an agent can reach a service that never leaves the customer's firewall. Worth reading if you are choosing a provider. What is less often written down is the set of questions you have to answer about your own sandboxes regardless of who supplies them:

What is the network policy? A sandbox that can reach the public internet can also exfiltrate whatever it was given. A sandbox that can reach your internal network is on your internal network.
What credentials does it hold, and for how long? The cleanest answer is none: the sandbox executes, the tool execution layer holds the credential and makes the call. That separation is the whole reason tool execution is its own component rather than something the loop does inline, and it is the control that is hardest to keep when the agent runs on a developer's own machine instead, which is the tradeoff in self-hosted AI agent.
What survives between steps? A filesystem that persists across an agent's steps is convenient and is also a place where step three's output becomes step seven's untrusted input.
What happens on cancel? A sandbox nobody destroyed is a running machine with credentials in its environment and no owner. This is the operational cost of cancellation being an afterthought.
Who can read the trace? A trace of an agent run contains everything the agent saw. Treat access to it the way you treat access to production data, a point made at more length in LLM observability.

None of this is exotic. It is the same isolation and least privilege thinking that has always applied to running untrusted code, which is what generated code is. The novelty is only that the untrusted code now arrives continuously and at volume.

Build or buy, one component at a time

The two general priors in circulation are that the model will eat your scaffolding, and that you should keep a thin seam. Both are true and neither tells you what to do with a specific component this quarter. Here is the call I would make for a startup with a small team and a product in production.

Component	Default	Flip it when
The loop or harness	Buy or use open source, then customize	Your domain's definition of a finished task cannot be expressed in anyone else's loop
Tool execution	Build thin, and own it	Never, in practice: this is where your credentials and your blast radius live
Execution environment	Buy	You have a hard isolation or residency requirement a vendor cannot meet
Run state	Buy, via durable execution	Your runs are short enough that a row in your own database is genuinely sufficient
The model seam	Build, and keep it small	Never: it is a few hundred lines and it is the lever behind every cost and quality decision
Budget enforcement	Build	Your provider exposes per-class limits that match your classes of work
Telemetry	Buy	You have a residency constraint that makes a hosted trace store impossible

The pattern underneath the table: build the parts where your product's correctness and your security posture live, rent the parts that are hard, undifferentiated, and somebody else's full time job. Nobody has ever won a market on their sandbox pool.

One caution on the harness row, which is the one most likely to be argued about. Whatever you build there to compensate for a current model weakness has a shelf life, which is the argument in build for the next AI model. The honest way to hold that is to write down, for each piece of scaffolding, whether it exists because the model is not good enough yet or because your domain requires it. The first list is a depreciation schedule.

This is the stack half of a wider operating question, which is how a small company decides what to run itself and what to rent, in AI and everywhere else. That is the ground our pillar on AI for startups covers.

What to do this week

Write the seven components on a whiteboard and put a name next to each. Not a vendor, a person. The ones with no name are the ones that will fail first, and the exercise usually takes twenty minutes.
Run your evaluation suite against a second model, changing nothing else. Whatever breaks is model-specific scaffolding you did not know you were maintaining. This is also the fastest test of whether your model seam is real.
Instrument two harness numbers. Cache hit rate and the share of completed tasks validated against the environment rather than self-reported. Both are cheap to capture and both move.
Name three classes of work and give each a model tier and a spend ceiling. Enforce the ceiling in the runtime so an overrun stops a run. A number you find on an invoice is not a budget.
Kill a run on purpose and watch what is left. Cancel an agent mid-task in staging and check for an orphaned sandbox, an in-flight tool call, and a queue entry nobody removed. Fix whatever survived.
Answer the five sandbox questions in writing. Network policy, credentials, persistence between steps, cancellation, trace access. If you cannot answer them today, that is the security review you have been postponing.
Write the depreciation schedule. For every piece of scaffolding you maintain, mark whether it compensates for a model weakness or encodes your domain. Review it when the next model ships.

Founders who can name the seven pieces and say who owns each one make faster decisions about what to build, what to buy, and what to delete when the next model absorbs it. Building that operating discipline, across the stack and not just the agent layer, is what we teach in AI Operating System for Startups.

Sources

How Inference Needs Change with Agentic AI (Greylock), the clip that frames this article. Baseten CEO and co-founder Tuhin Srivastava on inference moving from running a model to an ecosystem of tools for running tools, running other models, routing between them, and spinning sandboxes up and down.
Factory's Matan Grinberg: The Coming 'Dark Factory' Where Software Builds Itself (Sequoia Capital's Training Data), the main source here: harness performance properties, caching and compaction, the multi-model harness argument, and the token budget material.
90% of AI tokens will be asynchronous (Sequoia Capital), the clip with the asynchronous token forecast and the Copilot mode framing.
Background on Matan Grinberg and Factory: Sequoia's partnership note and, for the September 2026 round of $200 million at a $5 billion valuation, TNW, corroborated by Reuters. Profile: Matan Grinberg.
Baseten company background and Srivastava's role: Baseten and his author page.