What is an inference chip?

An inference chip is a processor built to run a model that has already been trained, rather than to train one. The design assumption baked into almost all of them is that inference means one prompt in and one response out, which is why they show their limits on agents: Y Combinator estimates that current GPUs reach only 30 to 40 percent of peak utilization on agent workloads, because that work keeps stalling between model calls, tool calls, and orchestration logic. The newest parts are being built against the new assumption rather than the old one. Google positions its seventh generation TPU, Ironwood, as the first Google TPU for the age of inference, and Nvidia now sells a separate rack, Groq 3 LPX, that attaches to its Vera Rubin systems purely to make token generation faster for agents.

Who makes inference chips for AI?

Nvidia is the default, both through its general purpose data center GPUs and, since December 2025, through the roughly 20 billion dollar Groq deal that produced the Groq 3 LPX accelerator. Google builds its own TPUs, with the seventh generation Ironwood aimed at inference: announced in April 2025, in preview from November 2025, and generally available during 2026. Amazon builds its own silicon for its cloud, Inferentia for serving and Trainium for training, and AMD sells the Instinct line as the main GPU alternative. The practical point for a founder is that you almost never buy any of this directly. You buy it through a model provider or a cloud, which means the question that matters is not which chip is fastest but which provider will tell you what your workload actually achieves on it.

Why are inference chips a problem for AI agents?

Inference chips are a problem for AI agents because they were designed for a different shape of request. Y Combinator's framing is that most AI chips assume inference means prompt in, response out, while agents loop, call tools, branch, backtrack, and hold context across dozens of steps. That work is bursty, and by YC's estimate current GPUs reach only 30 to 40 percent of peak utilization on it, because the machine keeps bouncing between memory bound model calls, IO bound tool use, and CPU bound orchestration. Nothing on the market today is designed for the loop itself: fast switching between models, speculative decoding built into the silicon, and memory that holds a run's key-value cache across an entire execution graph instead of throwing it away between steps.

Is inference better on CPU or GPU?

For anything you would put in front of a user, inference runs better on a GPU or a purpose built accelerator than on a CPU, because generating tokens is limited by how fast the machine can stream model weights out of memory and a CPU's memory bandwidth is far lower. CPUs remain reasonable for small models, low request volumes, and batch work where a slow answer is fine. For a founder building agents, though, the more useful version of the question is not CPU versus GPU at all. It is whether your workload is running at batch size one, where a single user is waiting on a single response and you are paying for latency rather than throughput. That is a different machine from the one most published benchmarks were measured on.

Inference Chips Are Built for the Wrong Job

An inference chip is a processor built to run a model that has already been trained. Training silicon and serving silicon have been diverging for years, and that part is well understood. What has not kept up is the assumed shape of the request.

Y Combinator put the problem plainly in a recent request for startups: "Most AI chips are designed for a world where inference means prompt in response out." One question, one answer, done.

That is not what an agent does. In YC's words, "They loop, calling tools, branching, backtracking, holding context across dozens of steps."

If you are building agents, that mismatch is already costing you, and it does not show up as a line on your bill. This article covers what the gap is, what the Nvidia and Groq deal actually was (the reporting is more interesting than the headline), and the part most founders can act on: what to change in your own stack when you are not the one building silicon.

Why inference chips miss the agent loop

A single model call is a clean workload. Load the weights, read the prompt, generate tokens, finish. A chip designer can plan around it.

An agent run is not one call. It is a model call, then a tool call that waits on a network, then some orchestration logic on a CPU deciding what happens next, then another model call carrying everything that came before, then a branch that turns out to be wrong and gets backtracked. Repeat across dozens of steps.

YC's estimate of what that does to a GPU: "Current GPUs hit 30 to 40% of peak utilization on these workloads because the work is bursty, bouncing between memory bound model calls, IO bound tool use, and CPU bound orchestration."

Treat the specific percentage as an estimate rather than a measurement, because YC does not publish the workload behind it. The claim holds even if the number does not. You are paying for a machine that is idle for a meaningful share of every agent run, waiting on an API, waiting on a filesystem, waiting on your own control flow. That is the same gap our post on renting a GPU cluster describes at the kernel level, where you buy peak FLOPS and receive whatever fraction your code keeps busy. The agent loop makes it worse, because the idle time is structural rather than a symptom of unoptimized code.

YC's argument is that the gap is a startup opportunity: "That gap is where purpose-built silicon wins."

The three missing features it names are specific enough to be worth reading closely, because they double as a description of what your agent is doing badly today. "Fast context switching between models, native speculative decoding" and, third, memory built to hold key-value caches across an entire execution graph: the cache of everything the model has already read, kept for a whole run instead of rebuilt at every step.

What the Nvidia and Groq deal actually was

The YC clip says "Nvidia bought Groq for 20 billion because it saw this coming." That is the popular framing, and it is close enough to be useful and wrong enough to matter.

What was actually reported, starting December 24, 2025: Nvidia agreed to pay roughly 20 billion dollars in cash for Groq's assets and to hire its leadership, including its CEO. Groq's own announcement described a non-exclusive licensing agreement for its inference technology, and said GroqCloud would continue operating. CNBC, which broke the price, reported it as Nvidia's largest deal on record. Reuters covered the licensing and hiring structure and attributed the 20 billion figure to CNBC, noting that neither company confirmed it. Nvidia later said publicly that it had not acquired Groq, only licensed its intellectual property and hired people, so treat the number as reported rather than confirmed.

The structure matters more than the price. A licensing agreement plus an asset purchase plus the hiring of key people gets you the technology and the team without the acquisition review that buying the company outright would have triggered. Analysts said as much at the time, and CNBC ran a follow-up on exactly that reading.

Then the outcome shipped. On August 24, 2026, Nvidia announced that Groq 3 LPX is in full production, sold as an extension of its Vera Rubin platform and aimed explicitly at agents. Nvidia's own framing of the problem is almost word for word the YC framing: "Agentic AI creates two distinct computing challenges: efficiently processing enormous amounts of context and generating tokens with extremely low latency."

The performance numbers are Nvidia's, measured by Artificial Analysis, and worth reading as vendor claims rather than neutral results: a record 3,400 output tokens per second running Gemma 4 31B at a 100,000 token context, and 4 times faster responsiveness for agents than what Nvidia calls the nearest alternative platform. Nebius is named as the first AI cloud to adopt it.

For a founder, the practical reading is not the benchmark. It is this: the gap YC described as a rare opening for a startup is being closed by the incumbent, in public, with a dedicated rack, roughly eight months after the deal was reported. Google is on the same track, describing its seventh generation TPU as the first Google TPU for the age of inference, announced in April 2025, in preview from November 2025, and generally available during 2026.

YC's own clip half concedes the point: "Google built TPU v7 for inference specifically, but nobody's designing for the agent loop itself." Worth noting that Nvidia had already unveiled Groq 3 at GTC in March 2026, about six weeks before that clip went up, which makes the point sharper rather than softer. A gap that visible does not stay open, and the direction of travel is what to plan around.

The compiler is the product, not the chip

The single most transferable line in the YC clip is not about hardware at all. "Groq's real insight wasn't the chip." It was the compiler that made the chip work.

That is accurate, and it is part of why Groq was worth that much to a company that already makes the best AI chips in the world. Nvidia's own public rationale leans more on the hardware, the LPU's memory architecture working as a dedicated decode stage alongside its GPUs, so read the compiler claim as YC's reading rather than the buyer's. Groq's design pushes scheduling out of the hardware and into compile time. By Groq's own account, the compiler maps and schedules the whole program in advance, so execution is deterministic down to the clock cycle and the chip does not need runtime arbitration logic to decide what runs next. Remove the runtime decisions and you remove the jitter, the contention, and the tail latency they cause.

Jensen Huang made the same argument about Nvidia's own history, on stage at YC's Startup School, describing the realization that built the company: "it's not about building a great chip, it's about accelerating an algorithm domain."

Two consequences follow for anyone reading this as strategy rather than trivia.

If you are building silicon, the software is the defensible half, and it is the half that takes longer. Hardware differentiation gets absorbed. A compiler that understands your workload domain better than anyone else's does not.

If you are buying inference, the compiler layer is also where lock-in lives. The chip is an implementation detail you will never touch. The runtime, the kernels, and the serving stack around it are what your code ends up shaped by, and they are what makes switching expensive two years from now. That is the same reason a competitive moat in AI rarely sits in the model itself.

Inference is turning into a capability, not a cost line

The most useful reframe here came from a different YC session. At the first YC Paper Club, Tanishq Kumar, a Stanford CS PhD student and first author of the Speculative Speculative Decoding paper with Tri Dao and Avner May, opened by dispatching the familiar argument: "inference costs are high. They dominate training costs when you're serving a model for billions of users."

Then he pushed past it. Inference costs do not merely dominate training costs, he argued. Even inside training, reinforcement learning is starting to exceed the compute requirements of pre-training, and in his words, "what is RL but a wrapper on inference".

Then came the claim he flagged as the one thing to take away from the talk. Inference today "is seen as a sort of like cost or convenience lever", but within a few years, he expects it to be "seen as a capability."

If your system's answer quality scales with how much thinking it does, then "the speed at which you can do inference, the tokens per second is exactly the peak intelligence that you can deliver."

That is a hard constraint disguised as a performance metric. An agent that gets a 3 minute budget before a user gives up can take as many reasoning steps as fit in 3 minutes. Make each step twice as fast and the agent does not just feel faster, it gets to try more approaches, verify more of its own work, and back out of more dead ends inside the same wall clock. The ceiling on your product's quality is partly set by hardware you do not own, which is exactly why the shape of the next generation of inference chips is a founder question and not only an infrastructure one.

Note also that speculative decoding, the technique Kumar works on, is one of the three features YC named as missing from current silicon. The research community and the chip designers are converging on the same list.

What changes for you if you are not building silicon

Almost nobody reading this is going to start a chip company. The useful question is what to do differently while the silicon catches up. Five things, and none of them require new hardware.

Measure the loop, not the call. Tokens per second on a single completion is the number every vendor quotes and close to the least useful one you have. What decides whether your agent is usable is time to a finished task, across every model call, tool call, and retry in the run. Instrument the whole run, tag each span by what it was waiting on, and you will usually find the surprise is not in the model calls at all. That is a straightforward extension of what LLM observability is for.

Find out how much of your loop is spent rebuilding context. Every step that re-sends the whole conversation is paying to reprocess text the machine already read. This is precisely the problem the missing hardware feature would solve, and it has a software answer available today: prompt caching, offered in some form by every major provider. If you are not using it on your agent's system prompt and stable context, that is likely the largest single latency and cost win available to you this quarter, and it costs an afternoon.

Assume you are running at batch size one. Your agent is one user waiting on one response. Vendor throughput numbers are measured with many requests packed together, which is a fundamentally easier problem. When you compare providers, ask for latency at your concurrency, not aggregate tokens per second, and be suspicious of any number quoted without a batch size next to it.

Stop switching models casually mid-loop. Routing different steps to different models is a real cost optimization, and it is also one of the things current hardware handles worst, which is why "fast context switching between models" is on YC's list of missing features. Route deliberately, at a small number of clear decision points, and measure the latency cost of each switch instead of assuming it is free.

Treat a persistent cache as a data boundary. Almost nobody does. A key-value cache that survives across an entire agent run, which is exactly what the next generation of chips is being designed to hold, is a store of everything your agent has read. When that cache lived for one response, a mistake in its isolation leaked one response. When it lives for a whole run, or is shared across runs to save cost, the blast radius is the entire task and potentially the entire tenant. Before you turn on any caching feature, ask your provider in writing how cache entries are keyed, whether they can ever be shared across tenants, and how long they persist. Ask now, while the answer is a paragraph in a doc, rather than after an incident. Where this fits in the broader picture of building on AI is covered in our pillar on AI for startups.

Should you start an inference chip company?

YC's Summer 2026 request for startups laid out an unusually concrete specification: silicon designed for the agent loop rather than the single response, with fast model switching, speculative decoding in hardware, and memory that holds a run's cache across the whole execution graph.

Two honest cautions before anyone treats that as a green light.

The window is narrower than the clip implies. Nvidia unveiled Groq 3 before that request was published and had the rack in full production four months after it, bought with a reported 20 billion dollars. Any pitch here now has to answer what you do that Groq 3 LPX and Ironwood will not do in eighteen months.

And the hard half is not the hardware. Groq's story says the compiler was the durable asset. If your founding team can design the chip but not the compiler that makes it worth using, you have built the replaceable half. YC's closing point is that this is a rare moment where understanding both the chip architecture and how agents actually execute matters, and that is really a statement about who is on the founding team.

What to do this week

Instrument one full agent run end to end. Not per call. Per task, with every span tagged by what it was waiting on: model, tool, network, your own orchestration. You cannot fix a utilization problem you have never looked at.
Turn on prompt caching for your agent's stable context. System prompt, tool definitions, any long document that does not change between steps. Measure latency and cost before and after on the same ten tasks.
Ask your inference provider three questions. Latency at batch size one for your model, how prompt cache entries are keyed and isolated, and what happens to a cached context when a run is idle for a few minutes. Get the answers in writing.
Count your mid-loop model switches. If your agent changes models more than two or three times in a run, price what each switch costs in latency before keeping it.
Write down what your agent's time budget actually is. The number of seconds a user will wait before abandoning the task. Every hardware decision downstream is really about how many reasoning steps fit inside that number.
Do not buy hardware. Nothing in this article argues for owning silicon. It argues for knowing which constraint you are actually up against before someone sells you a solution to a different one.

The pattern under all six is that inference is becoming an operating constraint on product quality, not a line item to be minimized, and that means it belongs in the same conversation as pricing, staffing, and roadmap rather than in a corner of the infrastructure budget. Setting those defaults deliberately across the company is what the AI Operating System for Startups is built around.

Sources

Inference Chips for Agent Workflows (Y Combinator), the request for startups this article distills.
Jensen Huang: The Mindset That Built NVIDIA (Y Combinator Startup School 2026), for the argument that the company was built on accelerating an algorithm domain rather than on the chip.
Inference, Diffusion, World Models, and More (YC Paper Club), where Tanishq Kumar of Stanford made the case that inference is becoming a capability rather than a cost lever. His paper with Tri Dao and Avner May: Speculative Speculative Decoding.
The Nvidia and Groq deal: Groq's announcement of the non-exclusive licensing agreement, CNBC, Reuters, and CNBC on how the deal was structured.
Nvidia's announcement that Groq 3 LPX is in full production (August 24, 2026), for the product claims, the Artificial Analysis benchmark figures, and the Nebius deployment. Performance numbers there are Nvidia's own.
How the Groq LPU works, Groq's explanation of compile time scheduling and determinism.
Ironwood: The first Google TPU for the age of inference and Google Cloud's TPU7x documentation, for the seventh generation TPU and its inference positioning. The Cloud TPU release notes are the source for the timeline: preview from November 2025, general availability in 2026. Several trade outlets reported the November preview as general availability, which it was not.
Nvidia unveiled Groq 3 at GTC in March 2026, before the request for startups was published: Tom's Hardware on that reveal and on the LPU working as a decode stage inside the Vera Rubin platform.