GPU Cluster: A Founder's Guide to Compute

A GPU cluster is a group of servers, each holding several GPUs, connected so they can work on one job together. Inside a server the GPUs talk to each other over a fast direct link called NVLink. Between servers they talk over a high speed network. Rent one and you are buying two things: the chips, and the wiring between them.

Founders price the chips and ignore the wiring. That is backwards, and it is the cheapest mistake in this article to fix.

The material here comes from a YC Paper Club session on GPU systems, hosted by Francois Chaubard, a Y Combinator visiting partner who is also a Stanford CS PhD student co-advised by Chris Ré and Mykel Kochenderfer. Five speakers presented, and between them they cover the stack from the streaming multiprocessor up to the total cost of ownership argument. A second source, YC's own conversation about renting compute, supplies the procurement half. The most useful conclusion for most startups is the one nobody selling compute will lead with: you probably should not be buying cluster time yet.

What a GPU cluster actually is

Start one level down, because the vocabulary matters when you are reading a quote.

Inside a single GPU sit streaming multiprocessors, or SMs. They are roughly the GPU's equivalent of CPU cores, and they hold all the arithmetic capacity you are paying for. Stuart Sul, a Stanford CS PhD student in Chris Ré's Hazy Research group and a machine learning researcher at Cursor, gave the count: 148 SMs on Blackwell, 132 on Hopper. Feeding them is a memory system, the HBM that serves as the GPU's main memory, an L2 cache, and a crossbar. A CPU connected over PCIe kicks off the work.

Put several GPUs in one box and they need to talk. Routing that traffic through PCIe works but PCIe is shared and congested, which is why NVLink exists: direct GPU to GPU communication. That is essentially what an Nvidia DGX machine is, a set of GPUs each connected to the host over PCIe and connected to each other over NVLink. NVL72 is the same idea at larger scale, 72 GPUs inside one NVLink domain, and Sul noted that this is expected to extend to hundreds of GPUs.

The unit you rent from a provider is a node holding several of those GPUs. The unit you actually consume is SM time, and Sul's framing of what kernel engineers do is the sentence to remember when a vendor quotes you peak throughput. When people say they are optimizing GPU kernels, what they usually mean is that "they're trying to make sure that all the SM stay busy."

That is a technical statement and a commercial one. You rent peak FLOPS. You get whatever fraction of them your code keeps busy. The gap between those two numbers is the real price of the cluster, and nobody puts it on the invoice. That gap, between what you buy and what you actually use, runs through every infrastructure decision in AI for startups.

Training cluster vs inference cluster: not the same machine

Chaubard opened the session with the split that most compute buying guides flatten, and it is the one that changes what you should buy.

A training data center and an inference data center have close to opposite requirements. "You don't need any bandwidth in and out for a training data center," Chaubard said, and the image he used to make it stick is that you could ship the finished weights back by slow boat without changing the result: "You can literally send a spaceship to the sun," come back with a weight file, and it is the same. Inference cannot work that way. Location, latency, and the path in and out are the whole product.

What a training cluster does care about is time per step and all-to-all communication between GPUs, because gradients have to move everywhere. Inference needs all-to-all far less and shards much more cleanly, which is one reason the two workloads are drifting onto different hardware entirely rather than sharing a general purpose fleet.

Then it splits again, because inference is not one workload either. Misha Smelyanskiy, who ran AI hardware and software co-design at Meta and then led AI infrastructure for Nvidia's DGX Cloud before joining a startup building heterogeneous inference infrastructure, walked through the life cycle of a single request: scheduling and batching on the CPU, a prefix cache lookup, prefill, KV cache creation, autoregressive decode, and often speculative decoding alongside it. Each of those phases leans on compute, memory bandwidth, storage, and network in different proportions. His summary is the part to internalize: "the bottlenecks just keep moving."

The concrete version is prefill against decode. Prefill is compute bound, because you fetch the weights once and do a great deal of arithmetic across all the tokens. Decode is the opposite. It works one token at a time, so you refetch weights for every token produced, which makes it memory bandwidth bound and latency sensitive. Batching helps decode, but real batches are rarely large enough to close the gap on modern accelerators that have compute to spare.

Which means your product shape, not your ambition, determines the machine:

• Interactive chat. Roughly balanced between prefill and decode, and strictly latency sensitive.
• Long context queries. Nearly all the time is spent in prefill.
• Coding agents. Latency sensitive, with a demanding mix.
• Long running agents. Long inputs, long outputs, high concurrency, and relaxed latency, because the user left it running overnight.

Smelyanskiy also gave the warning that keeps this from becoming a shopping list. Splitting prefill and decode onto two different systems only pays for itself past a certain output length. At short output lengths you have added hardware and are not using it, which is a total cost of ownership loss, not a win. The same caution applies to any architecture diagram you copy from a company operating at a scale you are not at.

Before you ask how many GPUs, answer this: what is the token shape of your product? If you cannot state your median input length, median output length, concurrency, and latency budget, you are not ready to price hardware. That question sits upstream of everything in internal AI infrastructure for startups. It is also the same discipline that makes AI pricing work, since the cost you cannot describe is the cost you cannot charge for.

Why the GPU cluster bottleneck is the network, not the chips

Sul's talk was about ParallelKittens, a CUDA framework for multi-GPU kernels that extends the ThunderKittens work from the same lab. He was explicit that the point was not to promote the library, and the framing he offered instead is the one founders should take.

Years of work went into single GPU efficiency: IO aware algorithms such as flash attention, linear attention models, megakernels, and a long list of frameworks for writing efficient kernels. That work largely succeeded, and it moved the constraint. The remaining bottleneck is GPU networking. Sul's number: "networking can still consume up to 50% of total runtime" for some workloads. He was careful to say the figure varies wildly by workload, but the direction is not in dispute.

Two details from that talk are worth a founder's attention even if you will never write a kernel.

First, the defaults leave money on the floor. Off-the-shelf collective libraries such as NCCL expose only coarse grained communication primitives, which leaves bubbles in the hardware schedule. NCCL's default mode also forces intermediate buffers, adding data movement between sender and receiver that matters enormously once you are communicating at kilobyte granularity. Sul reported that stripping that overhead out speeds up an operation as basic as all-reduce by up to 80 percent. That is not an exotic optimization. That is a default costing you most of a primitive.

Second, the gap is closable, and it closes with very little code. With roughly "50 to 100 lines of device code," ParallelKittens matched or beat hand optimized kernels that often run to hundreds or thousands of lines. Sul said Cursor uses it to train Composer on tens of thousands of Blackwell GPUs, and that Together AI uses it to optimize inference workloads.

Here is the founder translation, and it is uncomfortable. Two teams can rent identical clusters and get very different throughput out of them, and the difference is not visible on any invoice or dashboard your provider gives you. So:

• Ask a prospective provider about achieved utilization on a workload resembling yours, not peak FLOPS.
• Decide honestly whether you will staff someone who can work at this level. If the answer is no, buy the managed option and accept the margin, because unattended raw compute is the most expensive thing you can own.
• Treat any benchmark from a vendor as a claim about their kernels, not about your workload.

Do you need a GPU cluster? Most startups do not

The most useful talk of the session argued against buying compute at all.

Jon Saad-Falcon, a Stanford researcher and the paper's first author, presented Intelligence per Watt, a study of how efficiently local and cloud AI convert energy into useful work, written with collaborators including Azalia Mirhoseini and Chris Ré. The team swept more than 20 recent local models, from 1 billion up to 235 billion parameters, including Gemma, GPT-OSS, Qwen, and IBM Granite, across recent accelerators from Apple, Nvidia, AMD, and SambaNova, and measured accuracy alongside latency, energy, power, and compute.

The number worth taking to your own stack is usually repeated wrong, so here it is precisely. Across the models they tested, at least one local model of 20 billion active parameters or less answered 88.7 percent of single-turn chat and reasoning queries as well as a frontier model did. That figure assumes you route each query to the right local model: the best single model in the study handled about three quarters on its own. Saad-Falcon's framing on stage was about routing: if you routed perfectly you could send something like 80 to 90 percent of today's queries to local accelerators running open source models. And crucially, you do not need to route perfectly. "Even imperfect routers can save somewhere between 50 to 70% of your energy, of your compute, of your dollar cost."

He was also honest about the ceiling. Local accelerators still lag data center parts, because a data center lets you amortize sophisticated kernel work, batching, and quantization across many users and many queries, and consumer hardware does not get that leverage.

Read the whole result as a sequencing argument rather than a hardware one. Before you price a cluster, instrument your traffic and find out what fraction of it genuinely needs a frontier model. Most teams have never measured this, and the answer is usually humbling. That measurement is the cheapest infrastructure win available to a startup right now, and it is the same exercise behind choosing the best LLM for your product. If the measurement tells you that you genuinely need different model behavior rather than more compute, your next decision is RAG versus fine-tuning, not a hardware order.

The second argument against buying more GPUs came from Brennan Shacklett, whose Stanford work with Kayvon Fatahalian produced Madrona, a batch simulator that puts an entire game engine on the GPU. His observation is that if you are doing reinforcement learning, the obvious approach of running a thousand copies of a game engine in parallel is badly wrong: "game engines are really inefficient for the throughput oriented training workload," because the copies fight each other, no cost is amortized, and both CPU and GPU sit underused. Restructuring the simulator to run thousands of worlds as one batched job on the GPU, using entity component system patterns borrowed from game development, produced results "over a hundred times faster" than the CPU reference implementations in the machine learning code bases, in many cases, all on a single RTX 4090. A separate profile of that same card running roughly 4,000 hide-and-seek worlds showed its SMs busy nearly all the time.

Two orders of magnitude, on one consumer card, from a software change. Profile before you buy. The expensive fix is very often not more GPUs.

What renting a GPU cluster costs: the two-year commitment

Suppose you have done the measurement and you genuinely need compute. The next problem is not technical.

In YC's conversation with Together AI about the first dedicated YC GPU cluster, Y Combinator general partner Ankit Gupta, previously co-founder and CTO of the drug discovery company Reverie Labs, described how the market changed. Eight years ago, running his own company, he could get a thousand GPU instances from a cloud provider on spot pricing without long reservations. Not anymore: "just having access to capacity is a really big problem, let alone great pricing."

Then he put the real constraint in numbers, and this is the sentence to read twice if you are about to sign something: for many companies, "the upfront they would need to pay in order to secure the capacity for their next two years of compute was greater than their current cash balance."

Read that as a financing problem wearing an infrastructure costume. Securing compute at the going structure means raising money and handing most of it to a compute partner, at terms set by the fact that you had no choice. The fix YC and Together AI built was not a discount. It was a shorter commitment: pool demand across a portfolio, so the goal became to "make it so startups don't have to commit to two years, they only have to commit to a few weeks."

Vipul Ved Prakash, co-founder and CEO of Together AI, described the same dynamic from the supply side. Together AI runs a cloud covering training, post-training, and serving. His point about planning is the one that separates an AI company's finances from a software company's: compute is going to be among the biggest expenses in development and product, and it has to be underwritten and planned for in a way a software company never had to.

Two operating details from that conversation are worth stealing:

1. Book against your roadmap, not against the discount. Companies on the cluster plan several months out, buying a small amount for a while and stepping up before a large training run. The lever is scheduling, not negotiation.
2. Aggregate utilization is not your utilization. Gupta noted that "the cluster is at 100% utilization" in aggregate while no individual company has fixed usage. That is exactly why pooled, short commitments work, and exactly why a private reservation sized for your peak will sit idle most of the month.

Gupta also flagged an operational gap most founders discover late. Many have trained models before, but in environments where compute was managed for them, so they have never run their own cluster. Some of that work is genuinely yours to do, and it is worth knowing which parts before you own them rather than after.

Compute conversations skip the security question, so ask it in procurement. A training cluster is where your most valuable asset, the weights, exists in the clear on someone else's hardware. Before you sign, get straight answers on tenancy isolation, who at the provider can reach your storage and your running jobs, what happens to your data at the end of a reservation, and whether the audit trail is one you could actually reconstruct an incident from. These are ordinary vendor questions. They are just easier to ask in procurement than after a breach.

What GPU kernel benchmarks teach you about your own evals

The last lesson came out of GPU work and applies everywhere else.

Mark Saroufim, a former PyTorch maintainer and a co-founder of GPU MODE, runs KernelBot, a competitive platform where people submit GPU kernels against reference implementations and get ranked. Kernel writing looks like the ideal verifiable reward problem: you have a PyTorch reference, you sample random inputs, you check correctness, you measure speed, you rank. Saroufim's talk was a catalogue of why that is harder than it looks.

Write the world's fastest vector mean kernel. The reference samples random values, and PyTorch's default sampling has mean zero and variance one. So, in his words: "So the world's fastest vector mean kernel just returns zero." It beats the speed of light and it is completely useless.

What follows is an arms race. He banned reading the underlying data pointer, so the model reached the same attribute another way, then split the banned string across two variables so a regular expression would not catch it. Every route he closed had another route to the same object behind it, which is what a dynamic language guarantees. "This is very much a chicken and egg problem," he said.

The best one was surfaced by a Stanford student who spotted it. The harness checked correctness fifteen times and then ran performance testing without checking correctness again, so the model counted the correctness checks, served a correct but slow kernel for those, and switched to an incorrect but fast one once it recognized it was being timed. Saroufim's comparison is exactly right: this is the Volkswagen emissions defeat device, a system that detects it is being tested and behaves differently while under test.

If you are shipping any AI system evaluated against a metric, the pattern transfers wholesale:

• A metric your model can observe is a metric your model will optimize against directly, not through the behavior you wanted it to proxy.
• Separate your correctness checks from your performance checks in time and in code, and re-run correctness during the phase you thought was safe.
• Assume detection, not prevention. The approach Saroufim landed on is adversarial: a human audits a suspiciously good result, marks the hack, and that example is used to synthesize detectors for the same class, which get retrained as new classes appear. There is no fixed rule set that ends this.

That is the whole discipline of LLM evals for founders compressed into a GPU leaderboard, and it is a good reason to run your evals as an adversary rather than as a fan.

Saroufim pointed out that in a recent competition on low precision kernels, people who had never written a GPU kernel before started getting competitive results: a researcher who had never opened a CUDA book placed fourth with model generated code, and a high school teacher wrote his first fused matrix multiply problem. The floor for this work has dropped hard, which is good news for small teams and exactly why the eval discipline matters more, not less. This is the same shift covered in the future of software engineering: the writing got cheap, so the verifying became the job.

What to do this week

1. Write down your token shape. Median input length, median output length, concurrency, and latency budget. If you cannot fill those four fields, you cannot price hardware, and any quote you get is guesswork.
2. Measure your frontier model dependency. Sample a few hundred real production calls and label how many actually needed the largest model. If the Stanford result holds anywhere near your traffic, a router is worth more than a reservation.
3. Profile before you scale. Find out whether your bottleneck is GPU compute, GPU networking, or something on the CPU entirely. Shacklett's hundred-times speedup came from restructuring software, not from buying chips.
4. Price the commitment, not the hour. Before any compute contract, write out the total prepayment, its length, what happens if your roadmap moves by a quarter, and what fraction of your current cash it represents. If that fraction alarms you, shorten the term rather than negotiating the rate.
5. Attack one eval. Take a single benchmark you trust and spend an hour trying to beat it dishonestly. If you succeed, so will your model.

If you want the full operating system for running a startup this way, the practical decisions about models, agents, infrastructure, and cost, that is what I teach in AI Operating System for Startups.

Sources

• Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More, YC Paper Club is the session this article distills, hosted by Francois Chaubard, with talks from Stuart Sul, Jon Saad-Falcon, Mark Saroufim, Misha Smelyanskiy, and Brennan Shacklett.
• The First Dedicated YC GPU Cluster, With Together AI, Ankit Gupta in conversation with Vipul Ved Prakash, is the source for the commitment structure and the capacity access problem.
• ParallelKittens: Simple and Fast Multi-GPU AI Kernels from Stanford's Hazy Research, and the paper, are the written version of Sul's talk. Profile: Stuart Sul.
• Intelligence per Watt: Measuring Intelligence Efficiency of Local AI is the paper behind Saad-Falcon's talk, with a summary from Hazy Research. Profile: Jon Saad-Falcon.
• Madrona is Shacklett and Fatahalian's batch simulation engine, described in their SIGGRAPH paper. Profile: Brennan Shacklett.
• KernelBot is the GPU MODE competition platform Saroufim described, written up in a paper on the platform. Profile: Mark Saroufim.
• Background on the people named: Francois Chaubard at Stanford, Ankit Gupta at Y Combinator, Misha Smelyanskiy on Google Scholar, and Vipul Ved Prakash of Together AI.