AI Simulation: What It Can Actually Answer

AI simulation is the practice of building a model of a system you can act on, so you can read the consequences of a decision before paying for it in the real world.

That definition is doing more work than it looks like, because three different businesses now share the name and they have very different physics. One is close to a commodity you should buy. One only pays off if you own data nobody else has. And one is a research program where the simulator turns out to be harder to build than the product it was supposed to help you ship.

Two people who have spent years inside the expensive version of this problem talked about it publicly this year. Joon Sung Park, co-founder and CEO of Simile, went on 20VC shortly after the company raised a $200 million round at a $2 billion valuation, roughly five months after its $100 million Series A, taking total funding to about $300 million. Dmitri Dolgov, co-CEO of Waymo, laid out the lessons from building the simulator behind a fleet that has now driven more than 220 million fully autonomous miles.

They work on opposite ends of the category, one simulating people and one simulating streets. Between them they supply the three questions that decide whether any simulation is worth what it costs you.

The three things founders mean by AI simulation

Physical simulation reproduces an environment and its physics well enough to train and evaluate something that moves through it. This is the oldest family, it is what most of the market currently sells under the term, and it is where autonomous vehicles and robotics live. The current generation is built on learned models rather than hand-coded rules, which is why it now overlaps with world models, the models that predict how a scene changes when you act on it.

Training environments reproduce a slice of a job so a software agent can practice in it and be scored. A world of realistic documents, clones of the apps the work runs in, and tasks paired with verifiers. This is the family behind most agent progress in the last two years, and it is expensive for a reason that surprises people: the verifier costs more than the world.

Behavioral simulation reproduces people. You describe a population, the system models it, and you ask what that population would do if you shipped a product, changed a price, or ran a campaign. This is Simile's category, and the question of how far you can trust the answers is its own subject, covered in the piece on synthetic users.

Mapped back onto the opening: physical simulation is the one you should almost always buy, behavioral simulation is the one that only pays off if you own data nobody else has, and a training environment is the one where the simulator can quietly become harder than the product it was meant to serve.

The families share a purpose rather than a technique: replacing an experiment you cannot afford to run with one you can. Everything below applies across all three, because the failure modes are shared even when the code is not.

Are you buying a prediction or a counterfactual?

A prediction tells you what will happen. A counterfactual tells you what would happen if you did something different. Only the second is a decision tool, and most founders buy the first while believing they bought the second.

Park's position on this sounds like a provocation at first. Observational data, he argues, is excellent at correlating what you saw with what happens next, which makes it good at prediction. And then: "no one really cares about prediction." He allows one exception, which is anyone trying to predict the stock market. His example is a large retailer being told a product line will decline in two quarters. They hear it and the only available response is that this is terrible, followed by asking what to do. "What people actually care about is they want to shape the future."

To answer that second question you need something a forecast cannot give you. You need the model to represent a causal mechanism, so it can answer what happens if we do this instead. That is a counterfactual, and it is a different engineering problem. It is also why Simile's training data is not only observational: Park says the class of data they care most about is randomized experiments, "the kind of data that we care deeply about is a lot of randomized control trials," showing the model how behavior changes when one thing is varied and the rest is held still.

The same requirement shows up at the other end of the category, in almost the same words. Dolgov distinguishes open loop evaluation, where you passively score input and output pairs, from closed loop, where the agent acts, the world responds, and it acts again. His verdict: "the ability to take an action and evaluate that counterfactual is absolutely vital for building and deploying safety-critical agents in the physical world." One is simulating shoppers and the other is simulating traffic, and both land on the position that watching is not enough. You have to be able to intervene.

There is a practical buying test hiding in this. Ask the vendor, or your own team, what lever the output attaches to. If the answer is a number about the future with no accompanying change you could make, you are buying a forecast in a simulation wrapper. Park's framing of what a real one delivers is the standard to hold it to: "It's one thing to simply predict, but can we actually show here are all the steps that your ecosystem is going to take to get to that particular outcome?"

What is your ground truth loop?

Your ground truth loop is the mechanism that tells you, on a schedule you can live with, whether the simulation was right. Without one, accuracy never improves, and you will not find out.

Coding agents improved faster than everything else for an unglamorous reason. The reward signal was sitting right there. A suggestion gets accepted or rejected, and the system learns from a clean, immediate answer about whether it was any good.

Simulation looks like the opposite case. Everything it claims is about a future that has not happened, so where does the correction come from? Park thinks the objection has it backwards, and his answer is the most useful idea in the interview: "the world is our ground truth. We live in the ground truth world."

The mechanism is specific rather than poetic. Simile generates hypotheses continuously, at volume, and each one is written as a statement that reality will eventually settle. "Each hypothesis is mapped onto an end statement." Then they wait and score. Park puts the volume at "every single day, we can be generating tens of thousands of hypotheses," and they watch which ones become answerable and when, reading the hit rate over a month as a measurement of the simulator itself.

Waymo built the industrial version of the same loop, and Dolgov's description of it is the clearest architecture diagram a founder is likely to get for free. You cannot ship one model, he says, because "you can't just build a model and call it a day." You need three: the agent that does the work, the simulator that gives it somewhere to practice, and a third piece most teams skip. "And the critic is what rigorously evaluates and judges the performance of the agent and tells it how to improve."

The critic is the part that matters here, and it is the part that goes missing. A simulator without a critic is a very expensive opinion generator. It will produce beautiful, plausible, confidently wrong answers indefinitely, and nothing in the system will ever notice, because nothing in the system is scoring it. Dolgov's summary of where this ends up for physical agents is worth reading as a general rule and not a robotics one: "closed loop simulation is absolutely required for evaluation and is extremely valuable for training of your physical AI agents." Evaluation is the non-negotiable half. Training is the bonus, and in physical simulation that same ordering is where the first commercial demand shows up.

So before you buy or build anything, write down two sentences. How will I find out this simulation was wrong, and how long will that take? If you cannot answer the second one in weeks rather than years, you do not have a ground truth loop. You have a demo with a subscription attached, and it will be exactly as accurate in year three as it was in week one. Teams that already run disciplined evals on their models have most of the muscle for this and usually do not realize the same discipline applies here.

Is the simulator harder than the thing it simulates?

Often yes, and that is the answer that should stop most build plans before they start. Dolgov is unusually blunt about it for someone whose company did build.

His starting point is that "a real simulator isn't just some lightweight tooling that sits next to your AI." It is a large model in its own right, and the AI inside it "really needs to understand how the world works, the physics, the semantics, you know, the traffic, the weather." Then the sentence to write on the wall before anyone opens a repository: "the problem of building a good realistic simulator is just as hard as building the agent itself."

Sit with what that implies. If you are building an agent and you decide you need a simulator to train it properly, you have not added a workstream to your roadmap. You have roughly doubled it, and the new half is a research problem with a much less certain schedule than the original one. Waymo could take that on because the simulator is inseparable from the product and the safety case, and because the alternative was validating a safety-critical system on public roads.

Park's cost curve says something similar from the money side. Simile has a model in production that "used to cost about 100 times more to run than it does now," and he is clear about where the spending actually went. The early compute did not buy efficiency, it bought a point of view: exploring ways to train, exploring datasets, finding an approach worth optimizing. Efficiency came quickly once they knew what they were making. "A lot of the investment, however, does go to find that initial point of view."

That is the honest shape of the bet. The expensive, slow, uncertain phase comes first, before you know whether the thing works. Two positions justify entering it. Either the simulation is what customers pay you for, or you own proprietary data about a system nobody outside your company can model and the simulator is how that data becomes a product. If neither is true, buy the environment and spend your scarce engineering time on the critic, which is the piece vendors will not build for you anyway because it encodes what correct means in your business.

The moat is the data strategy, not the simulator

Park's answer to what makes an AI company defensible right now is a single sentence and it applies far outside simulation: "My fundamental thesis here is for AI companies of this generation, you need to have an interesting data strategy that's going to be defensible." Asked where he would invest if he could not say simulation, he came back to the same test, phrased as a question to put to any AI startup: "Do you have access to data that no one else has has access to?"

What makes Simile's version instructive is that the hard part is not volume. It is two things that do not sound like a data strategy at all.

The first is who you collect from. Simile does not compete for expert data, and Park is explicit about it: "We don't go after these expert programmers or expert scientists." They go after everyday people instead, and the property they optimize for is representativeness, whether the modeled population matches the real one. The second is what you collect: behavioral data rather than web text, which is the grounding requirement separating a synthetic user from a prompted persona. Closing that gap means running your own experiments rather than harvesting exhaust.

There is a design consequence buried in this that founders building on frontier models should notice. Simile is not trying to make its models smarter. Park describes the goal as the inverse: "What we care about is if we have a person make a mistake in this context, we want our models to make the same kind of mistake." He adds, in the same breath, "We want our models to be biased in the same way humans are." A model that reasons better than the population it represents is broken for this purpose, which is a genuinely unusual engineering goal and a useful reminder that a wrapper around someone else's model is not a moat while a proprietary data asset can be.

What an AI simulation costs, and which ones earn it

Expensive simulations are usually the ones worth buying. That runs opposite to normal software intuition, and understanding why saves you from optimizing the wrong end.

Simulations vary a lot in cost. A question about downstream consequences, or a segmentation study spanning a whole national market, runs far more expensive than a narrow one. Park's observation is that those expensive simulations are precisely the ones that return the most, because the decisions they inform are the costliest ones to get wrong. The value scales with the decision, not with the compute.

Pushed to its conclusion, he expects the ceiling to be much higher than today's software pricing. His prediction is that within roughly two to three years a single simulation session might consume 10 to 20 million dollars of compute and be worth 100 million dollars to the buyer. Pressed on who that buyer would be, he confirmed the top of the market: the largest enterprises and governments. Treat that as a founder's forecast about his own market rather than a fact, but the underlying logic holds regardless of the numbers: inference costs have been climbing as models think longer, and simulation is a natural next consumer of that budget.

The value case is not mainly optimization. Asked how he closes the gap between the value created and the price charged, Park's answer centered on avoided disasters, decisions that would have cost hundreds of millions had they been made. He was careful not to accept the clean framing that this is prevention rather than optimization, saying it is both, but prevention is the obvious one, which is why he calls it a painkiller.

Two more numbers are worth carrying around, because they describe the real competition. In one early sales call, a prospect asked Simile to rerun a study large consultancies had already completed. Park says "we predicted the outcome of studies that took 3 to 6 months, but just within 2 minutes." And more importantly, most questions never get studied at all: "You are looking at maybe 5% of those ideas get answered." The rest go untested for want of budget and time, which means "a lot of the decisions that we make as a society, we base on our gut instinct."

That last figure is the one to plan around. The honest opportunity for simulation is not displacing the studies you already run and trust. It is the ninety-five percent you currently decide by instinct, where the bar it has to clear is not a rigorous experiment but a guess in a meeting.

Before you let a simulation into a decision

Two failure modes deserve attention before any of this touches a real choice, and the second one is a security problem people do not file as one.

The first is reading a single number off a question whose answers genuinely spread out. Some systems converge, and there the simulation can be read for an answer. Some diverge, and there the only honest output is a distribution across many runs. The full test for telling them apart is in the piece on synthetic users, and it is the single most important thing to internalize before quoting a simulated result to a board.

The second is provenance. Simulated output looks exactly like real output, and that resemblance is the whole point of the technology, which makes it dangerous the moment it leaves the tool that produced it. A simulated response that reaches your warehouse without a label becomes a real customer statement in six weeks, and no downstream consumer of that table will ever know. The controls are unglamorous and worth insisting on: label simulated records at write time rather than at read time, keep the lineage back to the model version and prompt that produced them, and never let a simulated field silently satisfy a schema that real data also populates.

The consent question sits alongside it. If a behavioral simulation is grounded in data from real people, the scope of what those people agreed to matters, and a survey consent is not consent to be modeled indefinitely. Ask a vendor how a person's derived agent is deleted when that person withdraws. Ask what happens to simulations already run from it. The good answers exist. The point is that you have to ask, because nothing in the product surface will raise it for you, and this is a category where the security questions arrive later than the capability.

What to do this week

1. Sort the simulation pitches on your desk into the three families. Physical, training environment, or behavioral. The evaluation criteria are not transferable, and treating a behavioral vendor like a training environment vendor is how teams buy the wrong thing.
2. For each one, name the lever. What decision changes based on this output? If there is only a forecast and no action attached to it, you are buying a report.
3. Write your ground truth loop in two sentences: how you will learn the simulation was wrong, and how long that takes. Weeks is a loop. Years is a hope.
4. Put a date on the first scoring round. Pick five simulated answers you can check against reality inside a quarter, and calendar the check now. A loop nobody has scheduled is a loop that will not run.
5. Before building, ask whether the simulator is harder than your product. If the honest answer is yes and simulation is not what customers pay you for, buy it.
6. Audit your data strategy against Park's test. Do you have access to data nobody else has? If your simulation runs on the same public inputs as everyone else's, it is a feature and not a moat.
7. Put a provenance label on every simulated record at write time, before the first one reaches a shared table.

If you are building the operating system that decides which of these bets your company takes, that is exactly what the AI Operating System for Startups course is for, and the AI for startups guide is the free overview of where simulation sits next to evals, agents, and pricing.

Sources

• The AI Company Simulating the Entire Economy, Simile co-founder and CEO Joon Sung Park on 20VC with Harry Stebbings, the interview this article distills.
• Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Work on Y Combinator, the source for the closed loop simulation, agent plus simulator plus critic, and simulator difficulty material.
• Simile's funding: Announcing Our Series B and TechCrunch. The four co-founders, as named by Park in the interview: Joon Sung Park, Michael Bernstein, Percy Liang, and Lainie Yallen.
• The research the company came out of: Generative Agents: Interactive Simulacra of Human Behavior (the 25 agent Smallville simulation) and Generative Agent Simulations of 1,000 People, which arXiv now serves under a revised title in its latest version.
• Waymo's autonomous mileage and crash comparisons: Waymo Safety Impact.