What is an RL environment?

An RL environment is a simulated slice of a real job that an AI agent can practice in and be scored on. In the shape used to train today's agents it has three parts: a world (the messages, documents, spreadsheets, and files a real project would contain), the apps (working clones of the software a worker would use, reachable by the agent through tools, a command line, or computer use), and the tasks (a prompt paired with a verifier that decides whether the attempt succeeded). The verifier can be a unit test, a rule, or a written rubric. The agent attempts the task many times, the verifier scores every attempt, and reinforcement learning uses those scores to change the model's behavior. The same package with the training turned off is an eval.

What is the difference between an RL environment and an eval?

Mechanically they are nearly the same thing, which is the useful part. Both an RL environment and an eval are a task, a starting state, and a scorer. An eval runs the scorer to tell you how good a model is. An RL environment runs the same scorer thousands of times and feeds the scores back as a training signal. That means an eval suite you already trust is most of an RL environment, and it also means a sloppy scorer is far more dangerous in training than in testing: a rubric that can be gamed will simply teach the model to game it. Build the eval first. If it is accurate enough that you would let it decide a release, it is accurate enough to consider training against.

How much does it cost to build an RL environment?

Individual tasks are priced from $50 to $10,000 depending on complexity. Brendan Foody, co-founder and CEO of the expert data company Mercor, gave that range in a Sequoia talk, along with a legal-environment example at $2,000 per task. The cost base is human time, not compute. A task that takes a domain expert 10 hours at roughly $150 an hour carries about $1,500 of cost before margin. Frontier labs buy at a scale most startups will not: Foody said certain labs purchase around 50,000 tasks a month. The compute for the training run itself is comparatively small. He described a post-training run on 1,800 tasks using roughly $500,000 of compute, which is real money but far less than the authoring cost of the tasks at scale.

Should a startup build its own RL environment?

A startup should build its own RL environment only after the cheaper levers are exhausted, and only where the work is repetitive and valuable enough to justify the authoring cost. Prompting, retrieval, and context engineering are all cheaper and survive model changes, so run those out first. Then check three things. The task shape must be stable, because an environment built for a workflow you are still redesigning is wasted. The base model must be close: Foody's heuristic is that if you run 16 attempts and the model gets none of them right, training is mostly hopeless, whereas a model that fails on the first attempt but succeeds once or twice in 16 learns well. And you need people who can define what a good answer is, since the scoring rubric, not the simulated world, is the part that decides whether any of it works.

RL Environments Explained for Founders

An RL environment is a simulated slice of a real job that an AI agent can practice in and be graded on: the documents and messages a real project would contain, working clones of the apps a worker would use, and tasks paired with verifiers that decide whether the attempt actually succeeded. The model tries the task many times, gets scored every time, and learns the job from the scores.

That is the definition, and most pages that rank for this term stop roughly there, or continue into the machine learning vocabulary of policies and reward functions. The founder question underneath it is different. Your product calls a frontier model. The model is decent at your customer's job and not great at it. You have already spent months on prompts, retrieval, and context, and the gap has stopped closing. RL environments are the machinery for closing it by training instead of prompting, and until recently they existed almost exclusively inside frontier labs.

The clearest recent account of how they are actually built came from Brendan Foody, co-founder and CEO of Mercor, in a talk at Sequoia Capital. Mercor supplies expert-built evaluation and training data to the frontier labs and to application-layer companies, and raised a $350 million Series C in October 2025 at a $10 billion valuation. Foody's talk is worth a founder's attention less for the vendor story than for the anatomy: he takes the thing apart into three pieces, and it turns out only one of them is genuinely hard.

The three parts of an RL environment

Foody splits an environment into worlds, apps, and tasks.

The world is the context a real project would carry. Messages, slides, docs, sheets, email threads, files of realistic size and messiness. Not a clean toy dataset, but the accumulated debris of a job in progress.

The apps are high-fidelity clones of the software the job runs on: Salesforce, ServiceNow, Microsoft 365, Google Workspace. The agent reaches them through tool calls, a command line, or computer use, the same way it would reach the real thing.

The tasks are prompts paired with verifiers. A verifier can be a unit test, or it can be a written rubric, and Foody notes the same package serves either evaluation or training depending on what you do with the scores.

The published legal environment he walked through makes it concrete. Lawyers from large firms write out a scenario from a real matter they worked on, then build the outline for a data room that matches it: the messages, the emails, the files, the file sizes. That data room gets rendered into clones of Google Workspace. The prompt then asks the agent something a junior associate would actually be asked, in his example comparing the maximum total liability of two companies under a specific statute given everything in the data room. The rubric lists the criteria a correct answer would hit.

If that structure sounds familiar, it should. It is an LLM eval with the training loop attached. That is the single most useful thing a founder can take from the anatomy: an eval suite you already trust is most of an RL environment. The difference is that an eval runs the scorer once to tell you how good the model is, while an environment runs it thousands of times and feeds the result back as a training signal.

Which is also why the scorer, not the simulated world, is where the work is.

The verifier is the hard part

The instinct is that building the world is the expensive bit. Cloning Salesforce sounds harder than writing a grading rubric. It is the other way around.

Some domains grade themselves. Math has a clean answer, code has tests that pass or fail, and in those the model can learn from the result without a human anywhere in the loop. Francois Chollet has made the same point about why coding agents ran ahead of everything else: they work in verifiable domains, which is what supplies a reliable reward signal. Most professional work has no such oracle. Foody's line on why is the sharpest sentence in the talk: asking a model to identify where its own slide deck went wrong is unreliable, and "It's as if you would be asking a human to grade their own homework."

So a person writes the rubric, the way a professor writes one for an essay. And that rubric has to cover a solution space, not a solution. Foody describes the problem as needing to understand, for a task like building a deck, what the ten different good paths through it look like and what the dozens of possible mistakes are, then writing something comprehensive enough to catch all of it.

Get that wrong and you do not get a slightly worse model. You get reward hacking: the model finds the cheapest behavior that satisfies your rubric and optimizes into it, which is a much more expensive failure than a bad eval score, because it silently trains the wrong thing.

Mercor's check on this is a process Foody calls trajectory analysis. Roll out attempts from the model you are trying to improve, score them all with the rubric, then use a mix of automated quality control and human review to confirm the scores hold up. The bar he states is that the rubric's ranking should come out "the same as if we were to just have human stack rank those 100 trajectories."

He also gives the cleanest definition of data quality in this space. When customers say quality, he says, they mean two things: "First is realism and secondly is accuracy of verifiers." Realism is whether the environment reflects the actual distribution of a real lawyer's or banker's work. Accuracy of verifiers is whether the score means what you think it means. Founders reliably over-invest in the first and under-invest in the second, because realism is visible in a demo and verifier accuracy is not.

What an RL environment costs

Tasks are priced individually, and Foody was unusually direct about the numbers, which is the part that will settle the build-or-skip decision for most teams. A common shape is a customer saying they like a particular kind of environment and will pay around $2,000 per task to scale it up. The full range he gave is wide: "we have tasks that range from $50 to $10,000." Some take a human a few hours. Some take up to a month.

The cost base underneath that is human time. His worked example: a task that takes 10 hours of expert time at $150 an hour carries roughly $1,500 of cost before any margin. That number is the one to carry around, because it does not get cheaper with scale in the way infrastructure does. It is also why the volume metric Foody showed is expert time rather than tasks or compute: he put Mercor's throughput at roughly 2.5 million expert hours in a single quarter.

Frontier labs buy at a volume no startup will match. Foody said certain labs purchase on the order of 50,000 tasks a month.

Against all that, the compute is almost a rounding error. He described a post-training run on 1,800 tasks in the APEX-Agents shape, using about $500,000 of compute. (The public APEX-Agents release is smaller: 480 tasks spread across 33 simulated worlds.) On corporate law the model went from 4.7 percent to 26.6 percent. The result he found more interesting was that the gains generalized: the model also improved on Mercor's non-agentic APEX benchmark, which he notes does not have these data rooms, and on OpenAI's GDPval, a benchmark of 1,320 tasks drawn from 44 occupations across the top 9 industries by contribution to US GDP.

Read that as a founder and the shape of the bet is clear. The compute is affordable. The task authoring is not, unless the work being taught is repetitive and valuable enough that a few thousand examples cover a job your customers pay for over and over.

Why application-layer founders started caring

For most of the last few years this technology sat inside the labs. What changed is that open weight base models got good enough to post-train, so the pipeline that produced frontier capability became available one layer out.

Foody's framing of the strategic consequence is worth quoting because it is the version a founder should argue about at their own whiteboard: application-layer companies are realizing that "the three core pillars of their AI strategy are their compute, their algorithms or researchers, and the data sets they build. And data's often the most differentiating factor."

That proprietary data is the real bottleneck is not a new argument. What the environment shape adds is a specific kind of data asset, and it is the unusual one: not logs you already have, not a corpus you can license, but a set of tasks paired with rubrics that encode what excellent work looks like in your vertical. It is priced in expert hours rather than storage. It cannot be scraped, because it does not exist anywhere until someone who does the job sits down and writes it. That is a different defensibility profile from a data lake, and a more durable one.

The existence proof he points to is Cursor, an application-layer company that trained its own agentic coding model rather than only wrapping someone else's. Cursor has been public about the method: Composer was built with large-scale reinforcement learning in its own agent harness, and the Composer 2 technical report describes continued pretraining on an existing base model followed by large-scale RL. Foody expects dozens of comparable examples over the next year, with the owned model as the source of the moat.

Treat that as a claim from an interested party, because it is one: Mercor sells the input. But the mechanism is real, and it is the honest answer to founders who have concluded that a wrapper has no moat. The moat is not the wrapper. It is the training set only you can produce, and the customer relationships that let you produce it.

Where synthetic data helps and where it stops

Founders hear the cost numbers above and immediately ask whether models can just build the environments themselves. Foody's answer has a useful shape: mostly yes for the world, mostly no for the task.

He starts by pointing out that the confusion begins with the term. Reinforcement learning with verifiable rewards is itself a bet on synthetic data: instead of humans writing thousands of correct answers, the model generates its own attempts and learns from how they score. The synthetic generation is already the core of the method.

Models also do heavy lifting on populating environments. The lawyer building out a data room should be using a model to produce it, in exactly the way a lawyer writing a memo should. Foody makes the comparison directly, and it applies to your team too: the expert's job is to be the source of realism and judgment, not to type.

The limit is the task and its verifier. His reasoning here is the part worth internalizing: "you need humans almost definitionally to measure what is beyond the frontier of the model capabilities." You cannot ask a model to invent the environment and then tell you which of its own outputs are good, because the signal is too noisy to train on. A copilot working alongside an expert makes rubric writing considerably faster. A model writing the rubric alone gets roughly half the criteria right, and that much noise is unusable as a training signal.

For founders this maps to a clean division of labor. Generate the world, the documents, the fixtures, the variations. Have a human define the task and what counts as done. If you are already using simulated users to pressure-test a product, this is the same boundary in a different application: synthesis is fine for the setting, and judgment about quality still has to come from someone who has the job.

Two gaps in today's RL environments

Two things are missing from essentially every environment on the market today, and both are more interesting to a startup than to a lab, because gaps are where a small team gets to matter.

Long horizon work. In Foody's words, "right now agents mostly aren't trained to do things that are over 10 hours," and the next shift is building tasks that would take a human 100 or even 1,000 hours. Every environment you can buy today teaches short work. If your product's value is in a process that takes a person two weeks, nobody has an environment for it and you would be building from scratch.

Working with other people. This is the gap worth a founder's time. Foody asks people what share of their job requires interacting with other humans, and the usual answer lands around 60 to 70 percent. Then he asks what share of evals measure that, and the answer is closer to 1 percent, with tau-bench, which tests agents against simulated users under domain policies, as one of the few that touch it at all. As he puts it, "there's this giant realism gap associated with how you actually measure how well agents engage in social interaction."

Almost every real workflow a founder is trying to automate is full of handoffs: the clarifying question, the pushback from a colleague, the approval that arrives with conditions attached. Agents are trained on almost none of it. If your product lives in those handoffs, the frontier labs are not currently training for your problem, which is both the risk and the opening.

Some domains verify themselves, and security is one

Where success is programmatically checkable, the grading cost largely disappears and this whole cost structure inverts. Security is the clearest case, and a founder should check whether they are in that category before spending anything.

Asked about cyber defense and incident response, Foody's view is that security is one of the domains where "you can have an attacker and a defender agent," so you do not necessarily always need a human writing the verifier. Did the attacker get access or not is a fact, not a judgment. Humans are still needed to architect realistic environments with enough diversity, but the expensive part, grading, becomes considerably less human-intensive.

That is the general rule, not a security-specific one. Where success is programmatically checkable, you may need no rubric author at all, and the economics change completely. Code that compiles and passes tests, a migration that reconciles, a trade that settles, a claim that matches the policy: all of these can score themselves. Where success is a matter of professional judgment, you are buying expert hours whether you like it or not.

Worth adding the security caveat from the other direction, since the same mechanism cuts both ways. An environment with a self-scoring reward is the ideal training ground for offensive capability too, which is part of why offensive capability has been compounding quickly. In an a16z conversation on AI learning to hack, the security founders on the panel make the same observation from the attacker's side: the reward function is unusually well defined, because either the agent got access to the data or it did not. If you are building in security, plan for defensive capability to be tested against agents that have themselves been trained this way.

Should your startup build an RL environment?

Most startups reading this should not, yet. The honest sequence:

Exhaust the cheap levers first. Prompting, retrieval, and context engineering are cheaper, faster to change, and survive the next model release, which is the ordering the AI for startups guide lays out across the whole stack. Training does none of those. The RAG versus fine-tuning decision sits below this one and should be settled first.
Check that the task shape is stable. An environment encodes a workflow. If you are still redesigning that workflow monthly, you will throw the environment away.
Check the base model is in range. Foody's heuristic is worth stealing wholesale: look at the gap between one attempt and 16 attempts. If the model gets all 16 wrong, training is mostly hopeless. If it fails the first attempt but gets it right once or twice in 16, it learns very effectively from that. Run this before you spend anything, because it costs an afternoon and answers the question.
Confirm you have graders. Not engineers. The people in your company or your customers' who can say what a good output looks like and defend it. If that person does not exist and cannot be hired, stop.
Do the arithmetic honestly. A few thousand tasks at expert rates is the real number, and the compute is the small part. If the workflow you are teaching is not one your customers pay for repeatedly, the arithmetic does not close.

If all five hold, the sensible starting move is not to buy an environment. It is to build the eval suite for that one workflow to a standard where you would let it gate a release. That artifact is valuable on its own, it is the thing you would have to build anyway, and it tells you within a few weeks whether your organization can define quality precisely enough for any of this to work. Harrison Chase of LangChain makes the adjacent point about agent harnesses: evals and observability are what let you set up a data flywheel and compound the intelligence as you use the agent. The flywheel is the same artifact, one turn earlier.

What to do this week

Pick the single workflow where your product's quality gap costs you the most revenue, and write down what a correct output looks like in criteria a stranger could apply.
Run the 16-attempt test on it: take 20 real cases, run each through your current model 16 times, and count how many it ever gets right. That number tells you whether training is even available to you.
Audit your existing evals against the two-part definition: realism (do these cases match what customers actually send) and verifier accuracy (would a human rank the outputs the same way your scorer does).
Ask whether your domain self-verifies. If success is programmatically checkable, skip the rubric problem entirely and go straight to building tasks.
Name the grader. Identify the specific person who will define quality, and put their time in the plan rather than assuming an engineer can substitute.
Price the honest version: number of tasks you would need, times expert hours per task, times a real hourly rate. Compare that to what the workflow earns you before you take it to anyone.

Owning your own intelligence is becoming a legitimate strategy for application-layer companies rather than a lab-only luxury. It is also a substantially more expensive commitment than the pitch decks suggest, and the expensive part is the boring part: someone in your company writing down, precisely, what good work looks like. That skill is the prerequisite for every version of this, from a first eval to a full post-training run.

If you want the wider operating system this fits into, from evals and context to agents and pricing, that is what we teach in AI Operating System for Startups, and the AI for startups guide is the free overview.

Sources

RL Environments Explained: How AI Agents Learn Real-World Work, Brendan Foody, Mercor on Sequoia Capital's channel: the talk this article distills, including the three-part anatomy, the pricing figures, the trajectory analysis process, and the long-horizon and multi-person gaps.
Background on Mercor and its funding: the company's own Series C announcement and TechCrunch's report on the round, which puts it at $350 million at a $10 billion valuation. Foody's profile: LinkedIn and Forbes.
Benchmarks referenced: Mercor's APEX-Agents leaderboard and its paper, the earlier non-agentic APEX paper, OpenAI's GDPval, and tau-bench from Sierra.
Cursor's own write-ups on training its model with reinforcement learning: Composer and the Composer 2 technical report.
Cross-source evidence from our video knowledge base: Harrison Chase of LangChain on agent harnesses and evals (Sequoia), Francois Chollet on why coding agents work in verifiable domains (Y Combinator), and a16z on AI learning to hack, with Dylan Ayrey of Truffle Security and Feross Aboukhadijeh of Socket, for the self-verifying reward signal in security.