Robotics Data: Build, Buy, or Collect It

Buy the tooling, own the corpus. That is the conclusion, and the rest of this is how to get to it: where robotics data comes from, what each supply line actually costs, and which part of the pipeline your company has to own to end up with a moat rather than a vendor bill.

The question lands harder in robotics than in software because there is no equivalent of the open web. A language model team starts with a corpus that already exists. A robotics team has to manufacture its supply one recorded action at a time, which is the constraint that defines physical AI. It sets the hiring plan, the burn rate, and who owns the asset at the end.

On Y Combinator's Root Access, the co-founders of Encord described the layer they are building underneath that problem, shortly after announcing a $60 million Series C led by Wellington Management. Ulrik Stig Hansen had been doing a computer science master's at Imperial College London in the late 2010s; Eric Landau was working at a high-frequency trading firm. Their company sells this pipeline, which makes the interview an unusually specific description of it, and worth reading from the other side of the table.

What is data in robotics

Robotics data is the recorded trace of a robot or a human demonstrator doing a physical task, plus everything needed to make that trace trainable. One recording carries synchronized video from several cameras, depth and force readings, joint positions and velocities, and the action commands issued at each timestep. Around it sit the labels, the task description, and the evaluation results that tell you whether a policy trained on the trace actually works.

The property that matters is not volume, it is shape. Encord's co-founders put the contrast plainly, describing how much harder multimodal data is to interact with at scale: "text is low bandwidth, it's kind of easy to see and visualize, but video, sensor data, audio data, especially when you're working together in groups, is quite difficult". A text corpus is a list of strings you can grep. A robotics corpus is a pile of time-aligned streams where a one-second drift between a camera and a joint encoder teaches the model something false and nothing in the file looks wrong.

That is also why the job description differs. Hansen's account of why they started the company names the asymmetry directly: "out of the three ingredients for AI development, models, compute, and data, the one that took the longest, we spent a bunch of time wrangling the data, spent a bunch of time cleaning the data, felt like the most defensible." Wrangling and cleaning are not a preliminary step before the real work. In robotics they are most of the work, and that is precisely why the data layer is where the defensibility sits.

Where robotics data comes from

There are four supply lines, and a serious company runs more than one.

Teleoperation and human demonstration. A person drives the robot, or wears a capture rig, and the system records what they did. This is still the backbone of manipulation data, and it is labor, which means a per-hour cost that does not fall the way compute does. The shift away from pure teleoperation toward passive human video is the live argument in the field, and the wearable and egocentric-video routes are worth understanding before you budget for another year of teleop.

Purpose-built collection environments. Rather than hope demonstrations arrive, you stage rooms that look like the target workplace and repeat the task under controlled variation. Encord has opened an R&D facility in the Bay Area to do exactly this, building the environments robotics companies need and capturing the recordings in them. As one of the co-founders framed the division of labor, "we don't build the robots, but we work with the robotics companies". Collection is becoming a service you can rent rather than a team you have to hire.

Deployment. Once a robot is doing paid work, it returns the edge cases no staged environment produces. This is the most valuable supply line and the slowest to open, because it requires a customer first. It is also the one that compounds.

Simulation. Unlimited and cheap, and it reliably teaches a policy things that are not true about the world. Useful as volume underneath real recordings, dangerous as a substitute for them.

It is worth knowing the scale of the open baseline before you budget. DROID, one of the larger openly released in-the-wild manipulation datasets, holds roughly 76,000 demonstration trajectories, about 350 hours of interaction, collected across 564 scenes. That is a genuine gift to a small team and it is also a rounding error next to the text corpus behind any language model. Open data gets you a pretrained base. It does not get you a product.

The supply problem is broader than robot arms. A Y Combinator request for startups on physical world data makes the point that the physical world is still read through "sparse data from remote sensors" that were "designed for humans, not AI", while the largest industries "rely on limited data and intuition-based models". Falling sensor costs mean "dense physical world data collection is now feasible", and the companies cited are not robotics companies in the humanoid sense: one flies autonomous weather balloons to improve forecasting, another sends robots into hard-to-reach places and builds predictive models from what they bring back. If you are looking for an underpriced entry point, the collection layer for a specific industry is a more tractable business than the robot.

The quality bar is set by what a mistake costs

Physical systems carry a far lower error tolerance than software, and for the data layer that cashes out in a single line, which is how Encord's co-founders describe their own job: "getting the right data in, keeping the wrong data out."

Take it literally, because the error budget is unforgiving. As one of the co-founders put it, "a model is only as good as the data it's trained on", and "even like the slightest errors in the data set that can influence and impact like how the model actually works in the real world". The practical consequence for a founder is that the labeling specification is a safety document and should be reviewed like one. Two implications follow, and neither is expensive if you do it early:

• Version your labeling spec like code. When the definition of a successful grasp changes in month seven, you need to know which recordings were labeled under which definition. Teams that skip this discover a silent quality cliff and cannot locate it.
• Build the evaluation set before the training set. A held-out set of recordings that represents the failures you actually fear is the only thing that turns a vague sense the model improved into a number. This is the same discipline as LLM evals, with a higher cost of being wrong.

Humans stay in the loop, and they are a line item

The pitch deck says autonomy. The budget says people, for a while.

Encord's co-founders were direct that human involvement is not a transitional embarrassment: "you need humans at the frontier." Their description of where that frontier currently sits in physical AI is worth sitting with, because it is tasks like folding laundry and emptying a dishwasher, while in language models the same people are working on reasoning problems that are, in their words, "much more difficult". Physical AI is early in a way the funding environment does not always advertise.

There is a second, permanent role. Once robots are deployed, the founders noted, "once the robots are in production, they also need things like exception handling, they need help with like observability". Somebody has to be on the other end when the robot encounters something it has never seen, and that person generates the most valuable training data your company will ever have. One of the co-founders put the intent as a preference rather than a concession: "we want to be the managers and supervisors of these AI systems."

For a founder this collapses into one planning rule. Staff the exception desk from day one and treat it as a data acquisition function rather than a support cost, because that is what it is. The economics of running a deployment this way, and the intervention rate to manage, are worked through in what actually changed for a robotics startup.

The market is not hypothetical any more. Home robots that fold laundry are taking consumer orders at around $8,000, with first deliveries due this fall and human teleoperators still in the loop for parts of the task. Whoever staffs that loop is about to become a permanent role, so design it as one.

Build or buy the robotics data layer

Buy the tooling and own the corpus. Building an indexing, curation, annotation, and evaluation stack is a multi-year project that differentiates nothing, so the real decision is not an architecture question, it is what the vendor contract says about who owns the recordings.

The case for buying is strong and Encord makes it well. Indexing, curation, annotation, evaluation, and the pipelines connecting them are a multi-year engineering project that differentiates nothing, and running it at scale is, in their words, "operationally quite difficult to do, especially to do at scale". Every month your engineers spend on a labeling UI is a month they are not spending on the thing customers pay for. Buy the tooling.

The case for caution is in the same interview, stated as a feature. Describing why the platform compounds, one of the co-founders explained that customers "index all the data on our platform, they curate all the data on our platform, they annotate the data on our platform, and they embed their model to do the pre-labeling", and that the payoff is speed: "the faster you can get this like flywheel turning, the faster you can get your model to production". That is true, and it is also a description of a company whose stated ambition is that "we want to work with every single physical AI company in the world", explicitly on the Stripe model of sitting under every transaction.

Read the two halves together and the founder's move is obvious: the flywheel is real, and the question is whose flywheel it is. The answer is not to build one yourself. It is to buy the pipeline and keep the corpus, which means settling five things in writing before you sign:

1. Ownership. You own the raw recordings and every derived label and annotation, without qualification.
2. Export. You can pull all of it, in a documented format, on demand and without a professional services engagement.
3. Training rights. The vendor may not use your data to train shared, third-party, or foundation models. This is the clause that gets negotiated away quietly.
4. Termination. A defined handover window with the data delivered in a usable state, priced in the original agreement rather than at the moment you want to leave.
5. Data handling. A robotics corpus collected on a customer's site is a record of that site, with identifiable footage of the people who work there. Your storage jurisdiction, an enforced retention period, access to raw recordings restricted and logged separately from access to derived labels, and a deletion obligation that survives termination. When the vendor holds the data, these are the vendor's answers to give, in the contract.

Vendors who intend to keep your business will sign all five. A vendor who will not is telling you which asset they think they are buying. That is the same data moat logic that applies to any AI company, with one difference: in robotics you paid a per-hour human cost to create every row, so giving it away is a cash transfer and not just a strategic slip.

That fifth clause is also the one that closes enterprise deals. Security review will ask every one of those questions, and a robot working inside someone else's building puts them on the table early. Treating the data layer as a governed asset from the start is the posture that makes the rest of an AI operating system work, because the controls are what let you move faster through procurement, not slower.

One more thing from the same interview, because it is where robotics companies actually lose time. Asked what he wished he had known earlier, one of the co-founders said the surprise was "how much it costs to not make a decision": "there's a big opportunity cost of just indecision, and you're constantly paying interest on decisions that you don't make." Build or buy is exactly the call that sits open for another quarter while collection runs. Make it, write it down, and let the market correct you. Or as one of them put it about direction generally, "don't be afraid to let the market tell you what your company is".

What to do this week

1. Write down your four supply lines. For teleoperation, staged environments, deployment, and simulation, name where each one stands today and what one hour of usable data costs you. Most teams cannot answer the cost question, and it is the number that governs the runway.
2. Read the data clauses in your vendor agreement. Check ownership, export, training rights, termination, and data handling. If any is missing or vague, open the redline this week rather than at renewal.
3. Build the evaluation set before you scale collection. Fifty recordings of the failures you actually fear beats ten thousand recordings of the happy path.
4. Version the labeling specification. Put it in the repository with a change history and tag every batch with the version it was labeled under.
5. Staff the exception desk and instrument it. Whoever covers the robot's mistakes should be logging them into the training pipeline automatically, not into a support ticket queue.
6. Do a ten-minute classification exercise. List what your recordings would reveal about a customer's site if they leaked, then decide retention and access on that basis instead of on storage cost.

Getting the data layer right is one piece of running a company where AI is the operating system rather than a feature. That is what we teach in AI Operating System for Startups.

Sources

• The Data Layer for the Robot Economy (YC Root Access), the interview this article is built on, with Encord co-founders Eric Landau and Ulrik Stig Hansen on multimodal data, collection environments, humans in the loop, and the platform flywheel.
• Data for the Real World (Y Combinator), on sensors designed for humans rather than AI and the falling cost of dense physical world data collection.
• Encord company and funding details: Encord's $60 million Series C announcement, SiliconANGLE, and the Y Combinator company profile. Co-founder profiles: Eric Landau and Ulrik Stig Hansen.
• Open dataset figures (trajectories, hours, and scenes) from the DROID project page; the paper is the underlying reference. The cross-embodiment collection referenced alongside it is Open X-Embodiment.
• Home robot pricing, pre-order status, and the role of human teleoperators: The Verge and Business Insider.