Continual learning AI: what to build first

Continual learning AI is the idea that your product should get better the more people use it, by learning from real interactions instead of waiting for the next base model to ship. It is the most credible answer to a problem every AI company hits after launch: the models keep improving on a schedule you do not control, and your product does not improve at all.

At a Sequoia Capital event on owning your own intelligence, Arjun Karanam, a co-founder of Trajectory, walked through what has to exist before continual learning is even possible. Almost none of it is machine learning. It is logging, feedback design, and how you wrote your tool calls. That is the useful part for founders, because it is work you can start this quarter without hiring a research team.

What continual learning AI actually means

Start with the definition, because the term gets used loosely. Continual learning is a system that keeps improving from its own production interactions, rather than only from a training run that finished before your users existed.

Karanam is candid that the field agrees on little beyond that: "if you ask like six researchers what continual learning is, you're going to get like seven answers", as he put it. The purist reading is that only the weights count, and they should adapt in real time from a single example. His reading is wider: "the intelligence that your product is run off of is a system", and continual learning is whatever optimizes across that system, deciding which part to update for a given piece of information. His analogy is storage. We stopped deciding whether something belongs in RAM or on disk because that got abstracted away. Models, harnesses, and context are still a decision founders make by hand with very little to go on.

That framing decides who owns the work. If continual learning means retraining weights, it is a research problem you probably cannot staff. If it means improving a system made of a model, a harness, and context, most of it is ordinary engineering. Harrison Chase, co-founder and CEO of LangChain, described the same anatomy at the same event: "There's a harness that orchestrates a model and some context", and owning your intelligence means owning all three parts.

The experience gap, not the intelligence gap

Karanam's premise is that models are improving fast on one axis, IQ, while the axis that matters just as much, experience, is not what the model release cycle addresses. Talking to a model, he says, "it's their first day on the job". Hand a brilliant generalist a specialist role and, in his phrasing, "day one at an accounting firm is probably not the best accountant there". Give the same person a few years, or in his telling maybe even a few days, and the picture changes completely. That second axis is experience, and he calls the space between them the experience gap.

The reason it stays open is mundane. Your agents are generating the raw material for closing it right now, and you are deleting it. Every run produces a record of real work, and people act on that work, accept it, correct it, or throw it away. Almost all of that goes in the bin. The payoff for keeping it is immediate (faster, better, cheaper models) and, further out, "systems that compound with use".

This is where a founder should get slightly suspicious, so be clear on what is claimed. Nobody is saying a static frontier model is bad. The claim is that a static frontier model plus your accumulated production experience beats the same model without it, and that the second half is the part a competitor cannot copy. It is the same argument behind building a competitive moat with AI: the model is a commodity you rent, and the specific thing you learned from your own users is not.

Start with traces, including the ones you throw away

The first prerequisite is traceability, and Karanam's complaint about how most teams do it is specific: "what a lot of companies do is they trace the main action that is happening, but they throw away tool calls or sub agents that are made". A run where you kept the final answer and dropped the twelve steps that produced it is not something you can learn from. You know the outcome and nothing about the path.

This is a strictly larger ask than debugging. If you already instrument your agents, you are most of the way there, and the change is retention policy plus completeness rather than new infrastructure. Our post on LLM observability covers the instrumentation itself; the point here is that the shape of what you keep decides what you can ever train on. Chase made the same connection at that event: "Evals and observability really let you set up this data flywheel".

Two practical consequences. Sub-agent and tool-call spans need to be captured with their inputs and outputs, not just their names and durations. And traces need to survive long enough to be useful, which is a storage bill and a data retention decision your legal team gets a say in, so make it deliberately rather than by default.

Thumbs up is noise. Corrections are the signal

The second prerequisite is the one most teams get wrong, and it is a product decision rather than an infrastructure one. The obvious move is to put a thumbs up and thumbs down on every output. Karanam's verdict on that: "it's incredibly noisy".

His example is one every engineer recognizes. With a coding agent "you kind of just like accept everything that the agent does", and it is only several commits later that you discover it broke something and go back to undo it. The rating at the moment of output is close to meaningless, because the user does not know yet.

What carries real information is corrective behavior: "the edits, the undos and the retries". Those need to be both elicited by the interface and captured by the system, which is a design brief, not a logging ticket. If your product makes it easy to silently work around the agent in another tab, you have thrown away the signal.

He also separates feedback into two kinds, and the difference decides what you can do with it. A user who abandons the session or tells the agent it is bad has told you something is wrong and nothing else. In his words, "Those are cases where you know something went wrong, but not what right looks like". A correction, or a retry that lands, is different: now you have the wrong answer and the right one side by side, and you can be confident about the reward you assign. The first kind lets you penalize a behavior. Only the second kind tells you what right looks like.

The founder version of this: if you are building feedback into your product this quarter, build the correction path before you build the rating widget.

Your evals should come from your own traffic

The third prerequisite is evals, and the standard here is stricter than most teams apply. The ideal Karanam describes is that the product your user uses, the thing your evals run against, and the thing you train on are all the same. Every gap between them is a place where your measurements stop describing your product.

Three things follow. Evals should be drawn from real traffic, both what people do today and what they ask for at the frontier of what your product cannot yet do. Every task should be replayable, so you can re-run what a real user actually did, which he is honest about being a hard infrastructure problem. And grading should happen through the shipped harness, "grading the real harnesses that people use in production", not a stripped-down variant that is easier to script.

If you have no eval suite yet, that is the thing to fix before any of this, and LLM evals for founders is the place to start. A trusted eval suite is also most of the way to a training loop, which is the connection we drew in RL environments: the expensive part of a training environment is the verifier, and an eval is a verifier.

Build the harness as primitives, not as guardrails

The fourth prerequisite is that a lot of production harnesses were designed for models that no longer exist. Their main job was to stop the agent from doing something stupid, which was the right design when outputs were malformed at random. Karanam's read on the current moment: "we're very much in a let the agents cook world".

So the harness he argues for is a set of primitives rather than a set of enforced flows. Expose your primitives, the search tools and the private information, and let the agent orchestrate them. The design target he gives is concrete and worth writing on a wall: "every single thing that I can do on your UI, your agent can do via tool call as well".

Then there is the smallest recommendation in the talk and probably the cheapest thing on this list to fix. Make your tool responses informative. A write tool that returns nothing but a success flag tells the agent, and any future training run, almost nothing, or as Karanam puts it, "there's like no signal to learn off of". Return what was written, what was read, what changed. It costs a few tokens and it converts a dead step into a learnable one.

There is a real counterargument here and it deserves stating. Logan Kilpatrick of Google DeepMind, speaking on Sequoia's Training Data podcast, argues that what the industry still calls a model has quietly stopped being one, because tool calling and the scaffolding around it keep getting absorbed into the model itself. We took that argument apart in vertical AI agents. If it holds, harness work you do today has a shelf life. The reconciliation is that primitives and honest tool responses are exactly the parts that survive being eaten, because they describe your product rather than compensate for the model. Enforced flows are the part that gets absorbed. We went further into that tension in building for the next AI model.

Weights or context: the routing decision

Once you are capturing usable signal, you face the question that makes continual learning a system problem instead of a training problem: for any given thing you learned, what do you update?

Karanam's examples make the split legible. A fact, such as a company having been delisted, should not go into the weights. "You probably don't want to train that knowledge into the model", he says, because "It's probably context that should be available to the harness". A tool that repeatedly fails when the agent calls it a certain way is the opposite case: "That's probably relevant to everybody", so the model should learn to use that tool properly. A single user who never wants sub-agents used is context again, and often context scoped per organization or per customer rather than globally.

Read that as a hierarchy by who the information is true for. Globally true goes to the weights. True for one org goes to that org's context. True for one user goes to that user's context. That is also the cleanest way to see how this relates to AI memory: memory is the context half of the same problem, the part that persists across runs without changing the model, and context engineering is how you decide what the agent sees on any given run. If you are still deciding whether to customize a model at all, RAG vs fine-tuning is the earlier fork in this road.

Continual learning without training on customer data

Learning from real interactions collides directly with the contracts you signed to get those interactions. Most enterprise agreements say you will not train on customer data, and asked about that at the event, Karanam did not wave it away. He worked on the problem at Apple before founding Trajectory.

The approach he described is to avoid the direct path: "you're not actually training on customer data, you're instead sampling distributions from your customer data and then synthetically generating your own", then comparing the distributions to check the synthetic data is on distribution. You learn the shape of the work without training on the records.

As a CISSP, this is the part I would design first rather than retrofit, and I would be specific about a few things:

• Write down what each contract permits before you build the loop. A learning pipeline built on data you were never allowed to train on is a pipeline you rip out later, usually during a security review, usually at the worst time.
• Traces are now a sensitive data store. Full traces contain everything the agent saw, which is often more than any single log line in your product. Scope access to them, log the access, and set a retention period on purpose.
• Keep the tenant boundary in the design, not the query. Per-customer context that improves one org's agent must not leak into another's. That is an isolation property, and it should not depend on someone remembering a filter.
• Version what you deploy. If a model or a context set changed and behavior changed with it, you need to know which one and be able to roll it back. That is the same discipline as any other production change.

Is continual learning worth it for you yet?

Honest answer for most startups: not this quarter, and the prerequisites are worth doing anyway.

Nothing on this list requires you to train a model. Full traces, a correction path in the UI, evals drawn from real traffic, tool responses that say what happened, and a clear rule about what belongs in weights versus context are all things that make an ordinary AI product better on their own. They also happen to be the entry fee for continual learning if you decide to pay it later. Karanam says that when his team works with a company, the first thing they often do is audit these areas plus the model layer, and that for the most part there is a lot of work to be done. That is a fair description of nearly every AI product in production today.

The case for doing it eventually is the most interesting thing in the talk, and it is about demand rather than cost. Users calibrate what they ask for against what they think your product can do. They try something at the edge, "they'll see it fail, or they'll see it do something wrong, and they'll retreat back", and they stop asking until something changes. What Karanam says he sees with customers is the reverse loop: "users ask for things, the model can barely do it, but then during training it learns how to do it and then the user can now do this thing they couldn't do before".

That is a growth mechanism, not an efficiency one. Every capability your system learns expands what your users believe they can ask for, which produces harder requests, which produce more signal. Cheaper inference is the near-term prize. The frontier of what people even attempt with your product is the real one.

The move that unlocks the model half of it, for what it is worth, is getting comfortable running on open weights, since you cannot own a model you only rent. Karanam is straight about the cost: it is not a drop-in swap, and security, safety, and access provisioning all show up as work. Chase's version of the same point is that owning your intelligence means owning the model, the harness, and the context, and the ability to switch models is a large part of the value. Our take on picking that starting point is in best LLM for founders, and the wider strategic picture sits in the AI for startups pillar.

What to do this week

1. Pull one real production trace and check whether it contains every tool call and sub-agent step, with inputs and outputs. If it does not, that is your first ticket.
2. Grep your tool implementations for responses that return only a success flag. Make them return what was actually read, written, or changed.
3. Find where users correct your agent today. If the only feedback you capture is a rating, add capture for the edit, the undo, and the retry instead.
4. Build ten evals from last week's real traffic, including three requests your product currently fails, and run them through the shipped harness rather than a test rig.
5. Write one page listing what you learned from users in the last month and which bucket each item belongs in: globally true goes to the model, org-specific and user-specific go to context.
6. Read your two largest customer contracts for what they say about training on their data, and decide your synthetic or aggregate approach before you write any of this code.

Deciding what your company learns from its own work, and where that learning is allowed to live, is exactly the kind of operating decision we work through in AI Operating System for Startups. For the process version of the same idea applied to your company rather than your product, see how to build a self-improving company with AI.

Sources

• Continual Learning: How AI Agents Get Better With Every Use, the Sequoia Capital talk by Trajectory co-founder Arjun Karanam that this article distills, including the Q&A on customer data and on which tasks benefit most.
• When to Build Your Own Agent Harness, Harrison Chase of LangChain at the same event, for the model plus harness plus context anatomy and the evals-and-observability flywheel.
• Why the Model Eats the Harness, Logan Kilpatrick of Google DeepMind, for the counterargument that scaffolding keeps getting absorbed into the model.
• Background on Trajectory and its founders: WIRED on the launch and the team's Google DeepMind, Apple, OpenAI, and Meta Superintelligence Labs backgrounds, plus the company's own site, Trajectory.
• Profiles: Arjun Karanam, Ronak Malde, and Michael Elabd, the three co-founders.