Should a startup use RAG or fine-tuning first?

Start with RAG in almost every case. Retrieval augmented generation grounds a model in your own data at request time without touching the model's weights, so it is cheaper to build, cheaper to update, and it keeps working when a better base model ships. Fine-tuning changes the model itself, which means a labeled dataset, a training run, and a bill before you learn anything, plus a result that goes stale the moment a stronger model comes out. The provider guidance from OpenAI, IBM, and AWS all points the same way: reach for prompting and RAG first, and only fine-tune once a specific behavior or cost problem demands it.

What is the difference between RAG and fine-tuning?

RAG is for knowledge and fine-tuning is for behavior. RAG retrieves relevant chunks from your documents, tickets, or database and puts them in the model's context so the answer is grounded in your information, without retraining anything. Fine-tuning further trains the model's weights on thousands of examples so it internalizes a tone, a strict output format, or a narrow reasoning pattern. If your problem is that the model does not know your facts, that is a retrieval problem. If the model knows enough but will not answer the way you need, that is a candidate for fine-tuning.

When is fine-tuning actually worth it for a small team?

Fine-tuning earns its cost in three cases, and none of them is adding knowledge. First, when you need behavior that prompting cannot hold, like a strict output format or a consistent house voice the model keeps drifting from. Second, when cost and latency at scale are the goal: a fine-tuned smaller model can match a big model's quality on one narrow, high-volume task at lower cost. Third, when the instructions to guide the model no longer fit sensibly in a prompt. If you are reaching for fine-tuning to teach the model facts, use RAG instead.

Can you use RAG and fine-tuning together?

Yes, and mature systems often do. A common pattern is to fine-tune a model for format and voice while using RAG to supply current facts, so the model behaves consistently and stays grounded in up-to-date information. The order still matters: start with RAG because it solves the most common problem (the model not knowing your data) at the lowest cost, then add fine-tuning only when a behavior or cost target you can name is not met. Stacking fine-tuning on top of a working RAG setup is very different from fine-tuning first and hoping it also fixes knowledge, which it will not.

RAG vs Fine-Tuning: The Cheaper First Move

Every founder building on top of an LLM hits the same fork. Your model is good but generic. It does not know your product, your policies, or your customers, and sometimes it answers in the wrong format or the wrong voice. So you reach for the two levers everyone talks about: retrieval augmented generation (RAG) and fine-tuning. The internet frames them as rivals. For most startups they are not, and treating them as an either/or is how teams burn months and a lot of money on the wrong one.

The honest answer, and the one infrastructure teams at OpenAI, IBM, and AWS now give, is that you should almost always start with RAG. Fine-tuning is the expensive move you graduate to for narrow reasons, not the default. This piece lays out what each one actually does, why fine-tuning is the costlier first bet, when it genuinely earns its price, and a simple way to decide.

RAG and fine-tuning solve different problems

Start by separating the two, because half the confusion comes from treating them as two flavors of the same thing.

RAG leaves the model's weights alone. At request time it retrieves relevant chunks from your own data (docs, tickets, a product catalog, a policy database) and puts them in the model's context so the answer is grounded in your information. You are giving the model an open book, not teaching it a new skill.

Fine-tuning changes the model itself. You take thousands of examples and further train the weights so the model internalizes a behavior: a house tone, a strict output format, a narrow reasoning pattern. You are not adding fresh facts so much as reshaping how the model responds.

The clean rule the major providers converge on is that RAG is for knowledge and fine-tuning is for behavior. If the problem is that the model does not know your stuff, that is a retrieval problem. If the problem is that the model knows enough but will not answer the way you need, that is a candidate for fine-tuning. Naming which one you actually have is most of the decision.

Why fine-tuning is the expensive first move

Fine-tuning looks like the serious, grown-up option, so founders over-reach for it. Then they meet the two costs.

The first is the obvious one: money and effort up front. You need a labeled dataset, a training run, and the evaluation to know it worked. Ian Fischer, co-founder of the AI startup Poetiq and a researcher at Google DeepMind for roughly a decade before that, put the ceiling bluntly in a YC Lightcone conversation. Training a model from scratch, he said, "costs you know hundreds of millions of dollars," and even ordinary fine-tuning means "the compute is so expensive." Most startups are not spending hundreds of millions, but the shape holds: fine-tuning is the option with a real bill and a real project attached before you learn anything.

The second cost is the one teams forget: obsolescence. You fine-tune on today's best model, and a few months later a better base model ships. Now your fine-tune is frozen on the old one. As Fischer described it, "a new model's come out and it's better than the thing that you fine-tuned," and you are left choosing whether to pay to do the whole thing again. One of the hosts put the founder's version of it more sharply: "the second you're in fine-tuning land," you spend the money, and then "I just lit it on fire" when the next frontier model lands and you cannot catch up.

That is the trap. Fine-tuning does not just cost more today. It couples your product to a specific model at a specific moment, in a field where the models are the fastest-moving part. Building for the next AI model instead of this one is the opposite instinct, and it is the right one.

What RAG gives you that fine-tuning cannot

RAG wins the default slot because it dodges both of those costs and adds a few things fine-tuning structurally cannot.

It is cheap to start and cheap to change. No training run. You build a retrieval pipeline once, and updating what the model knows is a matter of updating the documents it retrieves, not retraining anything.
It stays current. When your pricing, policy, or product changes, you change the source. A fine-tuned model bakes yesterday's facts into its weights and will confidently repeat them until you retrain.
It travels with model upgrades. Because RAG sits outside the model, the day a better base model ships you point your same pipeline at it and keep your grounding. Nothing to redo.
It can cite its sources. Retrieval knows which document produced an answer, so you can show provenance. A fine-tuned model cannot tell you why it said something. For anything a customer or a regulator might question, that auditability is the difference between "trust us" and a trail.

That last point is where the security angle lives, and as a CISSP it is the one I would not skip. With RAG your proprietary data stays in your own store and gets pulled in per request, so you can scope which agent may retrieve which data and log exactly what was used. Fine-tuning dissolves your data into model weights, where access control and deletion get much harder. Grounding at request time is easier to govern than knowledge melted into a model.

None of this means RAG is free of engineering. Bad retrieval gives grounded-sounding wrong answers, and you still need real evals to know your retrieval is pulling the right context. But those failure modes are cheaper to find and fix than a stale fine-tune.

When fine-tuning actually earns its cost

RAG being the default is not the same as fine-tuning being wrong. It earns its price in specific cases, and knowing them keeps you from the opposite mistake of refusing a tool you genuinely need.

Reach for fine-tuning when:

You need behavior RAG cannot hold. A strict output format, a consistent house voice, or a narrow reasoning pattern the model keeps drifting from even with careful prompting. That is a behavior problem, which is fine-tuning's home turf.
Cost and latency at scale are the goal. In the provider guidance, the main reason to fine-tune is to make a task cheaper and faster, not to add knowledge. If you are running a high-volume, narrow task, a fine-tuned smaller model can match a big model's quality on that one task at lower cost and latency. That is distilling frontier performance into something you can afford to run.
The instructions no longer fit in a prompt. If guiding the model takes so many examples and rules that your prompt is unwieldy, fine-tuning can move that guidance into the weights.

The tell across all three is that none of them is "the model does not know our data." Every one is about how the model behaves or what it costs to run, not what it knows. When you catch yourself reaching for fine-tuning to teach the model facts, that is the signal to use RAG instead. The two also compose. Plenty of mature systems fine-tune for format and voice and use RAG for current facts. Start with RAG, then add fine-tuning only when a behavior or cost problem you can name demands it.

The bigger lesson: invest above the model

Step back and the RAG-versus-fine-tuning question is a specific case of a larger one that runs through most of building a startup on AI: where do you put your work so it survives the next model release? Anything you bake into a frozen model is a depreciating asset. Anything you build in the layer above the model, the retrieval, the prompting, the system around it, keeps compounding as the models under it get better.

This is exactly what Fischer's company is betting on. Poetiq, a seven-person team, builds what people now call harnesses: systems of code, prompts, and strategies that sit on top of existing models and make them perform better without touching the weights. Fischer's pitch is that when a new model comes out, the same harness is compatible with it, so the team can "continue to optimize for this new model, whatever the new model is that you want to use." Poetiq says it has used the approach to top hard benchmarks cheaply, reporting a result on the Humanity's Last Exam benchmark ahead of the prior state of the art for an optimization cost under $100k, and beating a frontier system on the ARC-AGI-2 reasoning benchmark at roughly half the cost by building on a cheaper base model.

You do not need a research lab to apply the principle. RAG is the most accessible version of the same idea: put your advantage in the layer you control, so every model upgrade lifts you for free instead of stranding you. It is worth one honest caveat, made by people who build agents for a living: a harness that patches around today's model can lose its edge when a smarter model absorbs that capability directly. The layer above the model is durable, but the specific tricks in it are not guaranteed to be. That is an argument for keeping your layer simple and swappable, which is another point in RAG's favor over a heavy custom fine-tune.

The same logic runs through how you pick the model underneath in the first place: treat the base model as a swappable input, and keep your real work in the part you own.

What to do this week

Write down the actual problem in one sentence. If it starts with "the model does not know," that is a RAG problem. If it starts with "the model will not answer the way we need," consider fine-tuning.
Default to RAG for anything knowledge-shaped, and stand up a small retrieval prototype over your real documents before you consider training anything.
If you think you need fine-tuning, name the behavior or the cost target it is solving. If you cannot name one that is not "knowledge," you do not need it yet.
Add evals early so you can tell grounded answers from confident wrong ones, and so you can prove a change actually helped.
Keep your custom layer thin and model-agnostic, so the next base model is an upgrade you absorb, not a rebuild you dread.

Choosing RAG or fine-tuning well is one piece of building a company on top of AI without losing control of cost, quality, or your own data. That whole operating picture, how to build with AI and still ship reliably, is what we teach in the AI Operating System for Startups.

Sources

The Powerful Alternative To Fine-Tuning (Y Combinator's Lightcone), the conversation with Poetiq's Ian Fischer that this article distills, on why fine-tuning is the wrong first move and why the layer above the model wins.
Poetiq, Ian Fischer's company, for the harness approach and its reported benchmark results.
Provider guidance on choosing between the two: OpenAI on optimizing LLM accuracy, IBM on RAG vs fine-tuning, and AWS Prescriptive Guidance.
Inside Claude Code With Its Creator Boris Cherny (Y Combinator), on how scaffolding gains around a model can be absorbed by later models.