What is an open source LLM?

An open source LLM is a large language model whose weights you can download, run on infrastructure you choose, and adapt to your own tasks, rather than reaching it only through one vendor's API. Strictly speaking, most models sold under that label are open weight rather than open source: you get the trained parameters and a license, but not the training data or the code used to produce them. The Open Source Initiative's Open Source AI Definition requires the weights, the training and filtering code, and enough information about the data that a skilled person could substantially recreate the system, under terms permitting the freedoms to use, study, modify, and share it. Very few popular releases meet that bar. The practical part still holds: you can run the model where you want and change it, which is what alters your architecture and your data flow.

Are open source LLMs as good as frontier models like GPT or Claude?

Not on the hardest open-ended work, and better than the frontier on a narrower class of tasks. The pattern reported by teams running both in production is that frontier models stay ahead on novel reasoning, long-horizon agentic work, and anything where you cannot predict the shape of the answer. Once a use case is solidified and running at scale, a small open weight model that has been fine-tuned on that one task can match or beat a large general model on it, while being faster and cheaper, provided you have the eval and the volume to justify the fine-tuning. Decagon, the AI customer support company, says roughly 90 percent of its workflow runs on open source models, and that its motive was latency rather than cost, with the saving arriving as a side effect. It still reaches for frontier models on anything new or experimental.

Is it cheaper to use an open source LLM?

Per token, usually yes. On your monthly bill, often not, because cheaper tokens reliably invite more tokens. OpenRouter's CEO Alex Atallah described a case on the platform where the price of one model fell roughly tenfold in two weeks while usage grew roughly thirteenfold, and he is careful to add that nobody has modeled the effect rigorously. The more useful lesson comes from Coinbase, whose widely shared roughly 50 percent cut in AI spend while usage rose came from five levers together: cheaper open weight defaults, task-based routing, aggressive caching, leaner context, and per-engineer spend visibility without usage caps. Only one of those five is the choice of model. Treat an open weight model as one lever in a cost system, not as the saving itself.

Are Chinese open source LLMs safe for a startup to use?

Technically yes, with controls you should already be running. Commercially, ask your customers before you build the dependency. The technical risk is manageable: keep inference inside your own boundary, treat the weights as an untrusted dependency you pin and verify, log prompts and outputs, and run your own prompt-injection and PII controls rather than assuming a vendor supplies them. The harder risk is not technical at all. Your enterprise customers' procurement and data-residency reviews may prohibit a model of a given origin regardless of how it behaves, and that answer tends to arrive late in a sales cycle, after the dependency is expensive to remove. Ask a design partner's security team before you build it, not after.

Open Source LLM: Where It Actually Belongs

An open source LLM is a large language model whose weights you can download, run on infrastructure you choose, and adapt to your own tasks, instead of reaching it only through one vendor's API.

The short version of the decision below: move a call to an open source LLM only when you already know the output shape, the use case has stopped changing, you have a task-specific eval that would catch the regression, and the data or a customer's security review argues for keeping inference inside your own boundary.

Almost every page that ranks for this term answers the next question with a list: the ten best open source LLMs, ranked by parameter count and benchmark score. Those lists are mostly accurate. They also answer a question no founder actually has.

You are not picking a trophy. You are deciding which specific tasks inside your product should stop calling a frontier model, and what you take on when they do. That is an architecture decision with a different right answer per task, not per company.

The clearest version of that decision came from Alex Atallah, co-founder and CEO of OpenRouter, in a 20VC interview with Harry Stebbings. OpenRouter is an unusual vantage point: it routes developer traffic across a large catalog of models and the independent providers that serve them, so it sees which models teams keep paying for after launch week rather than which ones win benchmarks. Atallah was co-founder and CTO of OpenSea before starting it.

His answer is not that open models are cheaper. It is that they are good at a particular shape of work.

What an open source LLM actually is

Most models marketed as open source LLMs are open weight. You get the trained parameters, a license, and usually a model card. You do not get the training data, the filtering code, or a recipe that would let you rebuild the thing yourself.

The Open Source Initiative sets a higher bar in its Open Source AI Definition: to call an AI system open source, the release has to provide the weights, the complete source code used to train and run it, and enough information about the data that a skilled person could build a substantially equivalent system, all under terms that permit the four freedoms to use, study, modify, and share it. Most popular releases do not clear it. They are open weight, which is the phrase Atallah uses throughout the interview and the more honest one.

This is not pedantry. Three practical things follow.

You cannot fully audit what went into the model. With no training data disclosure, a customer's question about provenance, contamination, or licensed material gets answered with the vendor's word and nothing stronger.
The license is a constraint, not a formality. Open weight releases carry real terms: acceptable-use clauses, field-of-use limits, sometimes thresholds that trigger at scale. Read it before you build a product line on it.
You still get the part that changes your architecture. Weights you can run where you want and adapt are the operational difference, and it is enough to change your deployment and your data flow. That is the entire reason to care.

Use the accurate word internally. Keep the marketing word in the docs your customers search.

It is also worth knowing whose models you are talking about. The families that come up repeatedly in these conversations are DeepSeek, Moonshot's Kimi, and GLM, all from Chinese labs, which is why the provenance question later in this article is a commercial problem and not only a technical one. On the American side Atallah names newer labs such as Poolside and Thinking Machines. Version numbers move monthly and any specific one here would be stale before you read it, so treat the families as the durable part and re-check the current release when you shortlist.

The real question is which layer, not which model

Atallah's rule is about task shape, and it is specific enough to act on. His rule names the condition: "When you have a deterministic task where you know the shape of the output, you know the type of problem that you're working on, and it's a type of problem that has been solved, like classifying some text". That is when you reach for the low-cost model, and the orchestrator then reads the result and carries on with the part it could not predict.

He is explicit about where that leaves open weights, and it is a claim about comparative advantage rather than price: "This is what open weight models are generally really good at compared to Frontier models."

The architecture that falls out of this is not one model swapped for another. It is a frontier model on top holding the plan, calling out to a bench of small cheap models underneath: "these sub agents are just very very low cost, and they're focused on deterministic tasks". You keep the expensive intelligence for the part of the job that is genuinely uncertain, and stop paying frontier prices to decide which of six categories a support ticket belongs to.

The best available proof that this works at scale comes from a different source. On a16z's podcast episode, Decagon co-founder and CEO Jesse Zhang said "today 90% of our workflow is on open source". Decagon builds AI customer support, and Zhang started it with Ashwin Sreenivas. Decagon draws the boundary on a second axis, maturity rather than task shape:

"at a certain point it's strictly better to use open source models"

The condition Zhang attaches is maturity: the use case is solidified, it is running in production at scale, and the shape of the agent has stopped moving. Decagon states the other half of that boundary in the same conversation, that "for everything that's new and sort of experimental" the team is "still going to use frontier models".

So there are two boundaries here, not one: a known output shape, and a use case that has stopped changing. Two independent vantage points, a router and a production vendor, arrive at the same place. Novel work stays on the frontier. Settled work moves down. Glean co-founder and CEO Arvind Jain puts a related number on it from the application layer, that "90% or greater of use cases can now be fully handled by many many different models including open source models", which is a claim about interchangeability rather than about maturity.

So the question is not which open source LLM is best. It is which of your calls have a known output shape. If you have not yet decided how to choose a model per task at all, do that first, because open weights are a special case of that discipline rather than a replacement for it.

Open source LLMs buy latency and control before they buy savings

Decagon did not move to open source models to cut costs. They moved for latency, to make voice agents respond fast enough to feel like a conversation. On the money, their line is blunt: "we didn't do these for cost benefits but that's like a nice side effect".

The second reason is the one that shows up in enterprise deals, and it runs opposite to the intuition that a startup should fear the unfamiliar model. Asked which models US companies are more nervous about, Atallah said frontier models, usually, because "there's much more confusion around the data policy about what's like actually happening to the prompts they're sending". Then the sentence that matters architecturally: "you can't run them on your own machine or in a provider of your choice".

He points out that the objection has a shape every enterprise buyer already knows: it pattern matches to "running on their own infra versus running in their VPC". That is not a new class of risk. It is a procurement conversation your customer's security team has had a hundred times and knows how to close.

This is the real asymmetry of an open weight model. It is the only option that lets you put the model inside the trust boundary you already sell. If your product handles regulated data, or you are stuck in a security review because prompts leave the customer's environment, a small open model on your own infrastructure can unblock the deal on its own merits, before anyone opens a spreadsheet. A self-hosted AI agent is the same tradeoff taken one step further, and it comes with the same bill: you gain the boundary, and you inherit everything that used to be the vendor's job.

Open source LLM costs: cheaper tokens do not mean a smaller bill

Founders reach for open weights to control spend, and then the spend does not go down.

Atallah invokes the Jevons paradox: "prices go down by 10x, the usage increases by more than 10x". He frames the general effect cautiously, noting that "no one has really done a great job modeling it", then gives a concrete case from his own platform. OpenAI cut the price of one of its own models to about a fifth, and then, in coordination with OpenRouter, roughly halved it again, so to about a tenth in two weeks. Usage over that window grew roughly thirteenfold. He calls that "a close to perfect Jevons paradox story" with "few other confounding variables in the story". One clean data point is still one data point, and note which model it was: a closed frontier one. The elasticity is a property of cheap tokens, not of open weights.

The direction is corroborated by the people selling the tokens. Lin Qiao, co-founder and CEO of Fireworks AI and previously the Meta director who led PyTorch, told 20VC she expects a "10x cost reduction in the next three years", and that "this 10x cost reduction will drive a 100x usage" (20VC interview).

Which brings us to the number everyone quotes at each other. Coinbase CEO Brian Armstrong reported cutting the company's AI spend by roughly half while token usage kept climbing, and the post traveled fast enough that 20VC spent a segment on it. Read the breakdown, as reported by Business Insider and The Information. The savings came from five levers working together:

Setting cheaper open weight models as the default for engineers.
Task-based routing, so each prompt goes to the cheapest model that can handle it.
Aggressive caching.
Leaner context on every call.
Per-engineer spend visibility, explicitly without usage caps, with more impact expected from whoever spends more.

One of those five is the choice of model. Four of them are infrastructure and discipline. Armstrong's own framing is that the levers work together, and the context lever is a habit rather than a purchase, which is why context engineering often returns more per week of work than a model migration does.

The 20VC panel added a caveat worth carrying: the cut mostly returned Coinbase to the spend level it had held around the previous November, roughly half a year earlier. The panel's sharper reading, and it is their inference rather than a claim in Armstrong's post, was that AI spend across the industry had been ramped up faster than the measurable productivity justified. That reading is not about open source at all.

Atallah sees the anxiety from his seat too, describing companies "freaking out about the amount of inference they're spending and they don't know how to think about it". Choosing a cheaper model does not teach you how to think about it. A routing policy and visible per-engineer spend do.

What switching to an open weight model costs

OpenRouter measures churn between models, which makes Atallah one of the few people who can say what stops teams from switching even when a better option exists. He gives three reasons, and all three apply to a move onto open weights.

The thing already works. "my app works and I don't want to break it". You have accumulated prompts, guardrails, and fixes around the current model's specific failure modes, and a swap puts all of that back in play.
The price gap closes on its own. The model you are already on gets cheaper over time, so a migration you justify on today's price difference can be underwater by the time it ships.
Trust is personal and unwritten. "A lot of people have these personal evals that are just these random tests that they give the models." If the new model fails somebody's private test, the migration dies in review no matter what the benchmark said.

Then there is the engineering, where Decagon's experience is the useful account: "most small models out of the box are not going to be good enough at the task that we want them to do", so you fine-tune, and "fine-tuning these models is non-trivial". They needed their own data, their own task-specific benchmarks because public eval sets did not measure the thing they cared about, and a dedicated research team they describe as expensive. They also retrain and deprecate models continually as the frontier moves, which is an ongoing cost rather than a migration.

Two consequences for a startup with no research team.

Do not fine-tune first. Adapt the context before you adapt the weights, because the RAG versus fine-tuning decision usually resolves toward the cheaper move, and a prompt you can revert in an afternoon is worth more than a checkpoint you cannot. When you do change models, re-audit what you are sending: Atallah notes that as models get better, accumulated instructions become a handicap, and points to Anthropic's published finding that stripping more than 80 percent of a system prompt removed contradictions and cost nothing on the coding evals, performing at least as well and in places better.

And build the eval before the migration, not after it. You cannot judge a model swap without a task-specific test on real customer cases, which makes LLM evaluation the prerequisite rather than the follow-up. Decagon's constraint applies at any size: a public benchmark will not tell you whether this model does your job.

What open weights change about your security posture

Running a model yourself does not remove risk, it relocates it, and the move helps you in one direction while costing you in another.

What improves is real. The data stays inside a boundary you control, which is the argument that closes enterprise reviews. Inspectability improves too, though it is worth being precise about which activity earns it. Atallah's point is about distillation: because most open weight models permit it, a lab training on their outputs can "see the output so you can like inspect them to make sure that they're aligned" during its reinforcement learning rollouts. That is a property of the license and the weights rather than of self-hosting, which is one more reason to read the license before you commit.

What you inherit is everything the API was quietly doing for you. Atallah describes the controls his own platform hands developers as switches rather than projects: "we have prompt injection protection", which flags prompts that look like injection attempts, plus PII reduction, both of which a team can turn on in a click. Pull the model in-house and every one of those becomes yours to build, run, and monitor. Teams tend to discover this after the first injection incident, which is a bad time to learn it.

His framing for how to hold this is the right one: "I think of the models a little bit like the internet." You cannot ban the internet at your company because parts of it are hostile, so "you can create guardrails and you should".

Concretely, treat a downloaded model as an untrusted third-party dependency, because that is exactly what it is:

Pin the exact version and verify what you fetched. Weights are a supply chain, and a floating latest tag is not a version.
Isolate inference. Give the serving process no standing credentials and no network reach it does not need, so a successful injection has nowhere to go.
Log prompts and outputs, with retention you can defend to a customer and redaction you can prove.
Run your own injection and PII controls at the boundary, and test them like any other control.
Write down the model's provenance and license in the same place you track your other dependencies, before a security questionnaire asks.

Provenance deserves its own paragraph. The strongest open weight families named above are Chinese, and Atallah puts it plainly: "America is very, very behind still." The technical risk is manageable with the controls above. The commercial risk is not technical at all: an enterprise customer's procurement or data-residency review may rule out a model by origin whatever it does on your benchmark, and that answer tends to arrive late in the sales cycle. Ask a design partner's security team early, while the dependency is still cheap to change.

Should you use an open source LLM? Four tests

Run each candidate call through four tests. Move it to an open weight model only if all four pass.

Shape. Do you know the form of the output before the call? Classification, routing, extraction, tagging, scoring, and redaction pass. Open-ended generation does not.
Solidified. Is this use case in production and stable, or still being discovered? If the spec is still moving, the frontier model is buying you iteration speed and you should keep paying for it.
Eval. Do you have a task-specific test on real customer cases that would catch the regression? No eval, no migration.
Boundary. Does this call carry data you would rather not send to a vendor, or does a customer's security review already block it? A yes here can justify the move even when the money is neutral.

Everything that fails a test stays on the frontier model and gets revisited next quarter, when the shape may have solidified. The frontier moves fast enough that the list changes, which is the argument for a routing layer and a thin seam rather than a permanent verdict. The broader version of this discipline, treating the model as a replaceable dependency and investing in the layer around it, is the engineering half of running AI for startups.

What to do this week

List your top five LLM calls by monthly spend. For each, write down the output shape in one line. The ones you can describe as a fixed set of categories or fields are your candidates.
Pick the single highest-volume call that passes the shape and solidified tests, and build the eval for it before you touch a model. Twenty to fifty real customer cases with known-good answers is enough to start.
Turn on routing and caching first, then swap the model. Measure them separately so you learn which lever actually paid.
Read the license of any open weight model you are considering, and record its provenance next to your other dependencies. Send the origin question to one design partner's security team this week rather than next quarter.
Before the model changes, delete what your prompt no longer needs. Accumulated instructions cost tokens on every call and contradict the newer model more often than they help it.

If you want the full operating system for these decisions, from choosing the job to pricing the product, that is what I teach in AI Operating System for Startups.

Sources

OpenRouter CEO Alex Atallah on 20VC with Harry Stebbings, the interview this article distills: the deterministic-task rule, the orchestrator and sub-agent split, why enterprises are often more nervous about frontier models, the Jevons paradox case on OpenRouter, the three reasons teams do not switch models, and the guardrails framing.
How Decagon Runs 90% of Its Agents on Open-Source Models on a16z's podcast, for the production counterpart: the 90 percent figure, the strictly-better boundary, latency rather than cost as the motive, and what fine-tuning actually costs.
Lin Qiao of Fireworks AI on 20VC for the forecast of a tenfold cost reduction driving a hundredfold rise in usage.
20VC's news episode on Coinbase's AI spend for the panel's reading of the Coinbase post, including the November baseline it returned to.
Glean co-founder and CEO Arvind Jain on 20VC for the application-layer view of how many use cases open models now cover.
The Open Source Initiative's Open Source AI Definition for the standard that separates open source from open weight.
Anthropic's note on context engineering for its Claude 5 generation models for the system-prompt result Atallah refers to in the interview.
Reporting on Coinbase's cost reduction and its five levers: Business Insider and The Information.
Backgrounds: Alex Atallah (personal site, LinkedIn), Decagon founders Jesse Zhang (LinkedIn) and Ashwin Sreenivas (LinkedIn), Lin Qiao (LinkedIn, CNBC), and Arvind Jain (LinkedIn).