Automated Prompt Engineering: Run the Loop

Automated prompt engineering is what you get when you stop hand-tuning the instructions you send a model and start searching for them. A program proposes candidate prompts and the calling structure around them, scores each candidate against tasks it can grade, keeps what wins, and repeats. The output is not a better prompt you wrote. It is a system a machine assembled and you accepted because it measured better.

The case for taking it seriously is a result from late 2025 that is still hard to wave away. Poetiq, a company of six people founded in June 2025 by two former Google DeepMind researchers, took the top verified score on ARC-AGI-2, the second generation of the reasoning benchmark François Chollet introduced in 2019 and built to be hard for exactly the models it tests. It did that without training a model of its own.

That leaderboard has moved on, as leaderboards do, and the current standings are a click away. Treat the numbers below as a dated snapshot. The snapshot is what makes the point, because it captures a moment when a team with no model and no weights bought a large accuracy gain with method alone, and the method did not expire when the scores did.

Ian Fischer, Poetiq's co-founder and co-CEO, walked through the work with François Chaubard, a visiting partner at Y Combinator, in an interview recorded at NeurIPS. The score is the headline. The useful part is how carefully Fischer draws the line around when the technique is worth paying for, because he draws it in a place most coverage of this result skips.

What a six-person team actually did

ARC Prize, which runs the benchmark, verified Poetiq in December 2025 at 54 percent on the ARC-AGI-2 semi-private evaluation set at $30.57 per problem. The standing best at that point was Gemini 3 Deep Think in preview, at 45.14 percent and $77.16 per problem. Better and cheaper at once, against a system built by the company that owned the weights.

The team had no access to those weights, which is the same position you are in. Both founders came out of Google and DeepMind: Fischer was co-founder and CTO of Apportable, the iOS-to-Android porting company Google acqui-hired in 2015, and his co-CEO Shumeet Baluja spent more than two decades there. They started Poetiq in June 2025 and have since raised a $45.8 million seed round.

The solver they ran on ARC-AGI-2 was not built for it. By Fischer's account in the interview, the system was designed and tuned on ARC-AGI-1 and never trained on ARC-AGI-2 at all, which is why he describes running it as a check rather than an attempt. He also notes the method is not tied to one provider: when Anthropic's Opus shipped shortly after Gemini 3, "we could just replace Gemini 3 with Opus and get similar results."

The action space is the prompt, not the weights

Chaubard frames the constraint plainly. If you do not own the model, "the only thing in your action space to change is the prompt itself."

The answer that comes back from Fischer widens it by a clause, and the clause is the entire technique: "It's the prompt and the system around the prompt."

That system is a loop rather than a single call. Poetiq's own write-up describes generating a candidate solution, taking feedback on it, analyzing that feedback, and calling the model again to refine, with a self-auditing step that decides when to stop rather than burning calls forever. Layered above that is a second thing, which is the part worth stealing: a search that builds those solvers. Fischer describes it as a system that is "improving itself by improving other systems."

Hold the two layers apart, because conflating them is where founders get this wrong. The bottom layer is a solver for one task. The top layer is a search that produces solvers. Fischer's analogy for the difference between his DeepMind research and the company: there "we put together a car by hand," and at Poetiq "we've like built a factory to build cars."

The factory is the asset. A hand-tuned prompt is a car, and it depreciates the moment the model under it changes. Our post on what belongs in a system prompt makes the same point from the other direction: teams that rebuild their standing instructions at every model release find that large parts of what they wrote were compensating for a weakness the new model no longer has. If you have a scored loop, that rebuild is a job you run. If you do not, it is a week of somebody guessing.

Fischer is also candid that the loop has a stopping condition and it is not intellectual. Asked whether the search plateaued or he stopped it, he says he stopped it, because running it on this benchmark was expensive. The ceiling was the budget.

What automated prompt engineering costs per task

Put ARC Prize's late-2025 ARC-AGI-2 numbers in one column and the story changes shape.

• Gemini 3 Pro: 31.11 percent, $0.81 per task.
• Gemini 3 Deep Think (preview): 45.14 percent, $77.16 per task.
• Poetiq: 54 percent, $30.57 per task.

Against the expensive option, Poetiq is about nine points better at roughly 40 percent of the cost. Against the cheap option, it is about 23 points better at roughly 38 times the price per task.

So this is not a technique that makes AI cheaper. In a setting that allows several attempts, it can buy a higher accuracy for less than the expensive model charges, which is a narrower claim and a more useful one. Automated prompt engineering does not lower your floor. It raises your ceiling and bills you for the climb.

Fischer says this himself, which is the most trustworthy thing in the interview. He is explicit about the limit: "in general, if you're only allowed one response," the system "will always be more expensive." The benchmark allowed two submitted solutions, Poetiq submitted one, and the accuracy gain was large enough to still come out ahead on cost. Change that setting and the arithmetic changes with it.

The founder translation: the question is never whether to automate prompt work. It is what a correct answer is worth on this specific task, and whether that is worth more than a few dozen cheap attempts at it. On a support macro suggestion, almost certainly not. On a contract clause the customer will sign, an extraction feeding an invoice, a migration script that runs once against production, the answer flips, and it flips hard.

That is the same unit of measurement that decides everything else about your model spend. Our post on AI inference cost works through why cost per completed task, not cost per token, is the only meter that tells you anything, and a prompt search loop is the clearest case of the two numbers pointing opposite directions.

You cannot search what you cannot score

Here is the precondition that quietly disqualifies most teams. A search needs a scoreboard. ARC-AGI is a clean target because every puzzle has exactly one right answer and grading is free. Almost nothing in a real product looks like that.

Ankur Goyal, who founded the eval company Braintrust, argues that evals belong at the center of an AI product rather than at the end of it, and that everything else should revolve around them. We worked through that case in full in our post on LLM evaluation for founders.

Automation is what turns that from good practice into a hard requirement. Without a scored task set, a prompt loop has nothing to hill climb on and you are back to a person reading outputs and forming an impression. With one, the loop runs while you sleep. The eval set is not a testing chore you do after building. It is the thing that makes the building automatable at all, which is why it is worth writing before the prompt rather than after. LLM observability covers where the traces those evals score actually come from.

One security note, because an automated search over prompts is an automated search over your system's instruction surface. Whatever the loop discovers becomes production instruction text that no human wrote and possibly no human read. That is a supply chain you now own. Treat the winning prompt like code: store it in version control, diff it against the previous winner, have a person review it before it ships, and keep the eval set that justified it next to it. A prompt that scored well on your benchmark can still contain an instruction you would never have approved, and the loop has no opinion about that.

Where this leaves the person who writes prompts

Chaubard, who is finishing a Stanford AI PhD and founded the retail computer vision company Focal Systems, lands the observation that gives the whole interview its point. Manually trying prompt variants against evals, he says, means "we're back to like feature engineering," the hand-crafted input design that deep learning was supposed to end. And "the whole thing of deep learning since 2012 is get yourself out of the loop."

Fischer agrees and says the quiet part directly. Poetiq, he says, is "quite intentionally automating ourselves, automating prompt engineers, automating people who are building agents." He calls it a power tool.

It is worth being honest that the idea has been available for years. Chaubard raises DSPy, the Stanford NLP framework published at ICLR 2024 that compiles declarative model calls into optimized pipelines, and says he tried it and did not see it work especially well. Fischer credits the project warmly and jokes about hiring its authors, then says his team has "trade secret insights" that go beyond it and "seem to make a big difference." Read that exchange for what it is: the general method is public, and the person who took the benchmark says the gap is proprietary. That is worth knowing before you budget a research project of your own, because the public version is free to try this week and the part that closes the remaining distance is the part nobody is going to explain to you.

Which is also why this is not really a prompting story. It is the same pattern as any other loop you can close inside a company, covered in our post on self-improving AI: find the work that is repetitive, define what better means in a number, and let the measurement drive the iteration instead of a person's taste. Prompts are simply the surface where founders can close that loop this month with no research team. The wider frame for how these pieces fit together is our pillar on AI for startups.

What to do this week

1. Pick one task where you can grade the answer. Extraction, classification, routing, schema conformance, anything with a checkable output. If you cannot grade it automatically, it is not the place to start.
2. Write the eval set before you touch the prompt. Thirty to fifty real cases from your own logs, each with the answer you wish you had gotten. This is the asset, not the prompt.
3. Compute what a correct answer is worth. Put a number on the task: the refund avoided, the review hour saved, the deal unblocked. Compare it against the cost of running twenty model calls instead of one. That comparison, not the technique, makes the decision.
4. Run an existing optimizer before you build one. Try DSPy or a comparable prompt optimizer against your eval set for a day. If a public tool moves your score, you have your answer cheaply, and if it does not, you have learned something about your evals.
5. Put the winning prompt under review. Version it, diff it, have someone read it, and keep the eval run that justified it attached. An instruction nobody wrote still ships to your customers.
6. Set the budget as a number, not a feeling. Decide the ceiling on calls per task up front. Fischer stopped his search because of cost, and yours will stop for the same reason, so choose the line deliberately.

Prompting was never going to stay a craft. It was always an optimization problem waiting for someone to write the loop. The teams that get there first are not the ones with better instincts about wording, they are the ones who built a scoreboard early enough to let a machine climb it. The scoreboard, the budget ceiling and the review step are operating decisions rather than model choices, and that ground is what AI Operating System for Startups covers.

Sources

• This Startup Beat Gemini 3 on ARC-AGI, at Half the Cost (YC Root Access), the interview this article distills, with Ian Fischer and François Chaubard at NeurIPS.
• Poetiq's verified ARC-AGI-2 results and the company's announcement post, the 54 percent score, the $30.57 per problem cost, and the description of the iterative solving loop.
• ARC Prize published the per-model ARC-AGI-2 semi-private numbers cited here for Gemini 3 Pro and Gemini 3 Deep Think, restated the Gemini 3 Pro baseline and the Poetiq refinement in its 2025 results analysis, and maintains the public leaderboard where the current standings live.
• Poetiq's $45.8 million seed announcement, founding date, team size, and the backgrounds of Ian Fischer and Shumeet Baluja.
• Background on Apportable's 2015 acquisition by Google: VentureBeat and Crunchbase.
• Braintrust's Ankur Goyal on Why Evals Are the Core of AI Development, Greylock, the evals precondition, treated at length in our own post on LLM evaluation for founders.
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, the Stanford NLP framework Chaubard raises in the interview.