Prompt Tuning Stops Working When the Workflow Is the Problem

FAPO is a practical attempt to make LLM pipeline optimization look more like debugging a system than rewriting a prompt.

Source note: Paul Kassianik, Baturay Saglam, Huaibo Zhao, Blaine Nelson, Supriti Vijay, Aman Priyanshu, and Amin Karbasi. “FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines.” arXiv:2606.19605v1, June 17, 2026. https://arxiv.org/abs/2606.19605v1

Why This Paper Matters

Most LLM applications are no longer one prompt and one answer. They are small workflows. A system retrieves context, summarizes it, asks a model to reason, formats the answer, checks constraints, maybe calls another model, then returns something a user is supposed to trust.

That creates a simple problem: when the output is wrong, the prompt may not be the source of the failure. The retrieval step may have missed evidence. The reasoning step may have used the wrong passage. The answer step may have added too much text for an exact-match scorer. The workflow may need another hop, a stricter parser, or deterministic constraint enforcement.

FAPO matters because it treats the LLM pipeline as the thing to optimize. The paper’s claim is that an agentic coding environment can inspect intermediate failures, try the smallest reasonable prompt change first, and only then change the workflow when the evidence says prompts are not enough.

The Idea in Plain English

FAPO stands for Fully Autonomous Prompt Optimization, but the name undersells the interesting part. This is not only prompt optimization. It is workflow optimization with a prompt-first bias.

The system gives Claude Code a structured workspace for one task. That workspace contains the task instructions, data splits, scoring rules, current prompts, chain code, allowed changes, and history. FAPO runs the current pipeline on training cases, records the final output and intermediate step outputs, groups failures by likely cause, proposes one scoped improvement, has another agent review the change, and evaluates the variant.

If failures look prompt-addressable, it edits the prompt. If failures look structural and the tenant scope allows it, it can change chain parameters or chain structure.

That escalation rule is the practical heart of the paper. FAPO does not jump straight to architecture changes. It starts with the cheap intervention, then moves up the stack when recorded failures show the cheap intervention is running out of room.

What the Researchers Tested

The authors compare FAPO with GEPA, a prompt optimizer for multi-step reasoning pipelines. GEPA searches instruction strings inside a fixed DSPy program. FAPO starts from the same baseline pipeline but can operate over a broader workspace when permitted: prompts, chain parameters, and chain architecture.

The evaluation covers six benchmarks and three task models. The benchmarks include HotpotQA for multi-hop QA, HoVer for fact verification, IFBench for instruction following, LiveBench-Math and AIME for math, and Papillon for privacy-conscious delegation. The three task models are GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B.

They also test CTIBench Root Cause Mapping, a cybersecurity task that maps CVE descriptions to CWE IDs. That task is constrained to prompt edits only, following the Foundation-Sec evaluation setup.

Each benchmark/model cell reports mean test score over three trials. FAPO is allowed up to 50 variants or 10 optimization rounds per trial. The system starts at prompt level and escalates only when prompt optimization appears insufficient and failure attribution identifies a structural bottleneck.

What They Found

FAPO usually beats GEPA

Across the six main benchmarks and three task models, FAPO beats GEPA in 15 of 18 model-benchmark comparisons. The mean FAPO-over-GEPA gain is 14.1 percentage points. In 11 comparisons, the win has non-overlapping mean plus/minus trial-standard-deviation ranges.

The headline is not that Claude Code magically writes better prompts. The headline is that pipeline-aware optimization can find fixes prompt-only search cannot reach.

The largest gains come when the workflow changes

The biggest improvements happen on HoVer and IFBench. These are the two cases where FAPO escalates beyond prompt edits.

On HoVer, attribution points to insufficient retrieval coverage. FAPO extends the retrieval chain from 3 hops to 4-5 hops, adds multi-query BM25 search, and adds entity-aware rescue. The result is a gain of 24.78 to 48.56 percentage points over GEPA, depending on the task model.

On IFBench, attribution points to format and constraint failures. FAPO adds deterministic post-processing nodes that enforce instruction constraints. The result is a gain of 19.84 to 38.95 percentage points over GEPA.

That is the most useful result in the paper. If a system fails because the workflow cannot retrieve enough evidence or cannot enforce constraints, no amount of prompt polishing is likely to be the cleanest fix.

Prompt-only FAPO still helps

The paper is careful not to claim that all gains come from broader architecture changes. On prompt-only comparisons, FAPO wins 9 of 12 comparisons, with six wins showing non-overlapping ranges.

The CTIBench-RCM results are a good example. Under prompt-only scope, FAPO improves GPT-5 from 72.1 percent to 76.1 percent test accuracy, Foundation-Sec-8B-Instruct from 63.9 percent to 71.0 percent, and Foundation-Sec-8B-Reasoning from 71.0 percent to 73.0 percent.

The model-specific prompts are also revealing. GPT-5 benefits from added NVD convention rules for specific CWE confusions. The smaller Foundation-Sec Instruct model performs best with a shorter prompt. The Reasoning model benefits from the phrase “standard NVD abstraction level.” The same task does not want the same optimized prompt for every model.

AIME is the weak spot

AIME is the one benchmark where GEPA leads FAPO across all three task models. The authors treat the result as inconclusive rather than a clear FAPO failure, because the FAPO changes are mixed relative to baseline and appear within the noise range.

Still, this matters. FAPO is not a universal improvement machine. On small-sample math tasks, the optimizer may overfit or fail to find a useful intervention.

Why It Happens

FAPO works because it makes failures inspectable. A normal final-answer score tells you that a case failed. It does not tell you where the mistake started.

FAPO records step outputs from the pipeline. That makes it possible to see whether the error starts in retrieval, intermediate reasoning, output formatting, or the final answer step. The optimizer is not guessing from the final answer alone. It has a trace.

The paper also puts useful limits around the agent. A tenant playbook defines what the optimizer can change. A scope contract states whether prompt text, chain parameters, or chain structure are allowed. A reviewer agent checks proposed variants for scope compliance, placeholder integrity, data leakage, scorer compatibility, state protocol compliance, and unsafe imports. Variants are immutable and logged.

This is why the paper feels more operational than many prompt-optimization papers. The authors are not only asking whether an LLM can improve a benchmark score. They are asking how to put an optimizer inside a constrained workspace without letting it quietly cheat.

What This Means for Builders

For builders of agentic products, the main lesson is to stop treating prompt edits as the only optimization surface. Once the system has multiple steps, the right fix may be a retrieval change, a parser, a deterministic rule, a longer chain, a shorter prompt, or a stricter output contract.

That means observability matters. If your pipeline does not record intermediate artifacts, you cannot reliably tell whether the failure is prompt-addressable. You will keep rewriting the final prompt because it is the only thing you can see.

The second lesson is to use scope contracts. An optimizer should know what it is allowed to change and what it must not touch. That is especially important in enterprise or security settings, where one task may have customer-specific data rules, scorer rules, or deployment constraints.

The third lesson is to make variants durable. FAPO’s tenant model records prompts, chain variants, configs, run artifacts, and optimization history. That turns optimization into a reviewable engineering process rather than a pile of ad hoc prompt experiments.

What This Means for Buyers and Operators

For buyers, this paper is a reminder to ask vendors how their agents improve after failures. “We tune the prompt” is not much of an answer if the product depends on retrieval, tools, formatting, and workflow state.

The better question is: when the system is wrong, can the vendor show where the mistake began?

Operators should also care about guardrails. Automated optimization can overfit, leak data, or silently change the rules of the task if the workspace is not constrained. FAPO’s split access controls and reviewer checks are not decorative details. They are the difference between a useful optimization loop and an agent that learns to game its own benchmark.

For security teams, the CTIBench-RCM result is interesting because prompt-only optimization improves CVE-to-CWE mapping across GPT-5 and two security-tuned models. But the more important point is model-specific tuning. The best GPT-5 prompt is not the best smaller-model prompt. Teams should expect optimization to be model-specific, task-specific, and brittle if treated as a copy-paste exercise.

What to Watch Next

The next question is whether this kind of optimizer works outside benchmark tenants. Real production pipelines have messy data contracts, permission boundaries, nonstationary inputs, latency constraints, cost constraints, and humans in the loop.

Another thing to watch is how much supervision the optimizer actually needs. FAPO uses Claude Code plus custom agents, commands, reviewer checks, and tenant playbooks. That may be exactly the right level of structure, but it is not “press button, get better pipeline” in the casual sense.

The field should also watch whether pipeline-aware optimizers become standard infrastructure. If agents increasingly build or repair workflows, then optimization history, step attribution, and scope contracts may become normal parts of the agent engineering stack.

Limitations and Caveats

The comparison with GEPA is not perfectly apples-to-apples. GEPA optimizes prompts inside a fixed program. FAPO sometimes has permission to change prompts, parameters, and chain structure. That broader search space is part of the point, but it also means the paper is not simply proving that one prompt optimizer beats another prompt optimizer.

The authors acknowledge reproduction differences as well. Their GEPA scores differ from the published GEPA results by -3.78 to +7.97 percentage points. They also use Claude Opus 4.6 through Bedrock as the GEPA reflector, replacing the original reflector model.

FAPO can also have higher variance when escalation to pipeline changes is allowed. Some trials find the structural intervention. Others stay closer to prompt optimization. That path dependence is important for anyone hoping to use this kind of system in production.

Finally, the benchmark set is broad but still bounded. The paper shows strong evidence that pipeline-aware optimization can beat prompt-only search in several controlled settings. It does not prove that autonomous optimization is safe or reliable for arbitrary production workflows.

Source

Paul Kassianik, Baturay Saglam, Huaibo Zhao, Blaine Nelson, Supriti Vijay, Aman Priyanshu, and Amin Karbasi. (2026). FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines. arXiv preprint arXiv:2606.19605v1. Available at: https://arxiv.org/abs/2606.19605v1