Some of the best teachers for reasoning models may be systems that cannot explain themselves.
Source note: Ziran Yang, Chengshuai Shi, Raj Ghugare, Benjamin Eysenbach, Karthik Narasimhan, and Chi Jin. “LeAct: Learning to Reason from Expert Actions.” Princeton University, arXiv:2607.21856, July 23, 2026. https://arxiv.org/abs/2607.21856
Why This Paper Matters
Reasoning models need reasoning data. That sounds obvious until you ask where the data comes from.
People can write explanations, but human annotation is expensive and slow. A stronger language model can generate chains of thought for a smaller one, but then the student inherits the teacher’s habits and limits. A model can also learn from its own successful attempts, though this risks reinforcing plausible-looking reasoning that happens to land on the right answer.
Meanwhile, many important fields already have expert systems that make excellent decisions. Game solvers choose strong strategies. Planning systems find efficient routes. Theorem provers verify formal steps. Robotics controllers select actions that move a machine toward a goal. These systems know what to do, but most do not explain their decisions in natural language.
LeAct asks whether their actions can still teach a language model how to reason.
The paper’s answer is yes, but showing the expert action to a model and asking for an explanation is not enough. The model may simply invent a convincing story after the fact. An explanation becomes useful training data only when it improves the model’s ability to recover the expert decision.
Explanation generation then becomes a testable loop instead of a writing exercise.
The Idea in Plain English
Imagine watching a great chess engine make a move. The engine does not give you a lesson. It just moves the bishop.
You could ask a student to explain why the bishop move was good. The student might propose several stories: it protects the king, attacks a weak square, or prepares a later tactic. Some explanations will sound intelligent without helping the student choose the move again in a similar position.
LeAct keeps only the explanations that make the student more likely to reproduce the expert move.
For every state, the student sees a qualitative summary of the expert’s preferred action and generates several possible reasoning traces. Each trace is then put back in front of the student. If the trace raises the probability the student assigns to the expert action relative to its normal baseline, it passes. If it does not help, it is discarded.
The method reverses the familiar “reason, then act” sequence during training:
- Start with the expert action.
- Generate possible reasons for it.
- Test whether each reason helps recover the action.
- Train on the reasons that pass.
The authors call the score forward delta: the improvement in the student’s prediction of the expert action when a candidate explanation is present.
A chain of thought enters the training set only if it changes the student’s decision in the right direction. Fluency, detail, and persuasiveness do not count on their own.
What the Researchers Tested
The researchers used Qwen3-8B as the student model across every experiment. Keeping one model size fixed let them focus on the quality of the supervision rather than mixing the effect with model scale.
They tested seven settings:
- Three versions of Leduc Hold’em, ranging from roughly 4,000 to 80,000 information sets.
- Liar’s Dice.
- Three-player Leduc.
- Flop Hold’em, with roughly one billion playable information sets.
- BuilderBench, a simulated robotics benchmark in which a robot arm stacks cubes.
The expert was not always the same kind of system. Smaller games used exact or near-exact game solvers. Flop Hold’em used a DeepCFR policy. BuilderBench used successful trajectories produced by frontier language models. LeAct therefore operated with full mixed-action distributions in some settings and single demonstrated actions in another.
The training loop sampled eight candidate explanations per state and retained the top two whose forward delta was positive. The model then learned both the selected reasoning trace and the expert action. When the expert exposed a full distribution rather than one move, the student was trained on that distribution. The paper calls this expert-policy forcing.
The main comparisons were:
- No-chain-of-thought behavior cloning: learn the expert action directly.
- Chain-of-thought expert iteration: generate reasoning forward, keep the attempts closest to the expert, and train on the model’s resulting actions.
- LeAct: generate reasoning backward from the expert action, score whether it helps forward prediction, and train against the expert policy.
All iterative methods started from the same cold-start data and used the same student model.
What They Found
Reasoning matters when memorization runs out
On the smallest Leduc game, direct behavior cloning was already extremely strong. The policy table was small enough for the model to memorize. LeAct did not create a meaningful advantage there.
The pattern changed as the game grew. On the 80,000-information-set version of Leduc, LeAct’s single-sample exploitability was 0.092, compared with 0.501 for chain-of-thought expert iteration. Lower is better, so LeAct was about 5.4 times closer to the solver. With best-of-eight inference, LeAct reached 0.022 versus 0.087.
Direct imitation remains the simpler answer when a policy is compact enough to memorize. Selected reasoning becomes more useful when the state space grows too large for table fitting.
The gains carried into unseen states
Flop Hold’em gave the model only a tiny sample of a game with roughly one billion information sets. On 20,000 held-out states, LeAct produced a lower divergence from the DeepCFR teacher than either baseline.
It also won when the policies played each other. Across 50,000 paired hands, LeAct beat direct behavior cloning by 60 milli-big-blinds per game and chain-of-thought expert iteration by 62. The seat assignments and card sequences were paired to reduce positional noise.
A second test held out entire card ranks from training. LeAct had roughly a threefold advantage over chain-of-thought expert iteration on the unseen ranks. The absolute performance was still far worse than in the full-tree setting. LeAct reduced the failure under distribution shift; it did not make the shift disappear.
The robotics result was promising but narrower
BuilderBench tested whether the idea survived outside card games. The task used a simulated robot arm to place cubes at target positions.
On the 26 in-domain tasks, LeAct was the only tested training recipe whose best-of-64 progress exceeded the common cold-start model. It reached 0.8853 average cube-placement progress, compared with 0.8229 for the cold start and 0.7006 for chain-of-thought expert iteration.
On 20 out-of-distribution tasks, none of which had a fully successful demonstration, LeAct again achieved the highest partial progress at 0.5777.
The result is too narrow to establish general robotic competence, but it shows that the selection mechanism can carry useful signal beyond mixed-strategy games, even when the oracle supplies a single action rather than a complete policy.
The filter did the real work
When the authors replaced forward-delta ranking with random ranking, the advantage disappeared.
The ablation rules out a simpler explanation: that any extra chain-of-thought data would help. The model improved because it learned from traces that changed its decisions, not because it saw more text.
The authors also tested whether the model was copying the expert’s numerical probabilities into its explanations. Candidate traces without verbatim policy numbers had higher average positive delta than those with copied numbers. And on a large audit, the practical text-based score agreed with a token-level score on the top candidate in 89% to 98% of eligible states.
Why It Happens
The paper treats the missing reasoning trace as a latent variable between the state and the action. The expert reveals the end of the causal chain but not the middle. LeAct searches for a middle that is useful to the student.
Backward generation makes that search easier. Instead of asking the model to reason freely and hoping it reaches the expert action, the model starts from a known-good destination. It can propose several possible routes back to that destination.
Forward scoring then closes the loop. A route survives only if it helps the student travel from the original state to the expert action. This check does not prove that the explanation matches the expert system’s internal mechanism. A game solver may not have any natural-language reasoning to recover. It does establish a narrower and more operational form of usefulness: the trace improves the student’s policy.
The filter also changes as the student improves. An explanation that beats a weak baseline may stop helping after the model learns the underlying pattern. Later rounds therefore admit only traces that add information beyond what the student already knows.
What This Means for Builders
Builders should first look for domains where high-quality action supervision already exists. LeAct does not require replacing human reasoning data everywhere.
A company may have an optimizer that assigns inventory, a planner that schedules jobs, a rules engine that approves transactions, a controller that operates equipment, or a simulator that produces strong policies. Those systems can generate vast numbers of state-action pairs without generating a single sentence. LeAct suggests that these pairs may be raw material for teaching a language model domain-specific reasoning.
The reusable part is the gate:
- Generate multiple candidate explanations.
- Measure each explanation against an external decision signal.
- Keep only those that improve the student’s action.
- Continue targeting the expert decision during training.
Unlike style or plausibility scores, this test uses an external decision signal. It also differs from ordinary distillation because the original teacher does not need to be a language model.
Builders should be careful with the word “reasoning.” The selected traces are useful internal scaffolds for a student policy. They are not automatically faithful explanations of the expert system, and they should not be shown to users as audited rationales without a separate verification layer.
What This Means for Buyers and Operators
Operators evaluating reasoning systems should ask: Does the explanation improve the decision against an external standard? Merely providing an explanation says little about whether it helps.
That framing is useful in procurement and governance. A polished rationale can create confidence without improving reliability. A training pipeline that tests explanations against a solver, simulator, verifier, or controller has a stronger feedback signal.
Organizations with mature operational systems may already own useful AI training data: years of decisions encoded in optimization software, planning rules, simulations, and expert policies. A large archive of prose is not required.
Turning those systems into teachers still requires careful work. The action space must be represented cleanly. The expert must be trusted within a defined scope. The student needs enough candidate diversity for the filter to find useful traces. And operators need an evaluation set that measures more than imitation on familiar states.
What to Watch Next
Scale is the first open test. The experiments use one 8-billion-parameter student. It remains unclear whether frontier-scale reasoning models gain as much from the same procedure or whether their stronger baselines make positive-delta traces harder to find.
The field should also test more consequential expert systems: formal proof checkers, industrial planners, protein-structure tools, scheduling optimizers, and robotic-control libraries. These domains would show whether the method works when actions are more complex, errors are costlier, and the expert is imperfect in ways unlike a game solver.
Another open direction is combining LeAct with online reinforcement learning. The current paper focuses on supervised fine-tuning. An expert policy could also serve as a step-level baseline while the model explores and receives outcome feedback.
Future work also needs to separate three properties that are easy to blur together: predicting the expert action, faithfully describing the expert’s mechanism, and helping a human understand the decision. LeAct provides evidence for the first. The other two need their own tests.
Limitations and Caveats
The evidence is promising but bounded.
Most experiments are games with structured, machine-readable action spaces. BuilderBench adds a simulated robotics setting, but it contains only dozens of tasks and replaces the formal likelihood score with a heuristic action-matching reward.
Best-of-N game evaluation is also oracle-aware: it selects the candidate closest to the expert policy. That is valuable for analyzing whether the model can generate a good decision, but it is not directly deployable when the expert is absent at inference time. The single-sample results are therefore the more conservative operational measure.
The study uses one trained policy per experimental cell. Several error bars reflect repeated inference sampling rather than independent training runs. Broader replication across model families and training seeds would make the estimates more robust.
The paper does not directly compare LeAct with policy-gradient methods such as GRPO. The authors position the work as a supervised fine-tuning and mid-training recipe, but practitioners considering a full training stack still need head-to-head comparisons.
Predictive usefulness is not truth. The generated traces may be student-friendly rationalizations rather than faithful reconstructions of the expert’s process. Requiring a trace to improve the action makes rationalization harder to pass, but it does not eliminate the risk.
Source
Yang, Z., Shi, C., Ghugare, R., Eysenbach, B., Narasimhan, K., & Jin, C. (2026). LeAct: Learning to Reason from Expert Actions. arXiv preprint arXiv:2607.21856. Available at: https://arxiv.org/abs/2607.21856