{"id":"b2ba1ff2-0fba-4738-9ad1-b24b417d38f6","arxiv_id":"2504.13145","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EEF improves LLM agent fine-tuning by extracting beneficial action segments from failed expert trajectories via rollouts from intermediate states, reporting a 62% WebShop win rate and a ScienceWorld score above 81.","lead":"This paper proposes EEF, a method that mines failed expert trajectories for useful action segments and adds them to the fine-tuning data of LLM agents. It reports new state-of-the-art results on WebShop and ScienceWorld, though the algorithm description is internally inconsistent and the experiments lack error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 never trains on actions from failed expert trajectories; the claimed EEF mechanism is absent from the reported experiments.","rationale":"The reader's weakest_assumption targets the validity of the prefix-labeling rule: success from s_l does not prove a0:l-1 caused the success. That is a legitimate concern, but it is downstream. The more direct problem is that Algorithm 1, as written, never builds any training example containing those expert actions. Lines 7–10 add rollouts sampled from intermediate expert states; the SFT step in line 14 only sees positive trajectories from D+. A rollout from s_l does not include the expert prefix. Thus the paper's central contribution—learning from beneficial actions in failed expert trajectories—is not represented in the algorithm. The Section 3.2 prose attempts to describe learning prefix actions, but the formal definitions and the example (τ_{sl} as a solution for s0) are inconsistent with that reading. This makes the experimental SOTA claim ambiguous: it may simply demonstrate that adding rollouts from expert-guided states to RFT improves performance, a different and less novel method. The concern is addressable by code inspection or a corrected algorithm, hence conditional rather than reject. I partially agree with the reader: they noted the algorithm mismatch in their rationale, but their stated weakest assumption is the labeling rule, which is secondary.","tokens_in":20690,"tokens_out":5553,"duration_ms":53056,"concrete_test":"Inspect the released training code (or add logging to Algorithm 1) to enumerate every action token used as an SFT label in each iteration, and check whether any token from a failed expert trajectory's prefix appears. If the training labels consist only of current-policy rollouts from expert states and from s0, the core 'beneficial expert actions' mechanism is not implemented and the SOTA claim needs to be reframed as RFT with additional intermediate-state exploration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EEF 'identifies beneficial actions from failed expert trajectories and integrates them into the training dataset.' Algorithm 1 does not do this. Lines 7–9 sample rollouts πθ(·|s_{m·l}) from expert intermediate states, and line 10 adds only positive rollouts to D+. Line 14 trains on Ds0∪Dr, where get_traj(s,D+) returns a positive rollout containing s. A rollout generated from s_l starts at s_l, so it cannot contain expert prefix a0:l-1; the SFT loss in line 14 is computed only over action tokens in those policy rollouts. The text's claim that 'we train the agent policy on those beneficial actions (green arrows)' (Figure 1) and the Section 3.2 statement that selected solutions require learning expert actions [a0,...,a_{l-1}] have no corresponding implementation. The example calling τ_{sl} a 'solution for s0' also contradicts the formal definition of a solution as a trajectory containing s0. Consequently, even if the labeling rule were correct, the experiments could not support it: the reported gains could come entirely from extra exploration at intermediate expert states rather than from learning beneficial actions in expert failures. This mismatch is the load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Exploring Expert Failures (EEF), an extension of Rejection Sampling Fine-Tuning (RFT) for LLM agents. The method is described as identifying beneficial actions inside failed expert trajectories by simulating the current policy from selected expert intermediate states, then training on the expert actions that lead to successful simulations while excluding harmful ones. The authors claim that this enables solving previously unsolvable subtasks and report state-of-the-art results on WebShop and ScienceWorld, including a 62% win rate on WebShop-11k and rewards above 0.81 and 81 on the two benchmarks. The paper includes ablations, case studies, and an efficiency analysis comparing EEF with RFT and other baselines.","tokens_in":20896,"tokens_out":7887,"duration_ms":73520,"significance":"The motivating idea is practically valuable: failed expert trajectories are abundant and cheap, and mining them without inheriting their errors could mitigate the simplicity bias of RFT in sparse-reward agentic environments. The reported results, if reproducible, would be a meaningful advance, and the use of weaker, cheaper experts (GPT-3.5) is an attractive feature. The paper also provides concrete case studies and a navigation-skill analysis that support the intuition that EEF helps acquire non-trivial skills. However, the central mechanism as described in the text and Figure 1 is not implemented in Algorithm 1, the labeling rule that certifies actions as beneficial is unvalidated, and the empirical results lack error bars and a fixed checkpoint-selection protocol. These issues are load-bearing because they directly affect whether the experiments support the paper's main claim. The contribution is therefore promising but not yet established.","major_comments":[{"comment":"Algorithm 1, lines 7–10 and 14, does not train on the claimed 'beneficial actions' from failed expert trajectories. Line 9 samples policy rollouts from expert intermediate states s_{m×l}; line 10 adds only positive rollouts to D+. In the subsequent training step, get_traj(s,D+) returns a positive trajectory containing state s, so a rollout starting at s_l contains s_l and the policy's own subsequent actions, not the expert prefix a_{0:l-1}. The SFT loss at line 14 is therefore computed only on policy-generated action tokens from the simulated rollouts. The statement in Section 3.2 that 'the agent only needs to learn the expert actions [a_0,...,a_{l-1}]' and the green arrows in Figure 1 have no corresponding implementation. Similarly, calling τ_{s_l} a 'solution for s0' contradicts the formal definition of a solution as a positive trajectory containing s0, since τ_{s_l} does not contain s0. Because of this mismatch, the experiments cannot support the central claim: the reported gains could come entirely from extra exploration starting at expert states, rather than from learning beneficial actions inside failed expert trajectories. This gap must be resolved either by changing the algorithm so that the selected expert prefix is actually included in the training set, or by reframing the contribution and claims to match what is implemented.","section":"Section 3.2, Algorithm 1, Figure 1"},{"comment":"The rule that a successful rollout from expert state s_l certifies the preceding expert actions a_{0:l-1} as beneficial for the original state s0 is an unproven assumption, and the symmetric failure-based rule for identifying recovery states is likewise unvalidated. Success from s_l could be an artifact of s_l being an easy state or of the policy's own later choices; failure from s_i after success from s_{i-l} does not necessarily attribute the failure to the intervening expert actions. No experiment tests this certificate directly, for example by comparing the selected action segments with random segments or by ablating the selection rule. Since this certificate is supposed to distinguish EEF from merely adding extra rollouts, it needs direct empirical support before the main claim can be accepted.","section":"Section 3.2, important state selection"},{"comment":"All reported win rates and rewards in Table 3 are single numbers with no error bars, no number of seeds, and no measure of variance. The text says 'We select the best model from all iterations' but does not specify the number of checkpoints, the validation criterion, or how many iterations were considered. The headline differences (e.g., 58.4% vs. 53.6% on WebShop-11k) may be within run-to-run variance or may reflect best-checkpoint selection over many fine-tuned models. The paper should report mean and standard deviation over at least three seeds and fix a single checkpoint-selection rule. In addition, Section 4.1 states 'The default parameters are M = 5, I = 4', while Table 2 lists I = 3 for both EEF rows; this inconsistency must be resolved.","section":"Section 4.1 and Table 3"}],"minor_comments":[{"comment":"The title contains a spacing typo ('T uning') and Section 2 is headed 'Backgroud'; please proofread the manuscript.","section":"Title and Section 2"},{"comment":"The notation τ_{s_l} is used ambiguously: it sometimes denotes a rollout generated from s_l and sometimes a trajectory that includes the expert prefix before s_l. Please define a single consistent notation and use it throughout.","section":"Section 3.1 and Figure 1"},{"comment":"The function 'need recover states(Di)' is not formally defined. The text describes recovery states as expert states, but the argument Di is the set of policy rollouts; please specify how the function identifies states from Di and how it relates to the expert trajectory states.","section":"Algorithm 1, line 12"},{"comment":"For EEF GPT-3&4, Table 2 lists Sim. # = 11, but the text says M=5 for expert states; please explain how the eleven simulations are budgeted across initial states, expert states, and GPT-3.5 trajectories.","section":"Table 2"},{"comment":"The claim of being 'the first method to surpass a score of 0.81 in WebShop' should be justified against published state-of-the-art numbers, not only against the baselines in Table 3; also, the abstract's SciWorld claim should state explicitly which EEF variant supports it, since the GPT-3&4 row in Table 3 is left as '–' for ScienceWorld.","section":"Abstract and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the mismatch between the described mechanism and Algorithm 1. If the authors' actual implementation does train on expert prefixes from failed trajectories, then the pseudocode and Section 3.2 must be rewritten to match; if it does not, then the contribution and claims must be reframed substantially. I would ask the authors to address this point head-on before the paper is reconsidered. The empirical section also needs a reproducibility pass (seeds, error bars, checkpoint selection). The paper has a useful motivation and plausible direction, but as submitted the main claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take. The paper says EEF learns beneficial actions from failed expert trajectories. The idea is genuinely interesting: use rollouts from intermediate expert states to label which expert actions were useful, then train on those labels. But Algorithm 1 doesn't do that. Lines 7–9 sample rollouts from expert states; line 10 stores successful rollouts; line 14 trains on trajectories containing the important state. A rollout from s_l starts at s_l, so it contains no expert actions before s_l. Therefore the green arrows in Figure 1 are never used as training labels. For hard subtasks, there is no successful trajectory containing s0 at all, so the initial-state branch also fails to pick up the expert prefix. As written, the method is 'explore from intermediate expert states and imitate successful rollouts.' That may be a fine exploration trick, but it is not 'learning from expert failures,' and that mismatch is load-bearing.\n\nWhat deserves credit: the motivating observation is plausible and the experiments are consistent. The gains over RFT and ETO are real-looking; the navigation-skill analysis in Figure 2 is a good diagnostic; and the GPT-3.5 data point is a nice cost-effective addition. The paper is also honest about its use of win/loss feedback only.\n\nThe other soft spots are smaller but real: no error bars or multiple seeds; 'we select the best model from all iterations' is not a reproducible rule; the definition of a solution (a positive trajectory containing s) is contradicted by the example calling τ_sl a solution for s0; and the efficiency plot in Figure 3 doesn't specify seeds or equal compute in a way that supports the curve. None of these alone would sink the paper.\n\nBottom line: the central mechanism is not implemented as described. The authors could fix this by concatenating the expert prefix with a successful rollout from the intermediate state and training on that prefix, or by reframing the contribution as intermediate-state exploration. Either way they need to re-run experiments. I'd send this to serious peer review because the direction is important and the results are suggestive, but I would not accept it in current form.","headline":"The idea is right, but Algorithm 1 doesn't train on any action from failed expert trajectories, so the headline results don't currently support the claimed mechanism.","tokens_in":21447,"tokens_out":6506,"would_cite":false,"duration_ms":57953,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Failed expert trajectories, mined by re-simulating from intermediate states, train LLM agents to solve hard subtasks that rejection-sampling fine-tuning leaves unsolved.","keywords":["LLM agents","agent fine-tuning","rejection sampling fine-tuning","negative trajectories","beneficial action mining","state simulation","WebShop","ScienceWorld"],"falsifier":"Take a set of failed expert trajectories, split each at sampled states, and scramble which expert action prefixes are paired with which successful rollouts; if scrambled EEF training retains the same win-rate gains on WebShop or ScienceWorld, then the success-from-state signal is not actually selecting expert-specific beneficial actions, and the claimed mechanism would be falsified.","tokens_in":20452,"feed_emoji":"🤖","tokens_out":7749,"duration_ms":69212,"temperature":0.7,"pith_summary":"Rejection-sampling fine-tuning (RFT) improves LLM agents by imitating successful expert and self-generated trajectories, but because experts and students both succeed mostly on easy subtasks, hard subtasks stay out-of-distribution. The paper claims those hard subtasks are not actually unsolvable: failed expert trajectories often contain correct plans and key actions, with the failure coming only from late mistakes. The proposed method, EEF, identifies the good parts by restarting the current policy from sampled intermediate states of failed expert trajectories; when the policy succeeds from a state, the expert actions immediately before it are treated as beneficial and added to the training set, while other actions are discarded. On WebShop, EEF reaches a 62% win rate and a 0.81 reward with GPT-3.5 plus GPT-4 data, surpassing RFT and GPT-4, and it is the first reported method to exceed 81 on ScienceWorld. If correct, this turns rejected negative data into a reusable resource instead of waste.","feed_headline":"Failed expert actions train better LLM agents","feed_subtitle":"Reusing failed expert runs lifts WebShop past 0.81 and ScienceWorld past 81.","key_machinery":"The load-bearing object is the expert-state simulation protocol used to label beneficial actions. Given a failed expert trajectory $\\tau_e=[s_0,a_0,s_1,\\ldots]$, EEF picks states at equal skip length $l=\\lfloor|\\tau_e|/(M+1)\\rfloor$, rolls out the current policy from each selected state, and compares outcomes: if the policy succeeds from $s_{i-l}$ but fails from $s_i$, the actions $a_{i-l:i-1}$ are classified as harmful and $s_i$ is marked a recovery state; if the policy succeeds from $s_i$, the preceding expert actions are treated as beneficial prefixes. Important states are initial subtask states plus the first such recovery state in each trajectory, and from the stored positive-trajectory repository EEF selects at most one solution per important state, training the model only on actions after that state with the supervised fine-tuning loss. This simulation signal is what converts negative expert data into positive training segments and what the authors argue prevents contamination from suboptimal actions.","core_discovery":"The paper's central claim is that a failed expert trajectory should not be discarded as a unit: although the whole trajectory has zero reward, it can be partitioned into beneficial action segments and harmful ones, and supervised fine-tuning on only the beneficial segments lets a smaller LLM agent solve subtasks that rejection-sampling fine-tuning cannot. The supporting evidence is empirical: EEF-style training solves previously unsolved WebShop subtasks and lifts win rates from 35.6% for GPT-4 and 53.6% for RFT to 58.4% with GPT-4 data alone and 62.0% with additional GPT-3.5 data, with reward passing 0.81; on ScienceWorld, the score exceeds 81. The paper also claims that because harmful actions are excluded, EEF can learn from weaker, cheaper experts such as GPT-3.5 Turbo without being dragged down by their mistakes.","pith_inferences":["My inference: the same success-versus-failure simulation signal could be turned into an automatic curriculum, starting the policy from states it already solves and pushing toward states just beyond its reach; the paper does not develop this framing.","My inference: the equal-interval sampling is a coarse credit-assignment heuristic, so a binary-search variant that isolates the exact harmful action boundary should extract more signal per failed trajectory.","My inference: if the labeling rule holds for expert failures, it should also hold for failures of the student policy, which would make the method self-improving rather than dependent on a stronger expert."],"forward_implications":["Hard, persistently out-of-distribution subtasks become learnable, so expert failure trajectories are a resource rather than noise.","Training on a few selected beneficial actions instead of whole trajectories should reduce exposure to suboptimal behavior and may improve sample efficiency relative to whole-trajectory imitation.","The same setup can absorb demonstrations from cheaper, weaker experts, shifting the data budget toward low-cost exploration.","Agents learn navigation and recovery skills—using Next and Back in WebShop—that both RFT and GPT-4 attempt but fail to deploy successfully."],"supporting_citations":[{"why":"Defines rejection-sampling fine-tuning, the framework EEF modifies and the main baseline it must beat.","marker":"Yuan et al. (2023)"},{"why":"GPT-4 is the expert whose failed and successful trajectories are mined and whose win rate serves as a baseline.","marker":"Achiam et al. (2023)"},{"why":"Introduces WebShop, the primary benchmark for the reported win-rate and reward results.","marker":"Yao et al. (2022)"},{"why":"Introduces ScienceWorld, the long-horizon benchmark where EEF exceeds score 81.","marker":"Wang et al. (2022)"},{"why":"Supplies the ScienceWorld expert demonstration trajectories used for training.","marker":"Xi et al. (2024)"},{"why":"Documents the simplicity bias in rejection sampling that motivates targeting hard subtasks.","marker":"Tong et al. (2024)"},{"why":"ETO is the exploration-based fine-tuning baseline EEF is compared against and contrasted with on negative data.","marker":"Song et al. (2024b)"},{"why":"NAT is the negative-aware baseline that treats all actions in failed trajectories as uniformly bad, which EEF improves on.","marker":"Wang et al. (2024a)"},{"why":"Provides the WebShop 11k dataset used in the large-scale experiments.","marker":"Ma et al. (2024)"},{"why":"Defines the Llama 3 8B initial model used in all fine-tuning runs.","marker":"AI@Meta (2024)"}],"fun_headline_variants":["Failed expert steps become training wins for LLM agents","Mining expert failures improves LLM agent tuning","LLM agents beat GPT-4 by learning from its mistakes","Reusing failed expert actions lifts agent benchmarks","From expert flops to agent top scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a successful rollout of the current policy from an expert intermediate state certifies the expert actions just before that state as beneficial for the original starting state; that inference would fail if the intermediate state is easy or the policy's own later choices are what produce success.","fun_headline_variants_meta":{"raw":{"variants":["Failed expert steps become training wins for LLM agents","Mining expert failures improves LLM agent tuning","LLM agents beat GPT-4 by learning from its mistakes","Reusing failed expert actions lifts agent benchmarks","From expert flops to agent top scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2989,"prompt_tokens":1003,"completion_tokens":1986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1913}},"tokens_in":619,"tokens_out":1986,"duration_ms":13632,"temperature":1.0,"reasoning_tokens":1913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:13:44.251627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of failed expert trajectories, split each at sampled states, and scramble which expert action prefixes are paired with which successful rollouts; if scrambled EEF training retains the same win-rate gains on WebShop or ScienceWorld, then the success-from-state signal is not actually selecting expert-specific beneficial actions, and the claimed mechanism would be falsified.","supporting_citations":[],"review_version":1}