{"id":"214ab78e-394a-4e25-a849-87338f9b0b9c","arxiv_id":"2412.16878","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RL-SaLLM-F uses an LLM to both label trajectory preferences and generate self-augmented imagined trajectories, achieving MetaWorld success rates comparable to privileged-reward teachers without using any privileged information.","lead":"This paper presents RL-SaLLM-F, a method that replaces the privileged 'scripted teacher' in online preference-based reinforcement learning with feedback from a large language model. The LLM judges trajectory pairs, double-checks its labels, and generates imagined better trajectories to train the reward model, reaching success rates similar to a reward-based teacher on MetaWorld tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim of 'comparable performance' lacks a quantitative final-performance table and is explicitly called 'reference only' in Remark 3, leaving the central claim under-supported.","rationale":"The reader's weakest_assumption targets the physical infeasibility of imagined trajectories (Remark 2). That is a meaningful mechanistic risk, but the empirical success across 8 tasks already provides partial mitigation, and the concern lacks a direct failure mechanism in the reported domains. The more immediate threat to the central claim is evidential: the headline comparison is presented without the statistical reporting needed to verify it. The learning curves in Figure 3 have wide shaded regions, and Remark 3 undercuts the comparison by labeling it 'reference only'. Providing a final performance table with per-seed data and significance tests would settle whether the claim holds. I therefore partially agree with the reader: I share their concern about the missing performance table, but I do not elevate the physical-infeasibility issue above this missing support. The existing conditional verdict is appropriate.","tokens_in":22090,"tokens_out":7608,"duration_ms":69402,"concrete_test":"Request per-seed final success rates for RL-SaLLM-F, PEBBLE, and SAC on all 8 tasks; compute mean and standard deviation, and run a paired bootstrap or Wilcoxon signed-rank test comparing RL-SaLLM-F to PEBBLE on each task. If RL-SaLLM-F is not significantly worse on the majority of tasks and effect sizes are small, the 'comparable' claim is supported; otherwise the abstract should be revised to a weaker claim such as 'comparable on a subset of tasks'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract—that RL-SaLLM-F achieves comparable performance to the scripted teacher—rests on Section 6.2, which reports only learning curves over 5 seeds (Figure 3) with no final mean±std table for PEBBLE/SAC. The only numeric table including RL-SaLLM-F (Table 4, Appendix B.5) reports no PEBBLE/SAC columns, and standard errors are large (e.g., Drawer Open 61.9±14.4, Button Press 83.1±19.8). Moreover, Remark 3 explicitly downgrades the comparison to 'reference only' because PEBBLE/SAC use privileged rewards. With 5 seeds, the observed curve separations could easily be within noise, making the unqualified abstract claim difficult to verify. Until per-seed results are released and a non-inferiority test is run against PEBBLE, the empirical basis for 'comparable' is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RL-SaLLM-F, an online preference-based RL method that replaces the 'scripted teacher' with a large language model (GPT-4o-mini). The LLM provides preference labels for trajectory pairs from the replay buffer and additionally generates self-augmented 'imagined' trajectories that better achieve the task goal; a double-check mechanism discards trajectory pairs on which the LLM is order-inconsistent. Experiments on eight MetaWorld tasks claim comparable or better success rates than PEBBLE with scripted-teacher feedback and comparable to SAC with true task rewards, while using no privileged reward information. The paper also identifies a 'query ambiguity' failure mode in LLM-based preference discrimination during early online training.","tokens_in":22227,"tokens_out":6691,"duration_ms":56807,"significance":"If the empirical claims are validated, this work offers a practical direction for online PbRL that requires neither human feedback nor privileged reward information. The idea of using the same LLM both as a preference labeler and as a generator of augmented preference data is novel, and the choice of a lightweight, low-cost model (GPT-4o-mini) strengthens the applicability. The paper provides a public code link and includes several useful additional analyses: comparisons with reward-free VLM/DT baselines, an offline-dataset reward-model evaluation, point-maze experiments, and a cost analysis. However, the central 'comparable performance' claim currently rests only on learning curves without a final numerical table for the scripted-teacher baselines or statistical significance tests, and Remark 3 explicitly downgrades the comparison to 'reference only', which is in tension with the unqualified abstract claim.","major_comments":[{"comment":"The claim that RL-SaLLM-F achieves comparable performance to the scripted teacher is not quantitatively supported. Figure 3 shows only learning curves averaged over 5 seeds, and no final success-rate table with mean and standard deviation is provided for PEBBLE and SAC. Remark 3 explicitly calls the comparison 'reference only', yet the abstract states 'comparable performance' without qualification. Since this is the central contribution, provide a final performance table with per-task mean±std for all methods and a significance test (e.g., paired bootstrap or Wilcoxon signed-rank) to justify 'comparable', or soften the abstract accordingly.","section":"Section 6.2 / Abstract"},{"comment":"The assumption that physically infeasible imagined trajectories are harmless for reward learning is load-bearing but untested. The paper argues that a Markovian reward model makes infeasible states benign, but if the reward model assigns high rewards to states outside the feasible state manifold, the policy could be misled. Add a controlled experiment that constrains generated trajectories (e.g., by checking state-space bounds or using a simple forward model) and compare learning performance; also report whether generated trajectories stay within the range of states observed in real rollouts.","section":"Section 5.3, Remark 2"},{"comment":"The statistical evidence for the headline comparison is insufficient. With only 5 seeds and overlapping standard errors (e.g., Button Press and Drawer Open), the observed curve separations may be within noise. Table 4, which reports numerical results, does not include PEBBLE or SAC columns. Release per-seed results and final aggregate numbers for all baselines, and state clearly whether the observed differences are statistically reliable.","section":"Section 6.2, Figure 3 / Table 4"},{"comment":"Label accuracy is evaluated against the 'scripted teacher', which is itself a privileged-reward proxy, so the accuracy numbers do not directly measure agreement with human preferences. Additionally, the self-augmentation loop uses the same LLM to generate and label trajectories, which could create confirmation bias in the reward model. The offline-dataset experiment in Appendix B.8 is a good first step, but it measures only label accuracy, not policy performance; a human evaluation or a study of reward-model generalization to unseen trajectories would be more persuasive.","section":"Section 6.5, Table 1 / Appendix B.8"}],"minor_comments":[{"comment":"The text says 'as shown in Figure 7' when referencing the bar-chart comparison of trajectory rewards; the correct reference is Figure 6.","section":"Section 6.6"},{"comment":"The double-check condition is specified only for the cases y=0, y_inv=1 and y=1, y_inv=0; it is unclear whether the case y=0.5, y_inv=0.5 (consistent 'equal preference' responses) is treated as valid or as a discard.","section":"Section 5.2"},{"comment":"The sentence 'Apart from the header, the first four rows' should read 'Apart from the header row'.","section":"Section 6.5"},{"comment":"The label 'Onput 3' appears to be a typo for 'Output 3'.","section":"Appendix C.2"},{"comment":"The caption says 'Training curves of PEBBLE with LLM feedback' but the plot shows both LLM query accuracy and ground-truth episode rewards; consider clarifying the axes in the caption.","section":"Figure 1"},{"comment":"The notation for the k-th nearest neighbor in the intrinsic reward is not defined; please clarify the distance metric and the value of k.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript carries the AAMAS 2025 proceedings header, so the editors may wish to confirm that this submission is a substantially extended version and that the code link points to an available repository (Appendix C.1 says 'we will open-source the entire code', implying the code is not yet public). The main empirical claim needs stronger quantitative support before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper introduces a genuinely new mechanism—using an LLM to generate imagined 'better' trajectories that are then used as preference pairs to train the reward model. That is a real departure from prior LLM/VLM-feedback work, which only judges given pairs. The double-check swap is simple but effective, and the ablations show each piece helps. The cost analysis with GPT-4o-mini is practical and honest.\n\nWhat I like: the paper is candid about its own limitations. Remark 3 explicitly downgrades the PEBBLE/SAC comparison to 'reference only' because those baselines use privileged rewards. Remark 4 notes that the learned reward deviates from the predefined reward yet still yields better success. That is the kind of honesty that makes me trust the authors' interpretation of their own ablation results.\n\nThe soft spots are real but addressable. The central claim—'comparable to scripted teacher'—rests entirely on learning curves over five seeds with large standard errors (e.g., Drawer Open 61.9±14.4, Button Press 83.1±19.8). There is no final performance table with per-seed numbers or a non-inferiority test. So the abstract overstates what the evidence shows. Also, label accuracy is computed against the scripted teacher, which is itself a proxy, and the double-check discards ~35% of queries; that selection bias is not analyzed. The biggest theoretical worry is Remark 2: imagined trajectories are allowed to violate physics, and the paper only argues Markovian rewards make this safe. No experiment directly tests whether infeasible augmentations hurt or help the reward model. That is the weakest load-bearing assumption.\n\nCode is linked but not verified; if the code page is live, that helps reproducibility, but it wasn't available at review time.\n\nOverall: the method is coherent, novel, and honestly reported. The evidence supports 'self-augmentation helps' but not yet 'effectively replaces scripted teacher.' That is a fixable gap. I'd send it to review; the authors should be asked to release per-seed results, add a statistical comparison, and test the infeasible-trajectory assumption directly.","headline":"The self-augmented feedback idea is genuinely new and worth a look, but the headline claim of matching a scripted teacher is not yet backed by numbers.","tokens_in":22782,"tokens_out":2252,"would_cite":true,"duration_ms":49356,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-augmented feedback from the cheap GPT-4o-mini LLM can replace the privileged 'scripted teacher' in online preference-based RL, matching its success rates on MetaWorld tasks.","keywords":["Online preference-based reinforcement learning","Self-augmented LLM feedback","LLM-driven feedback","Query ambiguity","Reward learning from preferences","Large language models","Robotic manipulation","MetaWorld"],"falsifier":"Take a trained RL-SaLLM-F reward model and compare its scores on a physically feasible expert trajectory and an LLM-generated trajectory that teleports the object to the target in one step; if the impossible trajectory receives a reward as high as or higher than the feasible one, the Markovian assumption is introducing bias. A direct experiment would train the same pipeline with deliberately infeasible but goal-reaching imagined trajectories and measure whether success rates collapse.","tokens_in":21883,"feed_emoji":"🤖","tokens_out":6970,"duration_ms":57853,"temperature":0.7,"pith_summary":"The paper sets out to remove the two crutches of online preference-based reinforcement learning: the 'scripted teacher' that labels trajectory pairs by comparing privileged task rewards, and real-time human feedback that is impractical to obtain during training. It proposes RL-SaLLM-F, in which a large language model both judges which of two recent trajectories is closer to the task goal and imagines a better trajectory to use as extra preference data. The paper identifies 'query ambiguity' — early replay-buffer trajectories are so similar and suboptimal that LLM labels are barely above chance — and counters it with a double-check that swaps trajectory order and discards inconsistent labels, plus self-augmented imagined trajectories. On eight MetaWorld manipulation tasks, RL-SaLLM-F with GPT-4o-mini reaches success rates comparable to PEBBLE's scripted teacher and, on some tasks, matches SAC trained on the true reward. The upshot is a recipe for online PbRL that needs no privileged reward information and no human in the loop.","feed_headline":"LLM feedback matches hidden-reward training on MetaWorld tasks","feed_subtitle":"Self-augmented imagined trajectories let online preference-based RL skip human and privileged reward labels.","key_machinery":"The load-bearing mechanism is the preference dataset built from two kinds of LLM output. First, sampled trajectory pairs are converted into coordinate text and judged by the LLM; a double-check query with the two trajectories swapped retains a label only if both answers agree. Second, the LLM is prompted to generate an imagined trajectory that improves on the better of the two sampled trajectories while sharing its initial state, and the imagined trajectory is stored as preferred over the real one. These triples train a Bradley-Terry reward model, which relabels the replay buffer for an off-policy SAC agent. The imagined trajectories are admitted even when physically infeasible, on the assumption that the reward model is Markovian and therefore only needs the sequence of states for comparison, not for execution.","core_discovery":"The paper's central claim is that self-augmented LLM feedback can replace scripted-teacher feedback in online PbRL without sacrificing task performance. Concretely, RL-SaLLM-F is claimed to achieve comparable success rates to a scripted teacher with privileged reward information across eight MetaWorld tasks using only the lightweight GPT-4o-mini, while using no predefined reward and no human interaction. The paper also claims that the double-check mechanism lifts LLM label accuracy from about 58% to about 65%, and that adding LLM-generated imagined trajectories lifts it to about 72% while being the single largest contributor to policy success. It further claims that the imagined trajectories need not obey physical constraints, because they are used only to train a Markovian reward model through preference comparisons and never to drive the policy or a world model.","pith_inferences":["Editorial inference: the Markovian-reward justification implies that the self-augmentation channel would work with any generative trajectory source, not just an LLM; the paper's own Decision-Transformer comparison suggests the benefit depends on whether the generator produces goal-directed trajectories without expert data.","Editorial inference: query ambiguity is likely a property of early suboptimal replay data rather than of LLMs specifically, so the double-check-and-discard rule could serve as a general noise filter for any imperfect preference judge, including humans.","Editorial inference: using the scripted teacher as ground truth caps measured label accuracy; if the LLM's judgments were compared against human preferences instead, the 'label accuracy' numbers would be a different and arguably more meaningful quantity.","Editorial inference: the text-coordinate interface is the main practical constraint, so combining the framework with perception models that extract object coordinates from images, as the paper suggests as future work, is the natural path to vision-based tasks."],"forward_implications":["Online PbRL can be run with no privileged rewards: a single text-promptable LLM supplies both preference labels and extra training data.","Discarding ambiguous preference queries is better than labeling them as ties, since early ambiguous pairs are mostly hallucinations that harm reward learning.","The LLM's generative ability becomes a data-augmentation channel, improving label accuracy indirectly by making the policy sample more diverse, higher-quality trajectories.","The cost-performance tradeoff is controllable: GPT-4o-mini is roughly 20 times cheaper per query than GPT-4o and still matches the scripted teacher, while GPT-4o improves accuracy further.","Because the reward model only consumes state-coordinate text, the approach generalizes across tasks in the same state representation with only the goal description changed, and it transfers to a point-maze navigation benchmark."],"supporting_citations":[{"why":"Supplies the online PbRL pipeline (unsupervised pretraining, replay relabeling, SAC) that RL-SaLLM-F builds on, and the scripted-teacher baseline it aims to replace.","marker":"[16]"},{"why":"Establishes preference-based RL from human feedback, the paradigm this paper extends to LLM-generated feedback.","marker":"[8]"},{"why":"Gives the Bradley-Terry paired-comparison model used to convert preference labels into a reward model.","marker":"[3]"},{"why":"Provides the MetaWorld benchmark and task suite on which all main comparisons are run.","marker":"[37]"},{"why":"Describes the GPT-4 family used as the LLM feedback source for both judging and trajectory generation.","marker":"[1]"},{"why":"The RL-VLM-F baseline that motivates using trajectory text instead of single images for preference feedback.","marker":"[33]"},{"why":"Defines the scripted teacher that compares privileged predefined rewards to label trajectory preferences.","marker":"[15]"},{"why":"Gives the suboptimality guarantee for preference-based reward learning that justifies learning rewards from pairwise comparisons.","marker":"[45]"}],"fun_headline_variants":["LLM feedback replaces hidden-reward teacher in online RL","Self-augmented LLM feedback matches privileged rewards on MetaWorld","No human labels: LLM feedback rivals scripted teacher in PbRL","LLM imagined trajectories unlock online RL without reward engineering","Double-checked LLM preferences boost RL to scripted-teacher level"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the assumption that the learned reward can be safely trained on LLM-imagined trajectories that may violate the environment's physics, so a reward model that is Markovian in state will not be misled by unrealistic states; the paper asserts this and supports it only with final success rates rather than a direct check.","fun_headline_variants_meta":{"raw":{"variants":["LLM feedback replaces hidden-reward teacher in online RL","Self-augmented LLM feedback matches privileged rewards on MetaWorld","No human labels: LLM feedback rivals scripted teacher in PbRL","LLM imagined trajectories unlock online RL without reward engineering","Double-checked LLM preferences boost RL to scripted-teacher level"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1238,"prompt_tokens":979,"completion_tokens":259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":171}},"tokens_in":595,"tokens_out":259,"duration_ms":2936,"temperature":1.0,"reasoning_tokens":171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:00:37.706206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained RL-SaLLM-F reward model and compare its scores on a physically feasible expert trajectory and an LLM-generated trajectory that teleports the object to the target in one step; if the impossible trajectory receives a reward as high as or higher than the feasible one, the Markovian assumption is introducing bias. A direct experiment would train the same pipeline with deliberately infeasible but goal-reaching imagined trajectories and measure whether success rates collapse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the online PbRL pipeline (unsupervised pretraining, replay relabeling, SAC) that RL-SaLLM-F builds on, and the scripted-teacher baseline it aims to replace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MetaWorld benchmark and task suite on which all main comparisons are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The RL-VLM-F baseline that motivates using trajectory text instead of single images for preference feedback."},{"cited_title":"scripted teacher","cited_arxiv_id":null,"evidence_quote":"Gives the suboptimality guarantee for preference-based reward learning that justifies learning rewards from pairwise comparisons."}],"review_version":1}