{"id":"a87e98a3-cf3f-449c-a9cd-09022813521d","arxiv_id":"2504.16408","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reward-filtered distillation from 24 labeled examples improves structured reasoning F1 for fine-tuned Llama-3-8B agents on the LLMSR@XLLM25 test sets.","lead":"This paper describes a competition system that builds training data from just 24 labeled examples by using GPT-4o to create more examples and a reward model to keep only the good ones. The authors report that filtered data improves the system's structured reasoning scores, but their comparison mixes quality with dataset size.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward-model validity is unvalidated; the appendix's own example gives 'True' to a statement its evidence contradicts, yet the reward scores are high, so Table 2's 'quality' gains may be an artifact of the proxy.","rationale":"The paper's headline claim ('quality, not scale') is an interpretation of Table 2, where four filtering strategies are compared. The reader already flagged reward validity and dataset-size confounds. I focus on the former because it is the load-bearing premise of the 'quality-guided distillation' narrative and because the paper itself supplies a counterexample: Appendix A's trace contains a logically wrong verification label that the reward model scores highly. This is not an external objection but an internal inconsistency between the claimed quality signal and the provided artifact. If the reward proxy is unreliable, the entire mechanism—'reward-guided filtering ... ensures semantic fidelity'—is unsupported, and the performance gains may be due to correlated stylistic factors. The proposed audit directly tests the proxy by measuring whether reward-selected data is genuinely more correct. I maintain the reader's CONDITIONAL verdict: the paper could be accepted if the authors supply this evidence (and ideally matched-size baselines and error bars), but as is, the central claim overreaches.","tokens_in":7398,"tokens_out":4726,"duration_ms":45354,"concrete_test":"Audit the distilled data: randomly sample 200 traces from each of the four filtered sets (Structure, 0-shot, 5-shot, Avg). Independently annotate each verification label as correct/incorrect relative to its evidence (two annotators, report Cohen's kappa). Compare correctness rates across sets and correlate reward scores with human labels within each set. If the Avg-reward set does not have a significantly higher verified-correct rate than Structure-filtered (or if reward scores do not correlate with correctness), the quality-proxy assumption fails and the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Sec. 4.2: 'high-quality supervision—not scale alone—drives generalizable reasoning') rests on the reward model's score being a faithful measure of reasoning quality. No validation of this proxy is provided. The one illustrative trace in Appendix A, Table 3, undercuts it: the statement 'The group leader knows the most...' is labeled 'Verification: True' with evidence 'The leader's acquaintanceship count is not detailed; no inference about hierarchy can be conclusively formed.' That is a contradiction—if no inference can be formed, the label should be False (or Unknown). The reward model assigns this trace high scores (5-shot 1.87, 0-shot 2.28, avg 2.08), and it survives filtering. This demonstrates that the reward signal can pass demonstrably incorrect verification labels. If reward scores are not tied to logical correctness, the performance ordering in Table 2 (e.g., Reason. F1 5.20 → 8.98) may be selecting for fluency, verbosity, or format rather than reasoning quality. The claim that quality rather than quantity drives performance then lacks its key premise. (Sec. 2.3 Eqs. 8–11; Sec. 4.2.)","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes Less is More, the authors' third-place approach to the LLMSR@XLLM25 shared task. Given only 24 labeled examples, the method induces task prompts through reverse thinking, synthesizes reasoning traces for unlabeled LogiQA questions via retrieval-augmented in-context learning with GPT-4o, and then filters these traces using a two-stage process: structural pruning followed by reward-model scoring under few-shot, zero-shot, and averaged configurations. Three task-specific LLaMA-3-8B-Instruct models (question parser, CoT decomposer, verifier) are fine-tuned with LoRA+ on the filtered data. The central empirical claim, made in Section 4.2, is that reward-guided filtering improves structured reasoning metrics over structure-only filtering, and that this demonstrates quality rather than quantity drives performance in low-resource reasoning. The paper includes internal ablations across filtering strategies and reports gains in Question F1, Statement F1, Evidence F1, and Reasoning F1.","tokens_in":7637,"tokens_out":3453,"duration_ms":35805,"significance":"If the central claim holds, the paper makes a useful practical contribution: it shows a concrete, reproducible recipe for turning a very small labeled seed into a larger but carefully filtered training set for structured reasoning, with code released. The modular multi-agent design and the unified LoRA+ fine-tuning setup are clean, and the comparison across reward-filtering configurations is a sensible ablation idea. The claim that data quality matters more than scale is consistent with a growing body of work on data selection, and the low-resource setting (24 labeled examples) is practically relevant. The paper is also transparent about training-set sizes and provides an illustrative example, which is helpful for understanding the data format. However, the empirical support for the headline claim is currently incomplete: the main comparison confounds filtering with dataset size, there are no variance or significance estimates, and the single illustrative example in Appendix A contains a verification label that contradicts its own evidence while receiving high reward scores.","major_comments":[{"comment":"The illustrative example that the authors chose to showcase contains a direct internal contradiction. The statement 'The group leader knows the most in the group, and the others know each other less' is labeled 'Verification: True', yet its own evidence reads 'The leader's acquaintanceship count is not detailed; no inference about hierarchy can be conclusively formed.' If no inference can be formed, the appropriate label is False or Unknown, not True. This trace receives high reward scores (5-shot 1.87, 0-shot 2.28, average 2.08) and survives the filtering described by Eqs. (8)-(11). Since reward-based filtering is the mechanism that the paper credits for the quality gains in Table 2, this example undercuts the assumption that the reward model is a faithful proxy for reasoning correctness. The authors should either correct the example, provide evidence that such contradictions are rare, or validate the reward model against human judgments of logical correctness.","section":"Appendix A, Table 3; Sec. 2.3"},{"comment":"The central comparison in Table 2 does not isolate quality from quantity. The structure-filtered condition retains 1,940 traces, while the reward-filtered conditions retain 1,309-1,377 traces (Table 1). The performance differences could therefore reflect not higher-quality supervision but the removal of hard or out-of-distribution examples from the training set, making the remaining data easier to learn. The sentence in Sec. 4.2 stating that the setup 'isolates the impact of training data quality alone' is not supported by the design. A size-matched baseline (e.g., random or difficulty-matched subsets of the structure-filtered data at the same sizes) and ideally multiple random seeds with error bars would be needed to support the 'quality, not scale' claim.","section":"Sec. 4.2, Tables 1 and 2"},{"comment":"The paper reports several F1 metrics (Question F1, Statement F1, Evidence F1, Reasoning F1) but does not define them. Since the entire quantitative claim rests on these numbers, the authors should state the exact computation: at which granularity (question-level, step-level, or span-level) each metric is computed, how the shared task defines them, and whether macro-averaging or micro-averaging is used. Without these definitions, the reader cannot assess whether the reported gains are meaningful or whether the metrics are computed consistently across conditions.","section":"Sec. 4, Tables 1 and 2"},{"comment":"The reward threshold S(x) > 0 is introduced without any sensitivity analysis. The paper reports results for three filtering strategies based on different definitions of S(x), but never varies the threshold itself. Since the threshold determines the size and composition of the filtered set, the claim that the average configuration is best would be stronger if the authors showed that the relative ordering of strategies is stable across reasonable thresholds (e.g., S(x) > 0, S(x) > 0.5, or percentiles of the reward distribution).","section":"Sec. 2.3, Eq. (11)"}],"minor_comments":[{"comment":"There are several typos: 'diffrent' should be 'different' and 'stratgies' should be 'strategies' in the final paragraph of Section 1.","section":"Introduction"},{"comment":"The figure labels contain typos: 'Question Paring' should be 'Question Parsing' and 'CoT Paring' should be 'CoT Parsing'.","section":"Figure 1"},{"comment":"The model version used for synthesis is given as 'gpt-4o-2024-08-06' in a footnote, but the body text refers only to 'GPT-4o'. It would be helpful to state the temperature and any other decoding parameters in the main text, since they affect the quality of the synthesized data.","section":"Sec. 2.2"},{"comment":"The paper states that all models are trained for 5 epochs with a batch size of 4 and gradient accumulation of 4, but does not report the total number of training steps or the actual batch size per model. Since the training sets differ in size, the number of optimizer steps differs across conditions, which is another potential confound; reporting the number of steps would help the reader assess this.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid system description for a shared-task submission, and the authors are refreshingly explicit about their pipeline and data sizes. The main concern is that the headline empirical claim ('quality, not scale') is supported only by a confounded comparison and a single illustrative example that undermines the reward model's validity. These issues are fixable with additional experiments and validation, so I do not recommend rejection. The paper would benefit from a human-validated subset of the filtered data and from size-matched baselines. I also note that the authors cite several 'Less is More' papers; this is appropriate given the theme, but the novelty claim should be positioned carefully relative to those works."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid shared-task system description. The authors combine RA-ICL, reverse prompting, reward filtering, and LoRA+ into a modular multi-agent pipeline that took third in LLMSR@XLLM25, and they compare three reward-filtering strategies on a public test set. Code is released, training-set sizes are reported, and the evaluation is on the shared task's official test split. That part is genuinely useful for anyone building low-resource reasoning systems.\n\nThe soft spot is the headline claim. The paper says reward-guided filtering improves Reasoning F1 by 3.78 points and concludes that high-quality supervision, not scale alone, drives performance. That conclusion needs more support. The reward model is a public general-purpose model trained on Reward-Bench, not on logical verification, and the paper gives no evidence that its scores track reasoning quality. In fact, the appendix's own example undercuts it. Table 3 shows the statement \"The group leader knows the most in the group\" labeled \"Verification: True\" with the evidence \"The leader's acquaintanceship count is not detailed; no inference about hierarchy can be conclusively formed.\" That is an internal contradiction—no inference, yet True. The reward scores for that trace are high (avg 2.08), so the filter would keep it. So the gains in Table 2 may come from selecting more fluent or format-compliant traces, not from better reasoning.\n\nSecond, the comparison is confounded: the filtered sets are smaller than the structure-filtered baseline (1,346 vs 1,940). The authors interpret smaller-but-better as evidence for quality, but they never run a matched-size control with a random subset. Without that, \"less is more\" is not cleanly isolated. Third, there are no error bars or multiple seeds, so the 3.78 point gain could be noise.\n\nNone of this makes the paper useless. As a system description it is clear and reproducible, and the comparison of filtering strategies is a decent empirical contribution. But the authors should tone down the \"quality not quantity\" claim, add a random-subset baseline and multiple runs, and audit the distilled data with basic human checks. The appendix contradiction is easy to fix and should be fixed.\n\nI'd send this to peer review—it deserves a serious referee—but the referee should require those controls before publication. It's worth a read for people working on structured reasoning under extreme data scarcity.","headline":"A competent shared-task system with a clean ablation, but the 'quality not quantity' claim is overreached: the reward proxy is unvalidated, the comparison is confounded by dataset size, and the appendix shows the reward model passing a logically wrong label.","tokens_in":8112,"tokens_out":4091,"would_cite":false,"duration_ms":39590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that reward-guided filtering—especially averaging few-shot and zero-shot reward scores—improves structured reasoning under 24-example supervision, and that data quality, not scale, drives the gain.","keywords":["structured reasoning","low-resource learning","reward-guided filtering","data distillation","multi-agent reasoning","chain-of-thought","question parsing","logical reasoning"],"falsifier":"Compare the average-reward-filtered set of 1,346 traces against 1,346 traces sampled uniformly at random from the structure-filtered pool and fine-tune identical models on each. If the random sample matches or exceeds the reward-filtered set on Reasoning F1 and Statement-Evidence F1, the paper's quality-over-quantity explanation is falsified. A second check: measure trace length and lexical complexity before and after filtering; systematic shortening would indicate the reward model selects for simplicity.","tokens_in":7216,"feed_emoji":"🧩","tokens_out":8796,"duration_ms":75081,"temperature":0.7,"pith_summary":"The paper takes a structured reasoning task with only 24 labeled examples and builds a pipeline that turns unlabeled logic questions into many candidate chain-of-thought traces, then filters those traces by a reward model before fine-tuning three specialist reasoning agents. The central claim is that this reward-guided filtering is what drives performance: switching from structure-only filtering to average-reward filtering improves Reasoning F1 by 3.78 percentage points, Statement-Evidence F1 by 4.76, and Statement Macro F1 by 3.41 under otherwise identical training settings. A sympathetic reader should care because the result challenges the usual reflex to collect more data: if the claim holds, scarce-data reasoning tasks can be improved by controlling data quality rather than scaling annotations.","feed_headline":"Filtered data lifts reasoning F1 by 3.78 points","feed_subtitle":"In a 24-example logical reasoning task, averaging two reward scores beats keeping more training data.","key_machinery":"The load-bearing mechanism is a two-stage data filter. First, structural pruning removes malformed JSON, traces with fewer than two reasoning steps, and parsing failures. Second, a publicly available LLaMA-3 reward model scores each remaining trace under a few-shot prompt, a zero-shot prompt, and both averaged, and a trace is kept only when the chosen score $S(x)>0$. This filter is what turns 7,376 unlabeled LogiQA questions into 1,346 training traces for the average-reward condition, and it is the only component varied across the main comparison, so it isolates data quality as the driver of the reported gains.","core_discovery":"On the paper's own terms, the discovery is that high-quality supervision, not scale, drives generalizable structured reasoning in low-resource settings. The authors generate training annotations via reverse-prompt induction and retrieval-augmented synthesis, then keep only traces whose reward score exceeds zero under one of three scoring configurations: few-shot reward, zero-shot reward, or their average. The average-based configuration performs best: the paper reports gains of 3.78 percentage points on Reasoning F1, 4.76 on Statement-Evidence F1, and 3.41 on Statement Macro F1, alongside a rise in Question Macro F1 from 56.87 to 66.71 even though question parsing was never directly rewarded. The paper reads this as evidence that accurate intermediate supervision improves even modules outside the reward loop, via better latent structure alignment.","pith_inferences":["My inference: a random-subset control would sharpen the claim—if 1,346 randomly chosen structure-filtered traces match the reward-filtered set's F1, the reported gains are about subset size, not quality.","My inference: the quality signal may be partly a simplicity signal. Checking whether retained traces are shorter or use less complex syntax than rejected ones would tell whether the reward model selects for clarity or for ease.","My inference: the same pipeline should transfer to other structured domains with scarce labels, such as legal or medical reasoning, provided a preference-scoring model exists; the paper does not test this.","My inference: iterative self-distillation could compound the effect, where the fine-tuned agents generate new traces, filter them with the same reward model, and retrain."],"forward_implications":["If quality-guided filtering is the active ingredient, then low-resource reasoning systems can spend compute on distillation and filtering rather than on collecting more human labels.","Because question parsing improved even though its outputs never entered the reward computation, better intermediate supervision appears to propagate structure improvements to downstream modules in the multi-agent pipeline.","Averaging few-shot and zero-shot reward scores is more robust than either signal alone, suggesting that multi-perspective quality estimates should be used when constructing training sets.","Each specialist agent—parser, decomposer, verifier—can be fine-tuned independently from the same distilled pool, so the framework scales to new subtasks without retraining the whole system."],"supporting_citations":[{"why":"Supplies the reverse-prompt induction (Reversal of Thought) method that derives the task-specific prompts from the 24 labeled examples.","marker":"Yuan et al., 2024"},{"why":"Supplies retrieval-augmented in-context learning, the mechanism that grounds GPT-4o-generated reasoning traces in retrieved seed examples.","marker":"Ram et al., 2023"},{"why":"The 'less is more for alignment' prior that motivates the paper's central claim that data quality, not quantity, drives performance.","marker":"Zhou et al., 2023a"},{"why":"Recent work on preference-data selection used to justify reward-guided selection of training traces.","marker":"Deng et al., 2025"},{"why":"LogiQA, the unlabeled source pool from which the pipeline synthesizes and filters reasoning traces, and the origin of the seed data.","marker":"Liu et al., 2021"},{"why":"Provides the LoRA+ adaptation method used to fine-tune all three reasoning agents from Meta-Llama-3-8B-Instruct.","marker":"Hayou et al., 2024"}],"fun_headline_variants":["Reward averaging lifts reasoning F1 by 3.78 points","24 examples: filtered data boosts reasoning F1 by 3.78","Quality-guided distillation: less data, better reasoning F1","Multi-agent filter raises reasoning F1 3.78 points","Average reward score beats more data in reasoning task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reward model's score faithfully measures reasoning quality; if the reward model is biased toward shorter, simpler, or more fluent traces, then the filtered set is easier rather than better, and the performance gap would not show that quality caused the gain.","fun_headline_variants_meta":{"raw":{"variants":["Reward averaging lifts reasoning F1 by 3.78 points","24 examples: filtered data boosts reasoning F1 by 3.78","Quality-guided distillation: less data, better reasoning F1","Multi-agent filter raises reasoning F1 3.78 points","Average reward score beats more data in reasoning task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1444,"prompt_tokens":903,"completion_tokens":541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":456}},"tokens_in":519,"tokens_out":541,"duration_ms":5468,"temperature":1.0,"reasoning_tokens":456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:03:33.986107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the average-reward-filtered set of 1,346 traces against 1,346 traces sampled uniformly at random from the structure-filtered pool and fine-tune identical models on each. If the random sample matches or exceeds the reward-filtered set on Reasoning F1 and Statement-Evidence F1, the paper's quality-over-quantity explanation is falsified. A second check: measure trace length and lexical complexity before and after filtering; systematic shortening would indicate the reward model selects for simplicity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LoRA+ adaptation method used to fine-tune all three reasoning agents from Meta-Llama-3-8B-Instruct."}],"review_version":1}