{"id":"1ce55108-29f6-4fef-93fd-416101b73b25","arxiv_id":"2505.12328","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A few-shot, untuned Meta-Llama-3-8B-Instruct system with regex post-processing ranks 5th on the LLMSR@XLLM25 structural reasoning shared task.","lead":"This system description reports a shared-task entry that uses few-shot prompting of an off-the-shelf 8-billion-parameter language model plus a simple post-processor to parse and verify reasoning chains. The entry ranked 5th on the LLMSR@XLLM25 benchmark, suggesting that minimal prompting can rival heavier pipelines, though the authors tuned hyperparameters on the test set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection inflates the reported macro-F1; the top-5 rank may be real but the score comparison is not unbiased.","rationale":"The paper is transparent about using the official test sets for ablations, but this transparency reveals the central weakness: the final configuration is chosen by optimizing the very metric that is then reported, so the numbers in Tables 1-3 are selected maxima rather than unbiased estimates. The reader flagged this exact assumption, and I agree. However, I do not think the paper should be rejected: the official 5th-place rank is an external fact that does not depend on the paper's own score reporting, and the system is simple and reproducible with released code and an off-the-shelf model. The issue is that the abstract's 'macro-F1 on par' sentence cannot be taken at face value until either a held-out evaluation or the full leaderboard data are supplied. A conditional decision asking for this clarification is appropriate. The final Test A row matching the selected maxima of the two ablations is a concrete red flag, and a fixed-configuration re-run on Test B or leave-one-out validation would settle whether the effect is large enough to change the qualitative conclusion.","tokens_in":6672,"tokens_out":5505,"duration_ms":56016,"concrete_test":"Run the official evaluator on Test A with the identical prompt pipeline and post-processor, but choose k before seeing Test A (e.g., by leave-one-out cross-validation on the 24 training examples, or by fixing k=1 as a baseline). If the recomputed Reasoning F1 is materially below 7.82, or if the k=2/k=3 combination is no longer the best configuration, the reported numbers are selection artifacts. Alternatively, request the full leaderboard and the Test B scores of all ablated configurations; if Test B does not also peak at k=2/k=3, the unbiased estimate is lower than Table 3 suggests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is in Sections 4.1-4.2: shot counts k=2 for QP and k=3 for CP are selected by sweeping on the same official test sets that later appear as final scores in Table 3. The final Test A row (75.26 / 33.04 / 13.85 / 7.82) is exactly the maximum of the two ablations, so the reported macro-F1 values are selected optima, not independent measurements. This matters because the abstract's claim of being 'on par with substantially more complex pipelines' rests on the magnitude of these numbers. If the shot counts had been fixed before seeing the test set, the Reasoning F1 (7.82) could drop toward the 5.18-3.91 range seen for other k values, and the qualitative comparison could change. The official 5th-place rank is a separate fact that may survive, since it comes from the leaderboard, but the paper does not state whether Test A or Test B was used for the sweeps and provides no variance estimates. Without a held-out validation split, the strongest empirical claim is not verifiable from the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a shared-task system description for LLMSR@XLLM25. The authors use the off-the-shelf Meta-Llama-3-8B-Instruct model with a few-shot, multi-turn prompt and a lightweight regular-expression post-processor to perform three subtasks: question parsing, chain-of-thought parsing into statement-evidence pairs, and verification of logical entailment. In Sections 4.1 and 4.2 they run k-shot ablations over the number of demonstrations for question parsing and for CoT parsing/verification, then fix the best settings and report final macro-F1 scores on Test A and Test B in Table 3. The paper claims that this simple system ranks 5th overall and achieves macro-F1 scores on par with substantially more complex and resource-consuming pipelines.","tokens_in":6878,"tokens_out":3892,"duration_ms":39747,"significance":"If the empirical claims are sound, this would be a useful data point for the LLMSR shared-task community and for process-reward-modeling research, because it shows that a minimal few-shot pipeline with an 8B model can reach a competitive leaderboard position. The paper is concise, the code is publicly available, and the authors are explicit about the model's weaknesses in logical verification, which is a genuine strength. However, the main quantitative claim is currently undermined by the fact that the model-selection ablations and the final reported scores are computed on the same official test sets, so the reported macro-F1 values are selected optima rather than independent estimates. The manuscript also does not provide a leaderboard comparison table, statistical significance tests, or an explicit development/validation split, which makes the 'on par with substantially more complex pipelines' claim unverifiable from the text alone.","major_comments":[{"comment":"The shot counts are selected by ablating on the same official test sets whose scores are later reported as the system's final performance. Concretely, the Test A ReasoningF1 of 7.82 in Table 3 is exactly the maximum of the four values 3.91, 2.50, 7.82, and 5.18 in Table 2, so the reported score is a selected optimum, not an independent measurement. This is load-bearing because the abstract's parity claim rests on the magnitude of these numbers. The authors should either hold out a validation split for the k-shot sweeps or report the final test scores for every k with confidence intervals, and they should rephrase the empirical claims accordingly.","section":"§4.1–4.2, Tables 1–3"},{"comment":"The manuscript is internally ambiguous about whether any development split exists. Section 3.3 reports that JSON guards reduced the unparsable rate 'on the dev set' from 16% to 2%, while Section 4 states that all experiments are conducted on the official test sets. If the same instances were used for prompt development and final evaluation, this is part of the selection problem; if a separate dev split exists, it should be described explicitly with its size and source. Without this information, the reader cannot determine whether the final configuration was chosen independently of the test data.","section":"§3.3 and §4"},{"comment":"The claims of 'ranks 5th overall' and 'achieving macro-F1 scores on par with substantially more complex and resource-consuming pipelines' are not substantiated in the manuscript. The official rank may be an external leaderboard fact, but no leaderboard table comparing the submitted system with the other systems is provided, and Section 5.1 explicitly calls the comparison 'informal.' The authors should include the official leaderboard results, state which phase (Test A or Test B) the ranking refers to, and report whether the differences are statistically meaningful.","section":"Abstract, §5.1, Table 3"},{"comment":"No variance information or repeated runs are reported, despite the fact that Llama-3-8B-Instruct is a stochastic decoder. In Table 2, the increase in Stmt F1 from k=1 (0.3066) to k=3 (0.3304) and in ReasoningF1 from 0.0391 to 0.0782 could plausibly be within decoding noise, so the claim that 3-shot 'strikes the best trade-off' is not yet supported. The authors should run each configuration multiple times with different seeds and report means and standard deviations, or at least report the fixed random seed used.","section":"Tables 1 and 2"},{"comment":"The paper does not state whether the k-shot sweeps in Sections 4.1 and 4.2 were performed on Test A or Test B. Since Table 3 reports both phases, this choice matters for interpreting the sentence in Section 4.3 that the hybrid setup 'achieves the highest overall macro-F1 on the public leaderboard.' Please specify the phase used for model selection and justify why that phase is appropriate.","section":"§4, final configuration"}],"minor_comments":[{"comment":"There is a typo in 'multi-steasoning' in Section 2.1, and 'Input Qusetion' in Figure 1 should be 'Input Question.'","section":"§2.1 and Figure 1"},{"comment":"The sentence 'official LLMSR@XLLM25 test sets1' has a missing space before the footnote marker, and the footnote should either be a proper URL citation or be integrated into the text.","section":"§4"},{"comment":"The reported drop in unparsable rate from 16% to 2% on the dev set would be more informative if the authors defined what counts as 'unparsable' and reported the number of dev instances used.","section":"§3.3"},{"comment":"The phrase 'Informal leaderboard comparisons indicate...' should be replaced with a formal comparison table, because the current wording does not allow the reader to verify the central competitiveness claim.","section":"§5.1"},{"comment":"The reference for the Meta-Llama-3-8B-Instruct model card lacks a URL or version identifier, and the capitalization of 'MANN and THOMPSON' is inconsistent with the other references.","section":"References"},{"comment":"For reproducibility, the exact prompt templates and the two QP and three CP demonstration exemplars should be included in an appendix, rather than only pointing to the GitHub repository, since repositories can change after publication.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a competent shared-task system report with a clear narrative and honest self-assessment of the model's limitations. The main issue is statistical rigor: the reported scores are selected on the test set, and the competitive claims are not backed by a leaderboard comparison. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. For a journal venue, the contribution is modest, so the revised version should make the empirical claims carefully qualified and reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a shared-task system paper, not a new method. The notable result is that a completely untuned Llama-3-8B-Instruct, with a compact few-shot prompt and a regex post-processor, lands 5th on the LLMSR@XLLM25 leaderboard. That is a useful baseline for the structural reasoning crowd, and the paper is honest about where the approach fails (negation, evidence hallucination, etc.).\n\nThe paper does a few things well. The prompt engineering is reported concretely: three-turn template, fenced JSON blocks, and the post-processor cut the unparsable rate from 16% to 2% on dev. The ablation showing QP and CP prefer different demonstration counts (2 vs 3) is a small but legitimate empirical observation. The code is released. The related work is standard but adequate.\n\nThe soft spot is the one the stress-test flags, and it is real. The shot counts are selected by sweeping on the official test sets, and then the same test set gives the final numbers in Table 3. So the reported macro-F1 values are selected optima, not independent measurements. The Reasoning F1 of 7.82 at k=3 could easily be lower if the configuration had been fixed before seeing the test set—the other k values in Table 2 range from 2.50 to 5.18. That directly weakens the abstract's claim about being 'on par with substantially more complex pipelines.' The official 5th place rank probably survives, since the leaderboard adjudicated the actual submissions, but the score comparison needs a held-out validation split or at least variance estimates.\n\nThe other issue is minor: the parity claim is asserted without showing the full leaderboard or naming the heavier pipelines. And the paper doesn't explicitly say whether the ablations were run on Test A or Test B, which is confusing given Table 3 reports both.\n\nCitation pattern looks fine; no self-citation problem. The limitations section is refreshingly direct about the model's logical weaknesses.\n\nWho should read it: anyone building process reward models or evaluating reasoning structure will want this as a baseline reference. It won't change anyone's research plans on its own.\n\nMy call: this deserves a proper peer review, but with a required revision—report a held-out validation split, show the full leaderboard context, and soften the parity claim to match the evidence. As it stands, it's a solid shared-task report whose central empirical claim is a bit oversold but probably not wrong in direction.","headline":"Solid shared-task report with a real test-set selection bias; the 5th place is credible but the 'parity with heavier pipelines' claim is oversold.","tokens_in":7400,"tokens_out":2200,"would_cite":false,"duration_ms":20907,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Few-shot prompting on an untuned 8B model reaches 5th place on a structured reasoning task.","keywords":["structural reasoning","chain-of-thought prompting","few-shot in-context learning","question parsing","statement-evidence verification","logical entailment","Meta-Llama-3-8B-Instruct","process reward modeling"],"falsifier":"Rerun the pipeline with shot counts chosen on a held-out validation split instead of the official test sets, then report macro-F1 on the untouched test sets. If the final numbers fall materially below the reported 75.26 / 33.04 / 13.85 / 7.82 (Test A) or the 5th-place rank does not reproduce, the headline result was inflated by selecting the configuration on the very data used for scoring.","tokens_in":6466,"feed_emoji":"🧠","tokens_out":8283,"duration_ms":71823,"temperature":0.7,"pith_summary":"This paper reports a deliberately minimal system for the LLMSR@XLLM25 shared task, where a model must extract all conditions from a problem, split a chain-of-thought into statement–evidence pairs, and judge whether each evidence span entails its statement. The authors claim that an untuned Meta-Llama-3-8B-Instruct, driven by a concise few-shot multi-turn prompt and a regex-based post-processor, ranks 5th overall and achieves macro-F1 scores comparable to substantially more complex and resource-consuming pipelines. The system uses two demonstrations for question parsing and three for CoT parsing and verification, and it follows a three-turn template: system rules, user problem, assistant JSON answer. A reader should care because the result suggests that prompt design and output hygiene, rather than fine-tuning or retrieval, can carry an 8B model to the top of a structured-reasoning leaderboard, while also exposing logical adjudication as the real bottleneck.","feed_headline":"Few-shot prompts put an untuned 8B model in 5th place","feed_subtitle":"No fine-tuning, retrieval, or ensembling: a two-stage prompt matches heavier structured-reasoning pipelines.","key_machinery":"The load-bearing mechanism is a two-call, three-turn prompted pipeline with a hand-picked few-shot budget: the first call extracts every explicit condition as an ordered list (Question Parsing), and a second call consumes the question, its chain-of-thought, and that condition list to emit statement–evidence pairs together with a boolean entailment verdict (CoT Parsing & Verification). Each call is a SYSTEM–USER–ASSISTANT conversation, demonstrations are fenced in ```json``` blocks, and a deterministic regex post-processor checks the schema, normalizes spans, merges duplicate conditions, and aligns statement and evidence counts. The key empirical lever is that the two stages receive different demonstration counts, k=2 for QP and k=3 for CP, chosen by separate ablations, and that separating the stages beats a single fixed prompt size.","core_discovery":"On its own terms, the paper's central discovery is that a fixed, off-the-shelf instruction-tuned 8B model can handle structured reasoning competitively if reasoning is decomposed into two prompted stages and the output is fenced as JSON. Question Parsing with k=2 demonstrations reaches a macro-F1 of 0.7526, and CoT Parsing & Verification with k=3 demonstrations yields statement, statement-and-evidence, and reasoning macro-F1 scores of 0.3304, 0.1385, and 0.0782 on the official Test A set (0.4726, 0.2017, and 0.1164 on Test B). The submission ranks 5th overall. The authors argue this shows that careful prompt design plus minimal hygiene techniques, including fenced ```json``` blocks that cut the unparsable rate from 16% to 2%, can rival far more elaborate pipelines, and that the remaining gap is not in condition extraction but in logical adjudication, where the model hallucinates evidence, paraphrases conditions, and mishandles negation.","pith_inferences":["A fairer estimate of the method's true strength would require choosing shot counts on a separate validation split; the paper's reported numbers are selected optima, since the same official test sets were used both to pick k and to report final scores.","The same two-stage split suggests a transferable recipe for other structured-reasoning benchmarks: spend prompt budget on decomposition and format, then invest model capacity in the verification step.","The generated statement–evidence–verdict triples are exactly the supervision Process Reward Models want; using them to train a small critic, rather than treating them as final output, is a natural extension the paper mentions but does not pursue.","The observation that adding a fourth demonstration sometimes hurts performance points to prompt-length or attention dilution as a possible constraint for 8B models; testing the same prompts on a longer-context model would separate exemplar-diversity effects from context-window effects."],"forward_implications":["A top-5 overall result can be reached without fine-tuning, retrieval, or ensembling, so strong structured-reasoning performance is not exclusive to heavy pipelines.","Question parsing and CoT parsing reward different demonstration counts, so stage-specific shot tuning is a practical lever when the total demonstration budget is small.","Fenced JSON output reduces the unparsable rate from about 16% to 2%, meaning output-format enforcement is a cheap and effective component of reasoning pipelines.","Because reasoning F1 stays low even after format fixes, the paper's own conclusion is that the bottleneck is the model's logic, not the interface: better verifiers or process-level training are the needed next step."],"supporting_citations":[{"why":"Supplies the untuned Meta-Llama-3-8B-Instruct base model on which the whole pipeline runs; the paper's claim is specifically about this off-the-shelf model.","marker":"Meta AI, 2024"},{"why":"LogiQA is the source corpus from which the shared task derives its fine-grained CoT data and test sets, giving the evaluation its domain.","marker":"Liu et al., 2021"},{"why":"Introduces the chain-of-thought prompting paradigm whose output structure this task constrains into statement–evidence pairs; the paper's prompts are CoT prompts in this lineage.","marker":"Wei et al., 2023"},{"why":"Establishes zero-shot CoT prompting, the baseline style the paper's few-shot multi-turn template extends rather than fine-tuning.","marker":"Kojima et al., 2022"},{"why":"Self-consistency decoding is the main alternative performance-boosting technique the paper deliberately avoids, sharpening the claim that prompt design alone suffices.","marker":"Wang et al., 2023"},{"why":"Process reward modeling is the downstream use case that motivates step-level statement–evidence labels and frames why the task's structure matters.","marker":"Lightman et al., 2023"},{"why":"Documents hallucination in free-form chain-of-thought, the problem the shared task's constrained verification format is designed to address.","marker":"Akbar et al., 2024"}],"fun_headline_variants":["Untuned 8B model ranks 5th in structured reasoning","Fenced JSON slashes unparsable rate from 16% to 2%","Two-stage prompt, no tuning, ranks 5th","Plain 8B LLM matches complex pipelines in reasoning","8B instruct model, concise prompts, 5th place"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The report assumes that choosing the demonstration counts from the same test scores that are then reported as the result does not inflate those scores, and this assumption is load-bearing because Tables 1 and 2 are used both to select the configuration and to present its performance.","fun_headline_variants_meta":{"raw":{"variants":["Untuned 8B model ranks 5th in structured reasoning","Fenced JSON slashes unparsable rate from 16% to 2%","Two-stage prompt, no tuning, ranks 5th","Plain 8B LLM matches complex pipelines in reasoning","8B instruct model, concise prompts, 5th place"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2150,"prompt_tokens":970,"completion_tokens":1180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1089}},"tokens_in":586,"tokens_out":1180,"duration_ms":8661,"temperature":1.0,"reasoning_tokens":1089,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:35:41.002639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline with shot counts chosen on a held-out validation split instead of the official test sets, then report macro-F1 on the untouched test sets. If the final numbers fall materially below the reported 75.26 / 33.04 / 13.85 / 7.82 (Test A) or the 5th-place rank does not reproduce, the headline result was inflated by selecting the configuration on the very data used for scoring.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the untuned Meta-Llama-3-8B-Instruct base model on which the whole pipeline runs; the paper's claim is specifically about this off-the-shelf model."}],"review_version":1}