{"id":"38cbb2f1-5774-4674-b18b-c5a79b6cd848","arxiv_id":"2412.05023","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new prompting recipe that mixes few-shot chain-of-thought examples with model-generated analogies is claimed to improve STEM question-answering on Mixtral 8x7B, alongside a new 928-question dataset.","lead":"This paper introduces StemStep, a dataset of 928 high school physics and math problems with step-by-step solutions, and tests several prompting strategies on open-source language models. The authors report that a combination of few-shot reasoning and analogical prompting improves answer accuracy on Mixtral 8x7B, but the evaluation methods are too loosely described to trust the headline numbers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported Analogical CoT gain rests on an evaluation protocol that regenerates low-similarity responses and scores outputs with a single unnamed human evaluator; without raw counts, the 66.2% figure cannot be distinguished from retry-inflated or annotator-biased numbers.","rationale":"I read the paper as making a narrow empirical claim: Analogical CoT prompting improves open-source model accuracy on high-school physics and math Q&A, with Mixtral 8x7B rising from 42% to 66.2%. For that claim to hold, the reported accuracy must be a well-defined estimator of model performance under identical conditions across all prompting methods. The weakest link is the evaluation protocol: it combines a deterministic TF-IDF similarity filter that triggers regeneration with a single human judgment, and it reports no raw counts. This is not a disagreement with the idea that prompting can help; it is a correctness risk in the only evidence table. The manuscript's own appendix exposes the retry rule without quantifying it, and the limitations section does not list this as a threat. The dataset and code are not provided, and K values are chosen post hoc from the reported curves. The worked examples in Section 4 show coherent chain-of-thought outputs, but selected examples are not evidence of aggregate accuracy. I agree with the reader's weakest assumption, and the concrete test would settle whether the reported gain survives a clean protocol. If it does not, the paper should be revised before acceptance.","tokens_in":10899,"tokens_out":4017,"duration_ms":41348,"concrete_test":"Recompute Mixtral 8x7B accuracy under Analogical CoT, K-shot CoT, and baseline on a fixed held-out split with regeneration disabled (exactly one generation per question), two additional annotators blinded to condition, and full per-question scores plus retry counts reported. If the no-retry Analogical CoT accuracy does not exceed K-shot CoT (64.5%) and baseline (42%) by a margin larger than annotator disagreement, the claimed improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Analogical CoT lifts Mixtral 8x7B from 42% baseline to 66.2% (Table 1), but this number is produced by a protocol that can manufacture accuracy. Section 5.4 defines Accuracy as a single evaluator's judgment that output is 'inline with the ground truth,' and the Appendix adds a retry rule: 'If cosine_similarity(T_A,T_B) < 0.3 then regenerate response.' The paper never states how many regenerations were allowed, how many outputs were regenerated in each condition, or whether the human score was assigned to the first attempt or the last retry. If Analogical CoT produced more low-similarity first attempts that were retried until one passed, the 24-point gap could be an artifact of retry count rather than prompt quality. Additionally, K=3 (Mistral) and K=6 (Mixtral) are selected from the same test curves used to report the headline, and StemStep is not released, so no independent audit is possible. The prompting idea may be plausible, but the empirical basis for the headline is not pinned to a fixed, auditable evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces StemStep, a dataset of 928 high-school physics and mathematics questions with step-by-step solutions, and proposes Analogical CoT prompting, which combines few-shot chain-of-thought examples with self-generated analogical examples. The authors evaluate Mistral 7B and Mixtral 8x7B on StemStep and report that Analogical CoT raises Mixtral 8x7B's response-match accuracy from a 42% baseline to 66.2%, while K-shot CoT reaches 64.5% (Table 1). They also observe that Mistral 7B benefits from few-shot CoT but not from analogical prompting. The central claim is that the proposed combined prompting method allows smaller open-source models to exploit analogical reasoning, and that the StemStep dataset is a useful evaluation resource for STEM education.","tokens_in":11125,"tokens_out":4583,"duration_ms":45886,"significance":"If the headline results were established under a fixed, auditable protocol, the paper would make a useful practical contribution: Analogical CoT is a simple, plausible way to improve open-source models on multi-step STEM problems, and StemStep could complement existing benchmarks such as SciQ, GSM8K, and JEEBench with a longer-form, step-based format. The paper is also honest about some limitations, noting in Section 7 that StemStep's small size (928 questions) limits its utility for fine-tuning and that quantized MoE models degrade in performance. However, the empirical significance is currently undercut by the evaluation methodology: the reported accuracies depend on a response-regeneration rule and a single subjective human evaluator, the few-shot order K is selected from the same test curves that produce the headline numbers, and the dataset is not released. These issues are load-bearing for the paper's main claim, and they are not acknowledged in the Limitations section.","major_comments":[{"comment":"The accuracy metric is not a fixed protocol. Section 5.4 defines Accuracy as a single evaluator's judgment that the model output is 'inline with the ground truth,' and the Appendix states that responses with TF-IDF cosine similarity below 0.3 to the ground truth are regenerated. The paper does not state how many regenerations were permitted per question, how many outputs were regenerated in each condition, or whether the human score was assigned to the first attempt or to the final (possibly regenerated) output. If the Analogical CoT condition produced more low-similarity first attempts that were retried until one passed, the 24-point gap between baseline and Analogical CoT in Table 1 could be an artifact of retry counts rather than prompt quality. Please report the full protocol: the retry cap, regeneration counts per condition, and accuracy computed on first attempts and on final attempts separately, together with confidence intervals or raw counts.","section":"Section 5.4 and Appendix"},{"comment":"The few-shot values K used for the headline comparison appear to be selected from the test set. The text says Mistral 7B performance 'peaks around K=3' and that for Mixtral 8x7B 'K=6 seems to show the most consistent performance,' and Table 1 then reports accuracy at exactly these K values. Selecting K from the same test curves used to report the results is a form of test-set fitting and makes the reported K-shot CoT advantage over baseline optimistically biased. Please select K on a validation split (or report a pre-registered selection rule), and report accuracy for all K values considered, not only the best one.","section":"Table 1 and Section 5.1"},{"comment":"StemStep is not released, all experiments are conducted on this private dataset, and no external benchmark is used, so the reported results cannot be independently audited. The dataset description is also thin: Section 3 says the data were 'scraped from online resources' without specifying sources, licensing, or quality filters beyond a 5-person survey of 50 questions each, and no inter-annotator agreement is reported for that survey. In addition, the dataset size is given as 928 in Section 3 but as '1000 Physics and Mathematics questions' in the Abstract and Conclusion. Please release the dataset (or a curated sample), provide a detailed datasheet, and reconcile the size discrepancy.","section":"Section 3 and Appendix"},{"comment":"The proposed Analogical CoT prompt fixes the number of supplied examples at 3 and the number of self-generated examples at 3, with no ablation or sensitivity analysis. The paper states that these values were chosen 'on analysis of prior work,' but the cited prior work (Yasunaga et al., 2023) uses pure analogical prompting rather than the combined CoT-plus-analogical prompt. Since the headline gain in Table 1 depends on this specific configuration, the absence of any variation of the number of supplied versus self-generated examples leaves the central claim underdetermined. I would expect at least a small ablation varying (3,3) to (3,0) and (0,3), or a justification from a validation split.","section":"Section 4.3 and Figure 5"},{"comment":"The sentence 'LoRA was employed to mitigate the computational challenges associated with inference' is technically inaccurate as written: LoRA is a parameter-efficient fine-tuning method, not an inference-time technique. If the authors used QLoRA to quantize the models for inference, they should say so explicitly and specify the quantization settings, because Section 7 states that 4-bit quantization degrades Mixtral's performance. This matters because the reported absolute accuracies are for quantized models; a reader cannot currently tell which numbers correspond to which quantization level.","section":"Section 4"}],"minor_comments":[{"comment":"The dataset size is inconsistent: the Abstract and Conclusion say 1000 questions, while Section 3 says 928. Please correct the inconsistency.","section":"Abstract / Section 3 / Conclusion"},{"comment":"The paper states that the prompt was tested on GPT-4 and Gemini Pro, but no results for these models are reported anywhere in the paper. Either report those results or remove the claim.","section":"Section 4.3"},{"comment":"Several figures (e.g., Figures 3, 4, and 5) appear to lack axis labels and captions that explain what is being plotted. The text refers to 'metrics recorded' without specifying which metric is shown; please add readable axis labels, legends, and captions.","section":"Figures 3-6"},{"comment":"There are numerous typos and grammatical errors, including 'neccesary', 'the the evaluator', 'succesfully', and 'succesfully'. The paper would benefit from a careful proofreading pass.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a workshop-paper feel: the empirical evaluation is not yet at the level expected for a full journal article. The most serious concern is the un-auditable evaluation protocol (regeneration plus single human judge) and the test-set selection of K; both are fixable by re-running the experiments with a transparent protocol and reporting raw counts. I do not see evidence of intentional deception, but the current numbers are not reliable. I would not recommend rejection if the authors are willing to redo the evaluation, because the prompting idea is plausible and the dataset, if released, could be a small but useful resource. However, I would not accept the paper in its present form. I also note a very high degree of self-citation to the same group's prior work in the Related Work section; this is not inappropriate in itself, but the references should be checked for relevance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper combines two known prompting tricks into a plausible new variant and builds a small new dataset, but the evaluation can't carry the headline claim. The 24-point gain for Analogical CoT on Mixtral looks like it could be a retry artifact plus a generous human judge.\n\nWhat's actually new: Analogical CoT (giving a few CoT exemplars and asking the model to self-generate more) is a straightforward but sensible combination of few-shot CoT and Yasunaga et al.'s analogical prompting. The paper also honestly documents that pure analogical prompting fails on Mistral and Mixtral, which is a useful negative result for anyone working on open-weight models. StemStep, 928 high-school physics and math questions with step-by-step solutions, is a reasonable contribution if it gets released; the average step count and length distributions are reported, and a small student survey (5 raters on 50 questions each, average 9/10) is a nice touch.\n\nSoft spots, in order of severity. First, the evaluation protocol in the Appendix: responses with TF-IDF cosine similarity below 0.3 to the ground truth are regenerated. There's no cap on regenerations, no count of how often it fired per condition, and no statement about whether the human evaluator scored the first attempt or the final retry. If Analogical CoT prompts produce more off-topic first tries that get re-rolled until one passes, the 66.2% number is inflated relative to a fixed, single-attempt protocol. Second, the 'Accuracy' metric is one unnamed human evaluator's judgment of whether the output is 'inline with the ground truth' — no sample size, no double-scoring, no agreement. That is too subjective to carry a 24-point claim. Third, the K values (3 for Mistral, 6 for Mixtral) were chosen from the same test curves used to report the headline, so part of the gain is fitted to the test set. The dataset is not released, so none of this can be independently audited. There are also a few smaller slips: the abstract says Mixture of Experts is the key, but the comparison is only Mistral vs Mixtral; LoRA is described as mitigating inference compute, which is not what LoRA does; the conclusion says 1000 questions while the text says 928.\n\nWho gets value: someone working on prompting recipes for small open models, or on STEM QA evaluation sets, could read this for the idea and the negative results. As a peer-reviewed claim of a concrete accuracy gain, it doesn't hold up yet. A revision with the dataset released, a fixed evaluation protocol (no retry, or a pre-registered retry policy), more than one annotator, and validation-based K selection would make the empirical claim testable. I'd send it to a workshop rather than a main conference in its current form.","headline":"Plausible prompt combination and a useful new dataset, but the evaluation protocol (retry-on-low-similarity, single human judge, test-set K selection) means the headline 66.2% gain is not reliably supported.","tokens_in":11702,"tokens_out":3118,"would_cite":false,"duration_ms":30048,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a combined prompt—three chain-of-thought examples plus self-generated analogies—raises Mixtral 8x7B's response-match accuracy on high-school physics and math from a 42% baseline to 66.2%, above its 64.5% K-shot…","keywords":["prompt engineering","chain-of-thought prompting","analogical prompting","few-shot prompting","mixture of experts","STEM education","physics and mathematics question answering","StemStep dataset"],"falsifier":"Re-run the Mixtral 8x7B comparison on StemStep without the regeneration step, scoring every initial response with the same human-judgment protocol; if the 66.2% Analogical CoT accuracy drops to the 64.5% K-shot CoT level or below, the claimed advantage is an artifact of discarding low-similarity outputs.","tokens_in":10698,"feed_emoji":"🎓","tokens_out":11074,"duration_ms":243249,"temperature":0.7,"pith_summary":"This paper is trying to establish that open-source language models can be made to reason through high-school physics and mathematics problems by changing the prompt rather than the model. To that end it introduces StemStep, a dataset of 928 step-by-step physics and mathematics questions, and proposes Analogical CoT prompting, which combines a few chain-of-thought examples with the model's self-generated analogies. The key empirical claim is that this combined prompt raises Mixtral 8x7B's response-match accuracy to 66.2%, above the 42% zero-shot baseline and the 64.5% K-shot chain-of-thought result, while smaller Mistral 7B benefits only from the few-shot chain-of-thought part. If the claim holds, it matters because it offers a cheap, training-free route to better STEM tutoring answers from models that otherwise struggle with analogical reasoning.","feed_headline":"Prompt combo lifts open-source model to 66.2% accuracy","feed_subtitle":"A step-by-step and self-generated-analogy prompt beats zero-shot and few-shot baselines on high-school physics and math.","key_machinery":"The central mechanism is Analogical CoT prompting, a prompt template that shows the model three fully worked example solutions and then instructs it to recall or generate additional analogous examples before solving the target question, so that the model supplies its own few-shot context. The paper argues that this compensates for the weak recall of open-source models that makes pure analogical prompting fail. The other load-bearing element is StemStep, the 928-question dataset of high-school physics and mathematics problems with step-by-step reference solutions that serves as the evaluation ground truth. The Mixture-of-Experts architecture of Mixtral 8x7B, with a router that selects among eight specialized experts, is treated as the reason the combined prompt works on that model but not on Mistral 7B.","core_discovery":"The paper's central claim is that analogical prompting, which works on large proprietary models, fails on open-source models because those models cannot recall relevant solved examples on their own, and that this failure can be repaired by seeding the prompt with three worked examples before asking the model to generate further examples. On the StemStep dataset, this Analogical CoT prompt lifts Mixtral 8x7B to 66.2% response-match accuracy, compared with 42% under a zero-shot baseline and 64.5% under K-shot chain-of-thought prompting. The authors also report that Mistral 7B does not gain from the combined method (32% versus a 31.5% baseline) but does gain from K-shot CoT (53%), and that the StemStep dataset and the proposed prompt together provide a way to evaluate and improve STEM education assistance without fine-tuning.","pith_inferences":["Beyond the paper: the bottleneck for analogical prompting on small open models may be example recall rather than reasoning, which suggests that adding a lightweight retrieval step—pulling similar solved problems from a small corpus—could give bigger gains than further prompt changes.","Beyond the paper: because Mixtral 8x7B is a gated mixture of eight smaller experts, an ablation against a dense model with a similar total parameter count would isolate whether the gain comes from the MoE architecture or from overall scale.","Beyond the paper: the same Analogical CoT template should transfer to other step-based STEM subjects, such as chemistry or biology, though the paper only tests physics and mathematics."],"forward_implications":["On Mixtral 8x7B, Analogical CoT reaches 66.2% response-match accuracy, beating the 42% zero-shot baseline and the 64.5% K-shot CoT baseline, so the combined prompt is the strongest prompting method in the paper's comparison.","Mistral 7B's analogical performance stays near baseline, so the benefit of the combined prompt is not universal across open-source models and depends on scale or architecture.","The proposed prompting style can be applied to any model and dataset without fine-tuning, which makes it a low-cost complement to training-based improvements.","Longer or more complex questions gain more from longer reasoning chains, so prompt construction should adapt the number of steps to problem difficulty.","StemStep provides a reusable evaluation set of 928 step-by-step high-school physics and mathematics questions for future prompt-engineering and fine-tuning studies."],"supporting_citations":[{"why":"Introduces analogical prompting, the technique this paper adapts for open-source models.","marker":"Yasunaga et al., 2023"},{"why":"Establishes chain-of-thought prompting, the reasoning format the proposed method extends.","marker":"Wei et al., 2023"},{"why":"Defines few-shot prompting, the basis for the K-shot CoT construction.","marker":"Brown et al., 2020"},{"why":"Provides the Mistral 7B model evaluated in the paper.","marker":"Jiang et al., 2023"},{"why":"Provides the Mixtral 8x7B model that achieves the headline 66.2% result.","marker":"Jiang et al., 2024"},{"why":"Supplies the earlier physics QA dataset and prompting pipeline that StemStep and the K-shot CoT prompts build on.","marker":"Anand et al., 2023a"},{"why":"Defines zero-shot reasoning prompting, used as the baseline for both models.","marker":"Kojima et al., 2023"}],"fun_headline_variants":["Analogical CoT prompt lifts open-source model to 66.2%","Seeded analogies boost open-source STEM model accuracy","Prompt method lifts Mixtral to 66% on physics questions","Open-source model gains from analogical prompting with examples","How analogical prompting fixes open-source model's STEM gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains assume the model's first response is what is being scored, but the evaluation regenerates any output whose text-similarity score to the reference is below 0.3 and then has a single human judge say whether the final output matches the ground truth, so the numbers could partly reflect a re-roll effect rather than the model's unaided performance.","fun_headline_variants_meta":{"raw":{"variants":["Analogical CoT prompt lifts open-source model to 66.2%","Seeded analogies boost open-source STEM model accuracy","Prompt method lifts Mixtral to 66% on physics questions","Open-source model gains from analogical prompting with examples","How analogical prompting fixes open-source model's STEM gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2047,"prompt_tokens":834,"completion_tokens":1213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":1128}},"tokens_in":450,"tokens_out":1213,"duration_ms":9381,"temperature":1.0,"reasoning_tokens":1128,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:57:58.804764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Mixtral 8x7B comparison on StemStep without the regeneration step, scoring every initial response with the same human-judgment protocol; if the 66.2% Analogical CoT accuracy drops to the 64.5% K-shot CoT level or below, the claimed advantage is an artifact of discarding low-similarity outputs.","supporting_citations":[],"review_version":1}