{"id":"7871e0bb-3da3-4035-ac48-c22b7cd5eeaf","arxiv_id":"2506.06401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A gradient-free Monte Carlo tree search over JSON key-step plans produces few-shot demonstrations that let LLaMA3-8B and LLaMA3.2-3B outperform GPT-3.5 on most of seven BIG-Bench Hard tasks.","lead":"DeBoP is an automatic method that searches over structured chain-of-thought demonstrations to improve the accuracy of lightweight language models on reasoning tasks. It uses Monte Carlo tree search to find a single optimized example, and the authors report that 3B and 8B LLaMA models then beat GPT-3.5 on most of seven BIG-Bench Hard tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'surpasses GPT-3.5' claim is unsupported because GPT-3.5 was not run on the same 100-sample test set; its numbers are cited from OpenCompass, so task/subset/prompt differences, not DeBoP, could explain the margin.","rationale":"The reader's conditional verdict is reasonable, and I agree the evaluation lacks statistical support. However, the most load-bearing gap is not only the absence of confidence intervals; it is that the GPT-3.5 rows in Table 1 are not produced by the same evaluation pipeline. Section 3.3 describes Stest as a random 100-sample subset for DeBoP, then says GPT-3.5 results were 'referenced from Opencompass, 2025.' Unless GPT-3.5 was run on that exact 100-item subset with the same prompts, temperature, and fext extraction, the reported per-task margins (e.g., SNK +9, HB -4) cannot be attributed to DeBoP. This concern is distinct from the reader's weakest assumption, though related: even a perfectly seeded DeBoP evaluation would not validate a comparison against external numbers. I therefore recommend keeping the conditional verdict, with the condition that the authors supply a matched GPT-3.5 evaluation on Stest and, ideally, confidence intervals for DeBoP. The method itself is coherent, has a public implementation, and the ablations are internally consistent; I am not claiming the approach is invalid, only that the headline comparative claim is not yet supported.","tokens_in":36,"tokens_out":9409,"duration_ms":199182,"concrete_test":"Run GPT-3.5-turbo on the exact 100-sample Stest used for Table 1, for all seven tasks, using the same prompt template, decoding temperature, and fext answer extraction as the DeBoP evaluation; also repeat the DeBoP selection/evaluation with at least 3 seeds (or bootstrap over Stest) to obtain confidence intervals. If GPT-3.5's recalculated per-task accuracies differ from the OpenCompass reference and DeBoP no longer wins a majority of tasks, the 'surpass GPT-3.5' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline accuracy claim is that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 on most BBH tasks. Table 1 compares DeBoP's 100-sample Stest results (Section 3.3, fixed seed 42) against GPT-3.5 accuracies 'referenced from Opencompass' (Section 3.3). There is no indication that GPT-3.5 was evaluated on the same 100-item subset, with the same prompt format, decoding settings, or answer-extraction function fext. A 100-sample random subset is not guaranteed to reproduce full-set BBH difficulty; per-task margins of -1 to +9 points could easily be subset-selection or prompt-mismatch artifacts. The reader's seed-variance point is valid but only covers DeBoP's side of the comparison; multiple seeds would not fix the absence of a matched GPT-3.5 baseline. Until GPT-3.5 is run on the identical Stest under the same evaluation protocol, the central 'surpass GPT-3.5' claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces DeBoP, a four-phase prompt optimization method for lightweight LLMs. DeBoP generates task-specific key-step plans in JSON form, collects demonstrations produced by the LwLLM on a development set, searches over mutated demonstrations with a gradient-free Monte Carlo Tree Search whose reward combines development-set accuracy and inference time, and finally teaches the selected demonstration to the model. Experiments compare LLaMA3-8B and LLaMA3.2-3B against Direct Prompting, CoT Prompting, StrategyLLM, Self-Discover, and GPT-3.5 on seven BIG-Bench Hard tasks, reporting accuracy and inference time. The central claims are that DeBoP outperforms recent automatic prompt optimizers on most tasks, that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 on most tasks, and that it reduces average inference time by roughly 60% relative to other automatic methods.","tokens_in":22716,"tokens_out":3893,"duration_ms":38480,"significance":"The paper addresses a relevant and timely problem: automatic prompt optimization for small local models without relying on strong LLM meta-cognition or external APIs. The method is clearly described, and the paper includes useful strengths: a public code release, ablations of the pipeline stages, a cross-model generalization study, an explicit single-versus-multi-demonstration analysis, and a supplementary investigation on a distilled reasoning model. If the comparative claims hold, DeBoP would be a practical contribution for resource- and privacy-constrained deployments. However, the headline claims currently rest on a statistically thin evaluation and an unmatched GPT-3.5 baseline, so the strength of the evidence is not yet commensurate with the claims.","major_comments":[{"comment":"The claim that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 on most tasks is not yet supported because the GPT-3.5 accuracies are cited from OpenCompass rather than produced on the same 100-sample Stest. Since the per-task margins in Table 1 are as small as 1 to 9 percentage points, a different subset, prompt format, decoding setting, or answer-extraction function could fully explain the differences. Please run GPT-3.5 on the identical Stest under the same evaluation protocol as the LwLLMs, or rephrase/remove the claim.","section":"Section 3.3 / Table 1"},{"comment":"All DeBoP results come from a single 50-sample development split and a single 100-sample test split with fixed seed 42, with no confidence intervals or significance tests. With 100 test items, a difference of roughly 5 percentage points is within one standard error, so several of the per-task comparisons in Table 1 could be sampling noise. Please report multiple seeds with confidence intervals, or provide significance tests; this is essential for the central 'surpasses GPT-3.5 on most tasks' claim.","section":"Section 3.3 / Table 1"},{"comment":"The MCTS reward in Section 2.2.3 depends on hyperparameters alpha, beta, and lambda, and the search also depends on the UCB exploration coefficient c, the number of key-step plans k, the 50-iteration budget, and the 20% early-stop probability. These values are fixed without sensitivity analysis, and since Sdev is drawn from the same BBH tasks used for test, selecting these values with knowledge of the tasks is a correctness risk. Please provide a sensitivity study or clearly state how each hyperparameter was chosen.","section":"Section 2.2.3 / Section 3.3"},{"comment":"The approximately 60% efficiency reduction is calculated from per-example inference time only, but the abstract's phrasing ('reducing computational time') could be read as end-to-end optimization time. The planning, collecting, and MCTS expansion costs of DeBoP are not reported, and it is unclear whether the StrategyLLM and Self-Discover time measurements include their generation/optimization stages. Please clarify precisely which time is measured and whether the efficiency claim covers only test-time inference or the full pipeline.","section":"Section 3.4.2 / Table 2"}],"minor_comments":[{"comment":"The ablation text defines PCT and PCMT with the same phrase 'PLANNING + COLLECTING + MCTS + TEACHING', which makes the ablation legend and conclusion difficult to parse; please correct the labels or the text.","section":"Section 3.4.3 / Figure 4"},{"comment":"Table 3 is hard to read because the first row appears to serve both as a header and as data. Please separate the model used to generate demonstrations from the model applying DeBoP, and clearly mark the no-DeBoP baseline numbers.","section":"Table 3"},{"comment":"The GPT-3.5 numbers are cited as 'Opencompass, 2025' without an author list, version, or exact retrieval URL; please provide a complete citation with the specific benchmark configuration and access date.","section":"Section 3.3 / References"},{"comment":"The selection probability formula for Probi has a minor typographical issue with redundant parentheses around Quant(pi); please tidy the equation.","section":"Section 2.2.2"},{"comment":"The Limitations section discusses MCTS breadth and potential misuse, but does not mention the statistical limitations of the evaluation or the unmatched GPT-3.5 baseline; please acknowledge these in the final text.","section":"Section 6 / Limitations"}],"recommendation":"major_revision","confidential_remarks":"The method itself is plausible and the paper is well organized, but the evaluation is not yet sufficient for the advertised claims. The most important fix is to run GPT-3.5 on the same 100-sample test set with the same prompt and extraction protocol; without that, the central comparison against GPT-3.5 should not be stated as a result. Adding multiple seeds or confidence intervals is also necessary. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. The method is a real contribution, and the headline accuracy claim is not established by the evidence. They are separate issues.\n\nWhat's new: DeBoP optimizes a single JSON key-step demonstration through a gradient-free MCTS, using six mutation operators generated by the lightweight model itself. That specific search structure is not in the cited prior art, and it is a sensible fit for models that cannot do reflective self-prompting. The ablations show the MCTS and teaching phases add value, and the cross-model generalization experiments are a nice touch. The appendix on DeepSeek-R1 is also a good faith effort to check an important boundary condition. This is a solid engineering contribution for resource-constrained or privacy-sensitive settings.\n\nWhere it goes soft: the central claim that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 is not backed up. The GPT-3.5 numbers are pulled from OpenCompass, not run on the same 100-sample Stest with the same prompt format, decoding settings, or answer extraction. For a 100-sample subset, per-task margins of a few points are exactly the kind of thing that subset selection or prompt mismatch can explain. The reader's worry about a single seed and no confidence intervals is valid but actually too mild; multiple seeds would fix DeBoP's variance but not the unmatched baseline. The efficiency claim is also narrower than it sounds, since it covers inference time only and excludes the MCTS optimization cost, which can be substantial. Those hyperparameter choices with knowledge of the task are a minor concern but not fatal; the structure is still a standard optimize-on-dev, evaluate-on-test procedure.\n\nSo: the method is worth taking seriously, and the paper deserves a real referee. But the version you would publish needs a matched GPT-3.5 baseline on the same test set, multiple seeds with error bars, and a total-cost comparison that includes optimization. Without those, the abstract overclaims. The right read is a conditional accept or major revision, not a desk reject and not a clean accept.\n\nFor your own work, I'd cite it as a prompt-optimization approach for small models, but I would not rely on the GPT-3.5 comparison.","headline":"DeBoP is a genuine, well-structured method for optimizing single demonstrations in lightweight LLMs, but the headline 'surpasses GPT-3.5' claim is not supported by the experiments as reported.","tokens_in":23245,"tokens_out":1502,"would_cite":true,"duration_ms":18006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeBoP automatically optimizes demonstrations so a local 8B model surpasses GPT-3.5 on most of seven hard reasoning tasks.","keywords":["lightweight LLMs","prompt optimization","Monte Carlo tree search","chain-of-thought prompting","in-context learning","demonstration selection","BIG-Bench Hard","behavior optimization"],"falsifier":"Run DeBoP end-to-end on the same seven BBH tasks with, say, ten random seeds and report per-seed accuracy on the 100-sample test sets. If the LLaMA3-8B average no longer exceeds GPT-3.5's published scores within error bars, or if a randomly sampled demonstration performs within noise, the central superiority claim is falsified.","tokens_in":22239,"feed_emoji":"⚡","tokens_out":4643,"duration_ms":41337,"temperature":0.7,"pith_summary":"The paper introduces DeBoP, an automatic prompt optimizer aimed at lightweight LLMs that cannot reliably reflect on or rewrite their own instructions. DeBoP turns prompt optimization into a search over discrete demonstration behaviors: a JSON key-step plan plus an execution, refined by a gradient-free Monte Carlo Tree Search. On seven BIG-Bench Hard tasks, DeBoP-optimized LLaMA3-8B averages 79.4% accuracy, above GPT-3.5's 76.3%, and reduces average inference time by roughly 60% compared to StrategyLLM and Self-Discover. The claim is that small, locally run models can reach or exceed API-scale quality through better demonstrations rather than more parameters.","feed_headline":"DeBoP search lets small LLMs outscore GPT-3.5 on seven hard tasks","feed_subtitle":"Automatic MCTS-selected demonstrations beat the API baseline while cutting inference time by roughly 60 percent.","key_machinery":"The load-bearing object is the demonstration as a question, key-step plan, and execution triple encoded in JSON, with the plan expressed as keys mapped to empty strings that the LwLLM fills in. DeBoP's four phases, PLANNING, COLLECTING, MCTS, and TEACHING, search this space without gradients or an external strong LLM. The MCTS reward is $\\Delta = \\alpha \\cdot \\mathrm{Quant}(\\hat{p}_i) + \\beta \\cdot \\exp(-\\lambda T(\\hat{p}_i))$, combining execution accuracy on the development set with a time-efficiency penalty, and UCB drives node selection.","core_discovery":"The central claim is that the behavior of a lightweight LLM on a task can be directly optimized by searching over structured demonstrations. DeBoP decomposes a demonstration into a standardized JSON plan of key steps and an execution that fills in those steps. Starting from a few plans generated from task guidelines, it scores candidate plans on a 50-sample development set, seeds a forest of demonstrations, and refines them through MCTS with six edit operations: consolidation, decomposition, elaboration, pruning, resampling, and simplification. The selected demonstration is then inserted into the conversation history as a teaching example. Across PIT, DU, SNK, DQA, LD, HB, and MR, DeBoP with LLaMA3-8B reaches 79.4% average accuracy, surpassing the GPT-3.5 reference at 76.3%, while its average inference time of about 5.1 seconds is close to manual CoT and roughly 60% lower than StrategyLLM and Self-Discover.","pith_inferences":["The same discrete search could in principle optimize not only demonstrations but any finite set of reasoning scaffolds, since DeBoP treats the plan as an editable JSON object.","One testable extension is to use a stronger model's generated execution as a richer teaching signal while keeping the LwLLM as the scorer, which the paper's GPT-3.5 cross row hints at but does not pursue.","The MCTS search itself is a form of test-time compute; comparing DeBoP's optimization budget against a simple best-of-N sampling baseline on the same development set would clarify how much of the gain is search versus structured prompting."],"forward_implications":["If DeBoP works as reported, lightweight local models can be pushed past an API-scale baseline like GPT-3.5 on selected reasoning tasks without manual prompt engineering.","The roughly 60% inference-time reduction relative to StrategyLLM and Self-Discover means automatic optimization no longer has to trade away the efficiency advantage of small models.","Because DeBoP optimizes a single demonstration, it implies that the best use of context for a small model may be one high-quality behavioral example rather than several.","The cross-model transfer results suggest demonstrations found on one small model can be reused to improve another model of similar scale."],"supporting_citations":[{"why":"Supplies the BIG-Bench Hard tasks and their challenging reasoning examples used as the evaluation suite.","marker":"Suzgun et al., 2022"},{"why":"Provides the Chain-of-Thought prompting technique from which DeBoP's demonstration structure is derived.","marker":"Wei et al., 2022"},{"why":"StrategyLLM is a primary automatic prompt optimization baseline that DeBoP outperforms.","marker":"Gao et al., 2023a"},{"why":"Self-Discover is the other automatic baseline that DeBoP compares against in accuracy and inference time.","marker":"Zhou et al., 2024"},{"why":"Supplies the UCB rule used in the MCTS selection stage of DeBoP.","marker":"Kocsis and Szepesvári, 2006"},{"why":"Provides the GPT-3.5 reference scores on BBH that DeBoP claims to surpass.","marker":"Opencompass, 2025"}],"fun_headline_variants":["MCTS-tuned prompts push 8B LLaMA past GPT-3.5","DeBoP: 60% faster inference, GPT-3.5-level accuracy","Small LLM outranks GPT-3.5 on 7 tasks via MCTS","Direct behavior search lifts lightweight LLMs beyond GPT-3.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best demonstration found on a fixed 50-sample development set, with one random seed, is robustly better on the held-out 100-sample test set; without multiple seeds or confidence intervals, a few accuracy points may be noise.","fun_headline_variants_meta":{"raw":{"variants":["MCTS-tuned prompts push 8B LLaMA past GPT-3.5","DeBoP: 60% faster inference, GPT-3.5-level accuracy","Small LLM outranks GPT-3.5 on 7 tasks via MCTS","Direct behavior search lifts lightweight LLMs beyond GPT-3.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1477,"prompt_tokens":1002,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":618,"tokens_out":475,"duration_ms":4403,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:14.390238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DeBoP end-to-end on the same seven BBH tasks with, say, ten random seeds and report per-seed accuracy on the 100-sample test sets. If the LLaMA3-8B average no longer exceeds GPT-3.5's published scores within error bars, or if a randomly sampled demonstration performs within noise, the central superiority claim is falsified.","supporting_citations":[],"review_version":1}