Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeBoP automatically optimizes demonstrations so a local 8B model surpasses GPT-3.5 on most of seven hard reasoning tasks.

desk verdict DeBoP is a genuine, well-structured method for optimizing single demonstrations in lightweight LLMs, but the headline 'surpasses GPT-3.5' claim is not supported by the experiments as reported. read the letter →

arxiv 2506.06401 v1 pith:USHSMDWC submitted 2025-06-06 cs.CL cs.AIcs.LG

Hongming Yang , Shi Lin , Jun Shao , Changting Lin , Donghai Zhu , Meng Han , Qinglei Kong This is my paper
classification cs.CLcs.AIcs.LG
keywords lightweightLLMspromptoptimizationMonteCarlotreesearchchain-of-thoughtpromptingin-contextlearningdemonstrationselectionBIG-BenchHardbehavior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DeBoP, an automatic prompt optimizer aimed at lightweight LLMs that cannot reliably reflect on or rewrite their own instructions. DeBoP turns prompt optimization into a search over discrete demonstration behaviors: a JSON key-step plan plus an execution, refined by a gradient-free Monte Carlo Tree Search. On seven BIG-Bench Hard tasks, DeBoP-optimized LLaMA3-8B averages 79.4% accuracy, above GPT-3.5's 76.3%, and reduces average inference time by roughly 60% compared to StrategyLLM and Self-Discover. The claim is that small, locally run models can reach or exceed API-scale quality through better demonstrations rather than more parameters.

What carries the argument

The load-bearing object is the demonstration as a question, key-step plan, and execution triple encoded in JSON, with the plan expressed as keys mapped to empty strings that the LwLLM fills in. DeBoP's four phases, PLANNING, COLLECTING, MCTS, and TEACHING, search this space without gradients or an external strong LLM. The MCTS reward is $\Delta = \alpha \cdot \mathrm{Quant}(\hat{p}_i) + \beta \cdot \exp(-\lambda T(\hat{p}_i))$, combining execution accuracy on the development set with a time-efficiency penalty, and UCB drives node selection.

What would settle it

Run DeBoP end-to-end on the same seven BBH tasks with, say, ten random seeds and report per-seed accuracy on the 100-sample test sets. If the LLaMA3-8B average no longer exceeds GPT-3.5's published scores within error bars, or if a randomly sampled demonstration performs within noise, the central superiority claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the behavior of a lightweight LLM on a task can be directly optimized by searching over structured demonstrations. DeBoP decomposes a demonstration into a standardized JSON plan of key steps and an execution that fills in those steps. Starting from a few plans generated from task guidelines, it scores candidate plans on a 50-sample development set, seeds a forest of demonstrations, and refines them through MCTS with six edit operations: consolidation, decomposition, elaboration, pruning, resampling, and simplification. The selected demonstration is then inserted into the conversation history as a teaching example. Across PIT, DU, SNK, DQA, LD, HB, and MR, DeBoP with LLaMA3-8B reaches 79.4% average accuracy, surpassing the GPT-3.5 reference at 76.3%, while its average inference time of about 5.1 seconds is close to manual CoT and roughly 60% lower than StrategyLLM and Self-Discover.

Load-bearing premise

The load-bearing premise is that the best demonstration found on a fixed 50-sample development set, with one random seed, is robustly better on the held-out 100-sample test set; without multiple seeds or confidence intervals, a few accuracy points may be noise.

Editorial extensions

If this is right

  • If DeBoP works as reported, lightweight local models can be pushed past an API-scale baseline like GPT-3.5 on selected reasoning tasks without manual prompt engineering.
  • The roughly 60% inference-time reduction relative to StrategyLLM and Self-Discover means automatic optimization no longer has to trade away the efficiency advantage of small models.
  • Because DeBoP optimizes a single demonstration, it implies that the best use of context for a small model may be one high-quality behavioral example rather than several.
  • The cross-model transfer results suggest demonstrations found on one small model can be reused to improve another model of similar scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same discrete search could in principle optimize not only demonstrations but any finite set of reasoning scaffolds, since DeBoP treats the plan as an editable JSON object.
  • One testable extension is to use a stronger model's generated execution as a richer teaching signal while keeping the LwLLM as the scorer, which the paper's GPT-3.5 cross row hints at but does not pursue.
  • The MCTS search itself is a form of test-time compute; comparing DeBoP's optimization budget against a simple best-of-N sampling baseline on the same development set would clarify how much of the gain is search versus structured prompting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces DeBoP, a four-phase prompt optimization method for lightweight LLMs. DeBoP generates task-specific key-step plans in JSON form, collects demonstrations produced by the LwLLM on a development set, searches over mutated demonstrations with a gradient-free Monte Carlo Tree Search whose reward combines development-set accuracy and inference time, and finally teaches the selected demonstration to the model. Experiments compare LLaMA3-8B and LLaMA3.2-3B against Direct Prompting, CoT Prompting, StrategyLLM, Self-Discover, and GPT-3.5 on seven BIG-Bench Hard tasks, reporting accuracy and inference time. The central claims are that DeBoP outperforms recent automatic prompt optimizers on most tasks, that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 on most tasks, and that it reduces average inference time by roughly 60% relative to other automatic methods.

Significance. The paper addresses a relevant and timely problem: automatic prompt optimization for small local models without relying on strong LLM meta-cognition or external APIs. The method is clearly described, and the paper includes useful strengths: a public code release, ablations of the pipeline stages, a cross-model generalization study, an explicit single-versus-multi-demonstration analysis, and a supplementary investigation on a distilled reasoning model. If the comparative claims hold, DeBoP would be a practical contribution for resource- and privacy-constrained deployments. However, the headline claims currently rest on a statistically thin evaluation and an unmatched GPT-3.5 baseline, so the strength of the evidence is not yet commensurate with the claims.

major comments (4)
  1. [Section 3.3 / Table 1] The claim that DeBoP-optimized LLaMA3-8B surpasses GPT-3.5 on most tasks is not yet supported because the GPT-3.5 accuracies are cited from OpenCompass rather than produced on the same 100-sample Stest. Since the per-task margins in Table 1 are as small as 1 to 9 percentage points, a different subset, prompt format, decoding setting, or answer-extraction function could fully explain the differences. Please run GPT-3.5 on the identical Stest under the same evaluation protocol as the LwLLMs, or rephrase/remove the claim.
  2. [Section 3.3 / Table 1] All DeBoP results come from a single 50-sample development split and a single 100-sample test split with fixed seed 42, with no confidence intervals or significance tests. With 100 test items, a difference of roughly 5 percentage points is within one standard error, so several of the per-task comparisons in Table 1 could be sampling noise. Please report multiple seeds with confidence intervals, or provide significance tests; this is essential for the central 'surpasses GPT-3.5 on most tasks' claim.
  3. [Section 2.2.3 / Section 3.3] The MCTS reward in Section 2.2.3 depends on hyperparameters alpha, beta, and lambda, and the search also depends on the UCB exploration coefficient c, the number of key-step plans k, the 50-iteration budget, and the 20% early-stop probability. These values are fixed without sensitivity analysis, and since Sdev is drawn from the same BBH tasks used for test, selecting these values with knowledge of the tasks is a correctness risk. Please provide a sensitivity study or clearly state how each hyperparameter was chosen.
  4. [Section 3.4.2 / Table 2] The approximately 60% efficiency reduction is calculated from per-example inference time only, but the abstract's phrasing ('reducing computational time') could be read as end-to-end optimization time. The planning, collecting, and MCTS expansion costs of DeBoP are not reported, and it is unclear whether the StrategyLLM and Self-Discover time measurements include their generation/optimization stages. Please clarify precisely which time is measured and whether the efficiency claim covers only test-time inference or the full pipeline.
minor comments (5)
  1. [Section 3.4.3 / Figure 4] The ablation text defines PCT and PCMT with the same phrase 'PLANNING + COLLECTING + MCTS + TEACHING', which makes the ablation legend and conclusion difficult to parse; please correct the labels or the text.
  2. [Table 3] Table 3 is hard to read because the first row appears to serve both as a header and as data. Please separate the model used to generate demonstrations from the model applying DeBoP, and clearly mark the no-DeBoP baseline numbers.
  3. [Section 3.3 / References] The GPT-3.5 numbers are cited as 'Opencompass, 2025' without an author list, version, or exact retrieval URL; please provide a complete citation with the specific benchmark configuration and access date.
  4. [Section 2.2.2] The selection probability formula for Probi has a minor typographical issue with redundant parentheses around Quant(pi); please tidy the equation.
  5. [Section 6 / Limitations] The Limitations section discusses MCTS breadth and potential misuse, but does not mention the statistical limitations of the evaluation or the unmatched GPT-3.5 baseline; please acknowledge these in the final text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DeBoP is a standard optimize-on-development-set, evaluate-on-test-set pipeline against external BBH tasks, with no equation-level reduction of the central claim to its inputs.

full rationale

The paper's derivation chain is self-contained rather than circular. DeBoP explicitly defines its objective as b* = arg max Eval(LwLLM(db), Sdev) over a development set Sdev, then evaluates the selected demonstration on a separate 100-sample subset Stest (Section 3.3: 'we conducted all experiments on a randomly selected 50-sample subset as Sdev and a 100-sample subset as Stest from the original dataset, using a fixed random seed (42)'). The reported accuracies on Stest are therefore not equivalent by construction to the quantities optimized on Sdev; this is standard empirical generalization, not fitted-input-as-prediction. Hyperparameters such as alpha=2 and beta/lambda=1/0.5 are stated as fixed experimental choices, not fitted to the test set. The mutation operators and teaching mechanism are method components, not disguised restatements of the benchmark outcomes. No load-bearing self-citation appears: the reference list contains no self-citations by the authors, and the method does not invoke a uniqueness theorem or prior-work ansatz to force its design. The GPT-3.5 comparison is cited from OpenCompass ('The result of GPT3.5 was referenced from Opencompass, 2025'), which is a legitimate external benchmarking source, although it introduces a potential evaluation-matching concern about identical subsets, prompts, and decoding settings. That concern affects the strength of the empirical claim but is not circularity under the stated criteria. The paper also frankly acknowledges limitations in the MCTS expansion process and potential misuse, further indicating that no hidden circular derivation is being used to prop up the central claim.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The free parameters are hyperparameters of the search and reward; the axioms are empirical assumptions about LwLLM behavior and the representativeness of the evaluation split.

free parameters (6)
  • alpha (plan selection pressure) = 2
    Used in the COLLECTING phase to amplify selection of high-accuracy plans; no sensitivity analysis is reported.
  • MCTS reward weights alpha, beta, lambda = 1, 1, 0.5
    Control the trade-off between execution accuracy and time efficiency in the Simulation stage; chosen by hand without ablation.
  • UCB exploration coefficient c
    Appears in the UCB selection equation but no value is reported anywhere in the paper.
  • Number of key-step plans k = 5
    Sets the size of Splan; the paper does not test other values.
  • MCTS iteration budget and early-stop probability = 50 iterations, 20% early stopping
    Computational budget that shapes the search; fixed by the authors.
  • Dev/test split size and seed = 50/100, seed 42
    One random split is used; no repeated sampling or cross-validation is reported.
assumptions (4)
  • domain assumption The LwLLM will follow the JSON key-step plan and produce parseable executions with a final answer.
    The whole COLLECTING and TEACHING mechanism depends on structured JSON outputs; no parse-failure rate is reported.
  • domain assumption The six mutation operators applied by the LwLLM produce demonstrations that improve or preserve dev-set accuracy.
    MCTS exploration assumes these operators are useful; the paper only evaluates the final combined pipeline, not each operator.
  • ad hoc to paper The reward Delta = alpha*Quant + beta*exp(-lambda*T) represents the true deployment objective.
    The additive combination and exponential time penalty are design choices without a derivation from user requirements.
  • domain assumption The 50-sample development set is representative of each BBH task distribution.
    Section 3.3 uses a single random subset; if the dev set is not representative, the selected demonstration may overfit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs." pith.science (2026). https://pith.science/paper/USHSMDWC

@misc{pith2026250606401,
  author       = {Pith},
  title        = {Pith review of: Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USHSMDWC}},
  note         = {Machine review of arXiv:2506.06401}
}
read the original abstract

Lightweight Large Language Models (LwLLMs) are reduced-parameter, optimized models designed to run efficiently on consumer-grade hardware, offering significant advantages in resource efficiency, cost-effectiveness, and data privacy. However, these models often struggle with limited inference and reasoning capabilities, which restrict their performance on complex tasks and limit their practical applicability. Moreover, existing prompt optimization methods typically rely on extensive manual effort or the meta-cognitive abilities of state-of-the-art LLMs, making them less effective for LwLLMs. To address these challenges, we introduce DeBoP, a new Direct Behavior Optimization Paradigm, original from the Chain-of-Thought (CoT) prompting technique. Unlike CoT Prompting, DeBoP is an automatic optimization method, which focuses on the optimization directly on the behavior of LwLLMs. In particular, DeBoP transforms the optimization of complex prompts into the optimization of discrete, quantifiable execution sequences using a gradient-free Monte Carlo Tree Search. We evaluate DeBoP on seven challenging tasks where state-of-the-art LLMs excel but LwLLMs generally underperform. Experimental results demonstrate that DeBoP significantly outperforms recent prompt optimization methods on most tasks. In particular, DeBoP-optimized LwLLMs surpass GPT-3.5 on most tasks while reducing computational time by approximately 60% compared to other automatic prompt optimization methods.

Figures

Figures reproduced from arXiv: 2506.06401 by the authors.

Figure 1
Figure 1. A toy example demonstrating how Strate￾gyLLM (Gao et al., 2023a) fails on LwLLMs. The strategies used by StrategyLLM are highlighted in blue. Due to the limited reasoning capabilities of LwLLMs, the accumulation of redundant reasoning, wrong rea￾soning, and omitted details leads to an incorrect final answer. To address these challenges, we in this pa￾per introduce DeBoP, a novel Direct Behavior Optimization Paradigm… view at source ↗
Figure 2
Figure 2. Overview of the DeBoP framework. As shown in the lower left corner, the demonstration, which serves as [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pareto Frontier Analysis [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Ablation study on individual DeBoP phase. P, C, M, and T are the initials of the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: PLANNING Prompt with two stages (stage 1). 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: PLANNING Prompt with two stages (stage 2, part 1) 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: PLANNING Prompt with two stages (Stage 2, Part 2) 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: COLLECTING Phase Prompt. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: MCTS Consolidation Prompt. MCTS Decomposition Prompt For the given task: {task definition} {question} we already have the following JSON structured solution: {demonstration} Could you determine whether any steps can be easily split to improve the clarity and accuracy o…
Figure 10
Figure 10. Figure 10: MCTS Decomposition Prompt. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: MCTS Elaboration Prompt. MCTS Pruning Prompt For the given task: {task definition} {question} we already have the following JSON structured solution: {demonstration} Could you determine whether one step (the least consequential one) can be removed without harming the …
Figure 12
Figure 12. Figure 12: MCTS Pruning Prompt. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: MCTS Resampling Prompt. MCTS Simplification Prompt For the given task: {task definition} {question} we already have the following JSON structured solution: {demonstration} Could you simplify and refine the reasoning flow (including both keys and values) to improve con…
Figure 14
Figure 14. Figure 14: MCTS simplification Prompt. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: DP&CoT Prompt of Penguin in a Table. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: DP&CoT Prompt of Date Understanding. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: DP&CoT Prompt of Snark. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: DP&CoT Prompt of Disambiguation QA. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: DP&CoT Prompt of Logical Deduction. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: DP&CoT Prompt of Hyperbaton. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: DP&CoT Prompt of Movie Recommendation. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 14 canonical work pages

  1. [3]

    In The Eleventh International Conference on Learning Representations

    Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations. Chang Gao, Haiyun Jiang, Deng Cai, Shuming Shi, and Wai Lam. 2023a. Strategyllm: Large language models as strategy generators, executors, optimizers, and evaluators for problem solving. arXiv preprint arXiv:2311.08803. Luyu Gao, Aman ...

    arXiv

  2. [6]

    arXiv preprint arXiv:2210.02406

    Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. Levente Kocsis and Csaba Szepesvári

    arXiv

  3. [7]

    Less is More: Summary of Long Instructions is Better for Program Synthesis

    Less is more: Summary of long instructions is better for program synthesis. arXiv preprint arXiv:2203.08597. Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhu Zhang

    work page Pith review arXiv

  4. [8]

    arXiv preprint arXiv:2305.14623

    Self-checker: Plug-and-play modules for fact-checking with large language mod- els. arXiv preprint arXiv:2305.14623. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023a. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477. Shengcai Liu, Caishun Chen, Xi...

    arXiv

  5. [10]

    arXiv preprint arXiv:2301.13379

    Faithful chain-of- thought reasoning. arXiv preprint arXiv:2301.13379. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

    arXiv

  6. [11]

    arXiv preprint arXiv:2110.08294

    Coherence boosting: When your pretrained language model is not paying enough attention. arXiv preprint arXiv:2110.08294. Meta. 2024a. Llama 3 8b instruct. Accessed: 2025-02-

    arXiv 2025

  7. [13]

    Hugging Face

    Llama 3.1-405b. Hugging Face. Accessed: 2025-02-13. Ning Miao, Yee Whye Teh, and Tom Rainforth

    work page 2025

  8. [14]

    arXiv preprint arXiv:2308.00436

    Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00436. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al

    arXiv

Show all 38 references
  1. [15]

    arXiv preprint arXiv:2210.17517

    Lila: A unified benchmark for mathemat- ical reasoning. arXiv preprint arXiv:2210.17517. Opencompass

    arXiv

  2. [16]

    Meta. 2024b. Llama 3.2 3b instruct. Accessed: 2025- 02-16. Meta

    2025

  3. [17]

    Accessed: 2025- 02-14

    BBH Dataset. Accessed: 2025- 02-14. 10 Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang

    2025

  4. [18]

    arXiv preprint arXiv:2308.03188

    Automatically correcting large language models: Sur- veying the landscape of diverse self-correction strate- gies. arXiv preprint arXiv:2308.03188. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis

    arXiv

  5. [19]

    arXiv preprint arXiv:2210.03350

    Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chen- guang Zhu, and Michael Zeng

    arXiv

  6. [20]

    gradient descent

    Automatic prompt optimization with "gradient descent" and beam search. arXiv preprint arXiv:2305.03495. Hua Xuan Qin, Shan Jin, Ze Gao, Mingming Fan, and Pan Hui

    arXiv

  7. [21]

    https://www.reuters.com/technology/chatgpt- fever-spreads-us-workplace-sounding-alarm-some- 2023-08-11

    ChatGPT fever spreads in US workplace, sounding alarm for some. https://www.reuters.com/technology/chatgpt- fever-spreads-us-workplace-sounding-alarm-some- 2023-08-11. Accessed on 2025-02-12. Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin

    2023

  8. [22]

    arXiv preprint arXiv:2308.10379

    Algorithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2308.10379. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al

    arXiv

  9. [23]

    arXiv preprint arXiv:2206.04615

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. Hong Sun, Xue Li, Yinchuan Xu, Youkow Homma, Qi Cao, Min Wu, Jian Jiao, and Denis Charles

    arXiv

  10. [24]

    arXiv preprint arXiv:2307.07415

    Autohint: Automatic prompt optimization with hint generation. arXiv preprint arXiv:2307.07415. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al

    arXiv

  11. [25]

    arXiv preprint arXiv:2210.09261

    Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yu- jin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W White, Longqi Yang, et al

    arXiv

  12. [26]

    arXiv preprint arXiv:2312.03863

    Efficient large language models: A survey. arXiv preprint arXiv:2312.03863. Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu

    arXiv

  13. [27]

    arXiv preprint arXiv:2310.16427

    Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:2310.16427. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al

    arXiv

  14. [28]

    arXiv preprint arXiv:2402.18649

    A new era in llm se- curity: Exploring security concerns in real-world llm- based systems. arXiv preprint arXiv:2402.18649. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al

    arXiv

  15. [29]

    arXiv preprint arXiv:2205.10625

    Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.10625. Jianpeng Zhou, Wanjun Zhong, Yanlin Wang, and Ji- ahai Wang

    arXiv

  16. [30]

    arXiv preprint arXiv:2402.03620

    Self- discover: Large language models self-compose rea- soning structures. arXiv preprint arXiv:2402.03620. 11 A Prompts in DeBoP A.1 Prompts in P LANNING The prompts used in the PLANNING phase are il- lustrated in Figs. 5, 6, and

    arXiv

  17. [31]

    Today is 3/5, and it is Jane’s second time in the year 1973 to see a me- teor shower. What is the date 24 hours later in MM/DD/YYYY?

    B Details of Baseline Methods B.1 Prompts used in DP and CoT Prompting The prompts of the seven BBH tasks with DP and CoT prompting are illustrated in the following fig- ures: Fig. 15 for PIT, Fig. 16 for DU, Fig. 17 for SNK, Fig. 18 for DQA, Fig. 19 for LD, Fig. 20 for HB, an...

    1973

  18. [32]

    Therefore, we can get the general and instructive solution:

    reports that demonstration-based prompting may hurt the per- formance of cutting-edge reasoning models such as DEEP SEEK R1. To verify whether this claim also holds in the LwLLM regime, we ran ad- ditional experiments on two BBH tasks—PIT 12 and DU—using DeepSeek-R1-LLaMA3-8B ...

    2025

  19. [33]

    So the answer is (D)

    10 days before today is December 14, 1937, that is 12/14/1937. So the answer is (D). Answer: (D) Question: Tomorrow is 11/12/2019. What is the date one year ago from today in MM/DD/YYYY? Options: (A) 09/04/2018 (B) 11/11/2018 (C) 08/25/2018 (D) 11/02/2018 (E) 11/04/2018 Soluti...

    1937

  20. [1937]

    What is the date 10 days ago in MM/DD/YYYY? Options: (A) 12/14/2026 (B) 12/14/1950 (C) 12/14/2007 (D) 12/14/1937 (E) 07/14/1938 (F) 12/14/1988 Solution: If today is Christmas Eve of 1937, then today’s date is December 24,

    2026

  21. [1958]

    It is their 5-year anniversary today. What is the date tomorrow in MM/DD/YYYY? Options: (A) 01/11/1961 (B) 01/03/1963 (C) 01/18/1961 (D) 10/14/1960 (E) 01/03/1982 (F) 12/03/1960 Solution: If Jane and John married on Jan 2, 1958, and if it is their 5-year anniversary today, the...

    1961

  22. [1963]

    they" might be referring to. There are two possible referents for

    The date tomorrow is Jan 3, 1963, that is 01/03/1963. So the answer is (B). Answer: (B) Question: {question} Figure 16: DP&CoT Prompt of Date Understanding. 22 DP&CoT Prompt of Snark Question: Which statement is sarcastic? Options: (A) Yes, because having interests and activel...

    1963

  23. [1994]

    These are all famous movies produced in the past few decades. Amongst all the options, the only movie similar to these ones seems to be Edge of Tomorrow (action, adventure, crime, mystery; 2014), as it is also a science-fiction movie and features Tom Cruise. So the answer is (...

    2014

  24. [1995]

    Amongst all the options, the only movie similar to these ones seems to be Forrest Gump (comedy, drama, romance; 1994)

    These are all famous Hollywood movies produced around the 1990s. Amongst all the options, the only movie similar to these ones seems to be Forrest Gump (comedy, drama, romance; 1994). So the answer is (C). Answer: (C) Question: Find a movie similar to Minority Report, Total Re...

    1994

  25. [2000]

    So the answer is (C)

    Amongst all the options, the only movie similar to these ones seems to be The Princess Bride (1987). So the answer is (C). Answer: (C) Question: Find a movie similar to Twister, The Silence of the Lambs, Independence Day, Braveheart. Options: (A) They Shoot Horses (B) Don’t Th...

    1987

  26. [2021]

    arXiv preprint arXiv:2108.07258

    On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258. Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. 2023a. Skills-in-context prompting: Unlocking composi- tionality in large language models. arXiv p...

    arXiv

  27. [2022]

    arXiv preprint arXiv:2211.12588

    Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023c. Teaching large language models to self-debug. arXiv preprint arXiv:2304...

    arXiv

  28. [2023]

    arXiv preprint arXiv:2303.05398

    Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal

    arXiv

  29. [2024]

    In 2024 IEEE Congress on Evolutionary Computation (CEC), pages 1–8

    Large language models as evolutionary optimizers. In 2024 IEEE Congress on Evolutionary Computation (CEC), pages 1–8. IEEE. Tengxiao Liu, Qipeng Guo, Yuqing Yang, Xiangkun Hu, Yue Zhang, Xipeng Qiu, and Zheng Zhang. 2023b. Plan, verify and switch: Integrated rea- soning with d...

    2024 arXiv

  30. [2025]

    arXiv preprint arXiv:2501.12948

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948. Shima Imani, Liang Du, and Harsh Shrivastava

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.