Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FORMAT-ADAPTER claims that an LLM can generate and select its own task-appropriate reasoning formats, improving multi-answer reasoning accuracy by 4.3% on average over prior methods on math and commonsense benchmarks.

desk verdict Useful extension of multi-format self-consistency, but the theory is not grounded and the headline 4.3% needs a cleaner evaluation before it can be trusted. read the letter →

arxiv 2506.23133 v1 pith:O32PJZI4 submitted 2025-06-29 cs.CL

Dingzirui Wang , Xuanliang Zhang , Rongyu Cao , Longxu Dou , Xianzhen Luo , Yingwei Ma , Qingfu Zhu , Wanxiang Che
show 3 more authors
Binhua Li Fei Huang Yongbin Li
This is my paper
classification cs.CL
keywords LLMreasoningformatsformatadaptationself-consistencyLLM-as-judgemulti-answervotingmathcommonsense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that the best reasoning format is different for different questions, so fixing one human-designed format wastes accuracy. Format-Adapter first derives an error estimate for ensembles of answers, then uses the LLM itself to generate candidate formats—languages, math notation, explanation depth—rewrite the prompt for each format, score the resulting answers, and greedily keep the formats that lower the estimated error before voting. On GSM8K-Hard, MATH500, ARC-C-Hard, and GPQA, the method reports average improvements of 4.1% over the best single-format baselines and 4.7% over multi-format baselines, an overall 4.3% average gain. If correct, this removes the need for human-labeled prompt formats and turns format adaptation into an automatic, per-question test-time choice.

What carries the argument

The load-bearing object is Equation 5, a two-term error identity for ensembles of predictors: $E_D[L(\bar\phi,y)] = \frac{1}{m}\sum_i E_D[L(\phi\circ f_i,y)] - E_D[\frac{1}{m}\sum_i L(\phi\circ f_i,\bar\phi)]$, where $f_i$ are reasoning formats and $\bar\phi$ is the average prediction. The first term is the average per-format error and the second is the divergence between formats, so lowering the first or raising the second lowers the ensemble error. Format-Adapter estimates the first term with LLM judge scores, approximates $\bar\phi$ by the most frequent answer, and runs a greedy selection that keeps a format only if it reduces the estimated value of Equation 5; the final output is the most frequent answer among the selected formats.

What would settle it

On a random sample of GSM8K-Hard or MATH500 questions, rerun FORMAT-ADAPTER's greedy selection twice—once with the paper's LLM judge scores and once with ground-truth correctness labels replacing the scores—and compare the Vote accuracy of the two resulting format subsets; if the judge-selected subset does not beat voting over all formats, the selection mechanism is not the source of the reported gain.

Watch

Extended reading notes

Core claim

The paper establishes Equation 5: when one model generates answers under several reasoning formats, the ensemble error decomposes into the average per-format error minus the average divergence between formats. Because the divergence term is subtracted, combining formats that disagree while remaining individually accurate lowers the voting error without retraining the model. FORMAT-ADAPTER operationalizes this by having the LLM generate candidate formats, rewrite the task instruction for each, score each answer with an LLM judge, and greedily select the subset that minimizes the estimated error, then answer by majority vote. The paper reports that this beats single-format self-consistency and prior multi-format methods, with the largest gains on questions where the human-labeled format was a poor fit.

Load-bearing premise

The selection step assumes the LLM judge's 1-10 scores are a trustworthy stand-in for the true correctness term in Equation 5, but the paper's own evaluation reports those scores align with correctness only 45.7%-66.2% of the time and tend to be too high, so the chosen formats may not be the ones that actually minimize reasoning error.

Editorial extensions

If this is right

  • For a correct Equation 5, adding a format that disagrees with existing ones but is not more error-prone should improve the vote, making format diversity a direct lever on accuracy.
  • Since formats are generated by the LLM rather than labeled by humans, the method should transfer to new tasks at the cost of one prompt, with no per-task annotation.
  • Table 1's Oracle numbers (up to 96.9% on ARC-C-Hard for GPT-4o) imply the generated formats already contain the right answer in almost every question, so the remaining Vote-Oracle gap is mostly a selection problem.
  • The paper's difficulty pattern—small gains on hard datasets with small models—indicates format adaptation surfaces knowledge the model already has rather than adding missing knowledge.
  • Because a higher number of formats consistently helps, the method gives a practical reason to scale format count rather than just sample count in self-consistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own score-quality results (45.7%-66.2%) suggest a cheaper extension: replace the LLM judge with a verifier or execution-based check where available, and compare whether the greedy selection keeps its gains; the paper itself lists training-data-based format selection as a possible fix.
  • The error identity is not specific to "format"; a natural test is whether the same selection loop works over sampling temperatures, decoding strategies, or tool-use variants, which would show whether diversity per se drives the gain.
  • Since the best category differs by task and model (Appendix D), one could rank formats once per task and reuse that ranking at inference, avoiding per-question scoring cost while keeping most of the gain.
  • If judge scores are inflated, the reported 4.3% may partly misattribute gains from diversity to selection; a direct ablation replacing judge scores with ground-truth labels would settle how much selection contributes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FORMAT-ADAPTER, a method that uses LLMs to generate multiple reasoning formats for a given task, generate answers in each format, score those answers with an LLM judge, and greedily select a subset of formats by minimizing an error measure derived from ensemble theory (Eq. 5). The stated aim is to avoid human-labeled reasoning formats while adapting suitable formats to each task. Experiments on GSM8K-Hard, MATH500, ARC-C-Hard, and GPQA with Llama3.1 and GPT-4o show average Vote improvements of 4.1% over single-format baselines and 4.7% over multi-format baselines, for an overall claimed 4.3% average improvement. The paper also provides ablations, oracle upper bounds, and analyses of format categories, scale, and selection ratios.

Significance. If the central claim were supported, the paper would offer a practical, human-label-free way to adapt reasoning formats and improve LLM reasoning, with consistent Vote gains over strong baselines (SC, ToT, DTV) in Tables 1-2 and high Oracle numbers indicating that format diversity is valuable. The authors report useful robustness runs (Table 9), a comparison against using all formats (Table 7), and an honest assessment of judge score quality (Appendix F.4). However, the theoretical derivation of the error measure is flawed, and the selection mechanism—the load-bearing component of the method—is driven by LLM judge scores whose quality is reported to be near chance in several settings. The evidence therefore does not yet establish that FORMAT-ADAPTER selects 'suitable' formats, as opposed to benefiting from format diversity alone.

major comments (3)
  1. [Appendix A.1 and §2.1] Equation 4's proof is circular: the premise 'lim_{m→∞} ¯φ = φ ∘ f' already asserts that the ensemble average converges to the model's inherent performance, and the proof then substitutes δ_i → 0 to conclude E_D[L(¯φ, y)] = E_D[L(φ∘f, y)], which is the same statement. Lemma 1 is also invalid, since convergence of the Cesàro average avg(δ_i) to 0 does not imply lim_{m→∞} δ_m = 0 (e.g., δ_m = (-1)^m). The theoretical justification for why single-format sampling 'only enhances robustness' is therefore not established.
  2. [§3.3, §3.4, Appendix F.4, Table 7] The selection mechanism minimizes Eq. 5 with the true error L(φ∘f_i, y) replaced by LLM judge scores, but Appendix F.4 reports score quality of only 45.7–66.2% across settings, at or below chance on several (e.g., GPQA with Llama3.1-8b: 45.7%). Section 4.4.4(iii) admits the judge assigns overly high scores so that many selected formats do not contain the correct answer. Table 7 shows that on GPQA with Llama3.1-8b, FORMAT-ADAPTER achieves 33.9 EM—identical to using all formats without selection—exactly the setting with the lowest score quality. The paper's central claim that FORMAT-ADAPTER selects suitable formats is not supported; the observed gains could arise from format diversity or from the divergence term in Eq. 5 rather than from correctly identifying formats that minimize reasoning error.
  3. [§4.3, Table 3] The '- Select' ablation in Table 3 replaces the greedy selection with majority voting over the highest-scored answer, so it removes the entire selection algorithm rather than isolating the quality of the judge signal. Because Appendix F.4 shows the judge is near-random in several settings, this ablation cannot establish that the performance drop is due to the loss of suitable-format identification. A random-format-selection baseline or an oracle selection using ground-truth labels (e.g., on a development set) is needed to determine whether the greedy selection contributes beyond format diversity; without such a test, the mechanism that distinguishes FORMAT-ADAPTER from simply using all formats remains unvalidated.
minor comments (5)
  1. [Throughout] The phrase 'the prove of Equation 4' and similar occurrences should be 'the proof of Equation 4'; also 'we adapt analysis experiments' should be 'we adopt analysis experiments.'
  2. [§1 and References] GPQA is cited as (Yadav et al., 2019) in the introduction, but the correct reference is Rein et al. (2024), which is listed in the bibliography; MATH is cited as (Saxton et al., 2019), but Saxton et al. (2019) is a different benchmark, and the MATH dataset should be cited to Hendrycks et al. (2021).
  3. [§4.4.3, Figure 6] The text states that performance 'initially increases significantly, then stabilizes, and finally experiences another notable rise,' but Figure 6 does not clearly show a consistent final rise across all four settings; please quantify the trend or refine the description.
  4. [§4.4.2, Figure 5] The symbol ¯∆EM is used in the figure and caption but is not defined in the text; please define it explicitly.
  5. [§3.4] The description 'we take the average prediction ¯φ(x) as the answer appearing most frequently among all outcomes' is better stated as taking the majority vote, since L is a 0/1 loss and the 'average' is a mode, not an arithmetic mean.

Circularity Check

1 steps flagged · score 2.0 of 10

One self-definitional step in the derivation of Equation 4, but it is not load-bearing for the empirical 4.3% claim; the main pipeline is evaluated against external benchmarks and no ground-truth label is fitted.

  1. self definitional [Section 2.1, Equation 4; Appendix A.1]
    "We assume an ideal scenario where the average of all predictors represents the inherent performance of the model, i.e., lim_{m→∞} ¯ϕ = ϕ ◦ f. It can be proven that the error in generating multiple answers using a single reasoning format satisfies: E_D[L(¯ϕ, y)] = E_D[L(ϕ ◦ f, y)] (4). ... Considering that: ¯ϕ = 1/m Σ ϕ_i = 1/m Σ ϕ ◦ f (δ_i → 0) = ϕ ◦ f."

    The theorem's premise already states the conclusion: assuming the ensemble average converges to ϕ∘f is exactly the claim that the ensemble error equals that of the noiseless model. The Appendix proof makes this explicit by taking δ_i → 0, so every predictor collapses to ϕ∘f and the ensemble average equals ϕ∘f by construction. Equation 4 therefore restates the assumption rather than proving that voting removes perturbation. This is a genuine reduction of the claimed proof to its input, but it is confined to the motivational theory; the empirical pipeline and the 4.3% improvement do not depend on this equation being non-trivial.

full rationale

The only circular step I can exhibit is the derivation of Eq. 4, where the assumption lim ¯ϕ = ϕ∘f is the same as the conclusion and the proof forces δ_i→0. This step is not load-bearing for the main claim: the Format-Adapter pipeline (format generation, answer generation, scoring, greedy selection) is evaluated on held-out answers against ground-truth labels, and no parameter is fitted to those labels. The LLM-judge scores in §3.3 are a proxy for the unknown correctness term; Appendix F.4 shows their quality is low (45.7–66.2%), but that is an empirical weakness of the selection mechanism, not circularity, since the judge is not constructed from the target outputs. Self-citations (FlexTaF, CLSP, MultiPoT) are used only as baselines, not as load-bearing justification. Overall, the central empirical claim has independent content, so the score is 2 rather than higher.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The method relies on a small set of unproven premises: the ensemble error decomposition of Wood et al. applies to the hard 0/1 loss used here; the sampling noise can be modeled as vanishing perturbations; the LLM judge scores estimate the label-dependent error term accurately enough for greedy selection; the curated 'Hard' subsets represent the full benchmarks; and the mode is the right final decision rule. The only hand-set quantity is the number of formats, which is generated per task and model and is shown to affect performance.

free parameters (1)
  • Number of generated reasoning formats per task and model (M) = varies, e.g., 15 for GSM8K-Hard with Llama-3.1-8B, 43 for GPQA with Llama-3.1-8B
    M is produced by the LLM format-generation prompt and is not standardized across baselines. Figure 6 shows performance increases with M, so the magnitude of the reported improvement depends on this uncontrolled quantity.
assumptions (5)
  • domain assumption Ensemble error decomposition (Eq 2) holds for the 0/1 loss L and the ensemble predictor (mode) used in this paper
    Eq 2 is attributed to Wood et al. (2024), whose diversity decomposition is not generally exact for hard-vote 0/1 loss; the paper applies it without stating the conditions. This underpins Eq 5 and the selection objective in Section 3.4.
  • ad hoc to paper The sampling variability of the model is captured by an additive perturbation delta-i that vanishes in the limit
    Introduced in Section 2.1 (Eq 3) and Lemma 1 in Appendix A.1; the limit assumption is equivalent to assuming the ensemble average equals the unperturbed model, which makes Eq 4's proof circular.
  • domain assumption LLM judge scores (1 to 10, rescaled) faithfully estimate E_D[L(phi-of-fi, y)]
    The selection step in Sections 3.3 and 3.4 uses these scores to stand in for the unknown true-label error. Appendix F.4 reports score quality between 45.7% and 66.2%, so this assumption is weak.
  • domain assumption The 'Hard' subsets (256 questions not well solved by current LLMs) are representative of the full benchmarks
    Section 4.1.1 defines GSM8K-Hard and ARC-C-Hard by this criterion but does not specify the sampling rule or the model used; all main GSM8K and ARC results are on these subsets only.
  • domain assumption Majority voting over the selected formats is the near-optimal decision rule given imperfect scores
    Section 3.4 chooses the most frequent answer instead of the highest-scored one 'due to inherent scoring errors', but no alternative rules are tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format." pith.science (2026). https://pith.science/paper/O32PJZI4

@misc{pith2026250623133,
  author       = {Pith},
  title        = {Pith review of: Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O32PJZI4}},
  note         = {Machine review of arXiv:2506.23133}
}
read the original abstract

Generating and voting multiple answers is an effective method to mitigate reasoning inconsistencies of large language models (LLMs). Prior works have shown that multiple reasoning formats outperform a single format when generating multiple answers. However, previous works using multiple formats rely on formats labeled by humans, which could be unsuitable for all tasks and have high labeling costs. To address this issue, we adapt suitable formats to the given tasks by generating and selecting formats. We first propose how to measure the reasoning error when generating multiple answers. Then, we introduce Format-Adapter, which utilizes LLMs to generate and select suitable reasoning formats by minimizing the error measurement we present. We conduct experiments on math and commonsense reasoning tasks, where Format-Adapter achieves a 4.3% performance improvement on average over previous works, demonstrating the effectiveness.

Figures

Figures reproduced from arXiv: 2506.23133 by the authors.

Figure 1
Figure 1. The comparison between the previous work [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The comparison between using the single for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of FORMAT-ADAPTER, which consists of: (i) Format Generation: Generate possible reasoning formats of the given task; (ii) Answer Generation: Generate the answer using each reasoning format; (iii) Answer Scoring: Score whether each generated answer is correct using LLMs; (iv) Answer Selection: Select the final answer with Equation 5. Red and green represent the reasoning formats of incorrect and correct r… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The performance on MATH with different formats using Llama3.1-8b. Different blue • denotes the result using different formats, where the formats used are randomly sampled from that generated by FORMAT￾ADAPTER. The correlation coefficient is −0.652. in [PITH_FULL_IMAGE…
Figure 5
Figure 5. Figure 5: The average improvement brought by FORMAT-ADAPTER with different reasoning categories having more than four formats. ∆¯ EM denotes the aver￾age improvement compared to Self-Consistency. Over￾all denotes using all reasoning categories. ated by the corresponding formats …
Figure 7
Figure 7. Figure 7: The average ratio over all datasets of each [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The performance of repeat sampling using [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: An example sampled from MATH answered using different reasoning formats. The correct part is marked in red, and the incorrect part is marked in green. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages

  1. [1]

    So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V

    Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. https://arxiv.org/abs/2001.09977 Towards a human-like open-domain chatbot . Preprint, arXiv:2001.09977

    arXiv 2020

  2. [2]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. https://doi.org/10.1609/aaai.v38i16.29720 Graph of thoughts: Solving elaborate problems with large language models . Proceedings of the AAAI Conference on Artificial ...

    doi:10.1609/aaai.v38i16.29720 2024

  3. [3]

    Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. https://doi.org/10.18653/v1/2020.acl-main.382 Make up your mind! adversarial generation of inconsistent natural language explanations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157--4165, Onlin...

    work page doi:10.18653/v1/2020.acl-main.382 2020

  4. [4]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. https://openreview.net/forum?id=YfZ4ZPt8zd Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Transactions on Machine Learning Research

    work page 2023

  5. [5]

    Smith, and Tao Yu

    Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. https://openreview.net/forum?id=lH1PV42cbF Binding language models in symbolic languages . In The Eleventh International Conference on Learning Representations

    work page 2023

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

  8. [8]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. https://doi.org/10.1162/tacl_a_00410 Measuring and improving consistency in pretrained language models . Transactions of the Association for Computational Linguistics, 9:1012--1031

    doi:10.1162/tacl_a_00410 2021

Show all 34 references
  1. [9]

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. https://arxiv.org/abs/2411.10541 Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541

    2024 arXiv

  2. [10]

    Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. 2024. https://openreview.net/forum?id=ndR8Ytrzhh Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning . In The Twelfth International Conference on Learnin...

    2024

  3. [11]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  4. [12]

    Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Libo Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024. https://arxiv.org/abs/2402.10691 Python is not always the best choice: Embracing multilingual program of thoughts . Preprint, arXiv:2402.10691

    2024 arXiv

  5. [13]

    Paulson, and Markus Wenzel

    Tobias Nipkow, Lawrence C. Paulson, and Markus Wenzel. 2002. Isabelle/HOL --- A Proof Assistant for Higher-Order Logic, volume 2283 of LNCS. Springer

    2002

  6. [14]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  7. [15]

    Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/v1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...

    2015 doi

  8. [16]

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.163 Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  9. [17]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  10. [18]

    Omer Sagi and Lior Rokach. 2018. https://doi.org/10.1002/widm.1249 Ensemble learning: A survey . WIREs Data Mining and Knowledge Discovery, 8(4):e1249

    2018 doi

  11. [19]

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. https://openreview.net/forum?id=H1gR5iR5FX Analysing mathematical reasoning abilities of neural models . In International Conference on Learning Representations

    2019

  12. [20]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314

    2024 arXiv

  13. [21]

    Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. 2024. https://openreview.net/forum?id=z7usV2BlEE Making large language models better reasoners with alignment

    2024

  14. [22]

    Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. https://doi.org/10.18653/v1/2022.naacl-main.339 Measure and improve robustness in NLP models: A survey . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2022 doi

  15. [23]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferenc...

    2023

  16. [24]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  17. [25]

    Webb, Henry W

    Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luj\' a n, and Gavin Brown. 2024. A unified theory of diversity in ensemble learning. J. Mach. Learn. Res., 24(1)

    2024

  18. [26]

    Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. https://doi.org/10.18653/v1/D19-1260 Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Lang...

    2019 doi

  19. [27]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. https://openreview.net/forum?id=5Xc1ecxO1h Tree of thoughts: Deliberate problem solving with large language models . In Thirty-seventh Conference on Neural Informat...

    2023

  20. [28]

    Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Baoxin Wang, Dayong Wu, Qingfu Zhu, and Wanxiang Che. 2024 a . https://arxiv.org/abs/2408.08841 Flextaf: Enhancing table reasoning with flexible tabular formats . Preprint, arXiv:2408.08841

    2024 arXiv

  21. [29]

    Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.546 A uto CAP : Towards automatic cross-lingual alignment planning for zero-shot chain-of-thought . In Findings of the Association for Computational Linguis...

    2024 doi

  22. [30]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  23. [31]

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao,...

    2024

  24. [32]

    Jin Peng Zhou, Charles E Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. 2024. https://openreview.net/forum?id=V5tdi14ple Don't trust: Verify -- grounding LLM quantitative reasoning with autoformalization . In The Twelfth International Conference on Le...

    2024

  25. [33]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  26. [34]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.