Pith. sign in

REVIEW 3 major objections 7 minor 31 references

Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-AT automatically sends each question to the cheapest LLM tier that can answer it, escalating only when needed, and does so with no training.

desk verdict A training-free cascade with real cost savings, but the pseudo-labeling assumption is shakier than the paper admits and the abstract overstates accuracy. read the letter →

arxiv 2505.20921 v2 pith:BLNWZLZZ submitted 2025-05-27 cs.CL cs.AI

Injae Na , Keonwoong Noh , Woohwan Jung This is my paper
classification cs.CLcs.AI
keywords LLMtierselectioncost-accuracytrade-offtraining-freeroutingpseudo-labelingaccuracyestimationcascadeinferenceiterativerefinementself-verification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a training-free framework can automatically pick the right LLM tier for each question, so that easy questions are handled by cheap models and hard ones escalate to expensive models only when a judge decides the cheap answer is invalid. The proposed LLM-AT system uses a Starter that estimates each tier's accuracy from past inference records, a Generator that answers, and a Judge that validates; if the judge says the answer is invalid, the system upgrades to the next tier. On MATH and MCQA benchmarks the paper reports accuracy close to the top-tier model while cutting API cost by roughly 59% on MATH and 88% on MCQA, and trimming wall-clock time by 20-44% compared to single top-tier inference. If this holds, it gives deployment teams a training-free way to cut LLM serving bills without a dedicated routing model.

What carries the argument

The load-bearing mechanism is the Starter's accuracy estimator with pseudo-labeled History. For a new question $q$, it forms an embedding, retrieves the top-$k$ most similar questions from past inference records, and computes for each tier $j$ a weighted correct rate $P_j(q) = (n^T_j + \alpha_T) / (n^T_j + n^F_j + \alpha_T + \alpha_F)$, where $n^T_j$ and $n^F_j$ are similarity-weighted counts of correct and incorrect pseudo-labels and $\alpha$ is a benchmark-derived prior. The Starter then selects the lowest-cost tier with $P_j(q)$ above a threshold (0.7 in the main runs). The Generator answers with CoT or PoT prompting; the Judge, an LLM of the same tier (or one tier higher for the cheapest tier), outputs 'yes' or 'no'; if 'no', the system escalates to the next tier and repeats. Pseudo-labels are assigned by the judge's verdict plus a monotonicity assumption that any tier above a correct tier is also correct.

What would settle it

Count, on a held-out set of questions with ground-truth labels, how often a lower tier in the paper's four-tier model system succeeds while a higher tier fails, and then check whether LLM-AT's pseudo-labeled history assigns the higher tier a 'correct' label in exactly those cases; if the violation rate is non-negligible, the estimated accuracy curves diverge from the actual curves and the reported cost savings shrink accordingly.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a self-supervised cascade of Starter, Generator, and Judge can reproduce most of the accuracy of the strongest model while spending a fraction of its cost. The accuracy estimator is the piece that makes this possible: instead of always starting from the cheapest tier and climbing, the estimator uses similarity-weighted pseudo-labels from past queries to predict each tier's chance of success, and the Starter launches at the cheapest tier whose estimate clears a threshold. The framework then treats the judge's verdict as the correctness signal, and the paper's experiments show that with reliable judges the cascade resolves the large majority of questions on the first attempt (83.5% on MATH, 95.9% on MCQA), so the savings come mostly from avoiding unnecessary high-tier calls.

Load-bearing premise

The load-bearing premise is that correctness is monotone in tier: if a cheaper model answers a question correctly, every more expensive model is assumed to answer it correctly too, and an answer that matches a validated answer is assumed to be genuinely correct.

Editorial extensions

If this is right

  • LLM-AT can be dropped into an existing API pipeline without training data or fine-tuning; the only new components are the embedding-based retriever and the judge prompt.
  • The threshold parameter gives operators an explicit accuracy-cost dial: raise it to push more questions to higher tiers, lower it to save money.
  • The judge's F1 score directly bounds how trustworthy the pseudo-labels are, so the framework inherits the judge's failure modes.
  • When a new model tier is released, no retraining is needed--only the benchmark prior and the price table need updating.
  • Because most questions resolve on the first attempt (83.5% on MATH, 95.9% on MCQA), the main cost driver is the Starter's choice, not the escalation loop itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper measures cost on API list prices with no caching; a deployment with prompt caching would change the relative cost of the judge and repeated generator calls, possibly shrinking the advantage over single top-tier inference.
  • The similarity-based estimator inherits the embedding model's coverage: on rare question types with few neighbors in History, the pseudo-label counts collapse toward the benchmark prior, so the starter's behavior on the long tail is essentially the prior.
  • A stress test the paper hints at but does not run: replace the tier set with a compressed price spread (e.g., two models close in price and accuracy); as the spread closes, the optimal policy shifts toward always using the top tier, and the transmission mechanism's gains approach zero.
  • Transfers to open-ended generation would need a different validity signal than the judge's yes/no, because 'valid' for free-form text is not a well-defined equality check; the paper itself lists this as a limitation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces LLM-AT, a training-free framework for cost- and time-aware inference over LLM tiers. A Starter selects an initial tier using an accuracy estimator built from pseudo-labeled historical inference records; a Generator produces an answer with that tier; and a Judge validates the response, triggering one or more upgrades to higher tiers if the response is judged invalid. The authors evaluate the framework on MATH and a constructed MCQA benchmark, reporting large API-cost and execution-time reductions relative to single-inference and iterative baselines, with accuracy close to that of the top-tier o1. The paper also provides code and several auxiliary analyses (ablation, overhead decomposition, robustness to history size, performance reversals).

Significance. If the results hold, LLM-AT offers a practical, deployment-ready approach for cost-aware inference with existing proprietary LLM APIs, requiring no additional training and no gold-label annotations beyond what the system itself generates. The paper's strengths include the transparent cascade mechanism, the explicit attempt to estimate tier accuracy from history without supervised labels, and the breadth of empirical analyses (ablation, overhead, cold-start, and performance-reversal studies). The manuscript is nevertheless weakened by an unvalidated pseudo-labeling scheme that is contradicted in part by the paper's own reported numbers, a lack of sensitivity analysis for several key hyperparameters, and an internal numeric inconsistency between the main text and an appendix table.

major comments (3)
  1. [§3.4.1, Eq. (1)] The pseudo-labeling assumptions are load-bearing for the accuracy estimator but are not validated against ground truth, and the paper's own data contradict them. Assumption (i) equates judge-validity with correctness; Table 2 reports judge precision for GPT-4o on MATH as 0.748, meaning roughly a quarter of judge-valid responses are incorrect. Assumption (ii) states that if a lower tier is correct, every higher tier is also correct; Table 5 shows GPT-4o outperforming o1-mini on Number Theory (0.813 vs 0.781), a direct violation. Since Eq. (1) computes P_j(q) from n_T and n_F obtained from these pseudo-labels, the accuracy estimator inherits systematic bias, which in turn affects the Starter's threshold decision. The authors need to provide a direct validation of the pseudo-labels against gold labels (e.g., on a held-out portion of the data) or a sensitivity analysis showing that Starter decisions are robust to label noise at the levels implied by Table 2. Without this, the claim that the training-free accuracy estimator enables reliable initial tier selection is not supported.
  2. [§4.5, Fig. 3] The main trade-off curves are obtained for a single hyperparameter configuration (threshold 0.7, λ=5, top-k=5). The paper does not report how the results vary as these parameters are changed, even though the threshold directly controls the trade-off between starting too low (causing extra upgrades) and starting too high (incurring unnecessary cost). The claim that LLM-AT 'optimizes' the accuracy–cost and accuracy–time trade-offs is therefore only demonstrated for one point in the parameter space. A sensitivity analysis for at least the threshold and λ should be provided; without it, the practical guidance for deploying LLM-AT is incomplete.
  3. [Appendix B.2, Table 9 vs §5.1] There is a numeric inconsistency in the reported API cost for MCQA. Section 5.1 states that LLM-AT costs $7.14 on MCQA, while Table 9 in Appendix B.2 reports $16.89 for the same setting. The appendix value is identical to the MATH result, suggesting a copy-paste error, but the discrepancy is load-bearing for the main cost-saving claim. The authors must reconcile these numbers and ensure that all tables, figures, and the abstract are internally consistent.
minor comments (7)
  1. [Abstract] The abstract claims 'superior performance' for LLM-AT, but the MATH accuracy is slightly lower than that of single o1 (0.778 vs 0.793, as shown in Appendix B.2 Table 8). The wording should be tempered to 'comparable performance' or the specific comparison should be qualified.
  2. [§5.3] The statement 'Based on the reliable performance of the judge, we consider that the pseudo-labels of correctness can be trustworthy' is too strong given Table 2's precision of 0.748 for GPT-4o on MATH. A precision of 0.748 means 25% of positive judge labels are wrong; the authors should more carefully qualify the reliability and discuss the implications for pseudo-label trustworthiness.
  3. [§4.4, Fig. 3] No repeated runs, confidence intervals, or error bars are reported for the main results. Since LLM API responses can be stochastic (the paper does not state the sampling temperature or random seed), the reader cannot assess the stability of the reported cost and accuracy numbers. Please report the experimental protocol for stochasticity and include variance estimates where feasible.
  4. [Fig. 4] The validation of the accuracy estimator in Figure 4 is based on median alignment of estimated and actual accuracy distributions. A quantitative calibration measure (e.g., correlation or mean absolute error per question) would be more informative and would help assess whether the estimator is reliable at the individual-question level, which is what the Starter relies on.
  5. [§3.4.2, Eq. (1)] The hyperparameters αT and αF are described as 'prior distributions' but are used as additive smoothing constants. Clarify the interpretation and the rationale for setting λ=5 and using benchmark scores as priors; the current phrasing is confusing.
  6. [§5.4] In the 'Recent 30 history' setting, it is unclear whether the 30 most recent entries are global or per-tier, and whether the estimator uses the same top-k retrieval over this small set. A more detailed description of this setting would aid reproducibility.
  7. [Throughout] There are several typos and formatting issues (e.g., 'substask' in the Introduction, 'theLLM' in section headings). A careful proofread is recommended.

Circularity Check

1 steps flagged · score 4.0 of 10

Accuracy estimator's 'correctness' is defined as judge validity, so the estimator predicts the judge's own verdict rate; final system accuracy is still gold-evaluated.

  1. self definitional [Section 3.4.1–3.4.2, Eq. (1)]
    "If the judge evaluates the response as valid, the answer is labeled as correct. ... The estimated accuracy Pj(q) of tier j is computed by dividing the number of correct labels by the number of all labels as Pj(q) = (nT + αT)/(nT + nF + αT + αF)."

    By Eq. (1), P_j(q) is the rate of 'correct' labels in the history. Section 3.4.1 defines a 'correct' label as 'If the judge evaluates the response as valid, the answer is labeled as correct.' Therefore the accuracy estimator's output is, by construction, the historical judge-validity rate rather than gold accuracy. The starter then chooses the initial tier from this quantity, and the judge is the same module that accepts or rejects responses in the cascade, so the estimator 'predicts' the judge's own verdict rate.

full rationale

The paper's central cost-accuracy results are measured with gold labels (Section 4.4, Figure 3), and the judge's reliability is checked against gold labels in Table 2, so the main claims are not defined in terms of the estimator's output. The one genuine self-referential step is the accuracy estimator: its 'correctness' labels come from the judge, and it outputs the judge-validity rate while being described as expected accuracy. This is a definitional conflation rather than a full collapse of the system's evaluation. The only self-citation (Jung et al. 2019 for smoothing priors) is minor and not load-bearing.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four hand-set hyperparameters (threshold, lambda, top-k, benchmark prior) and on pseudo-labeling assumptions that combine judge validity with monotone tier correctness. No new physical or conceptual entities are introduced.

free parameters (4)
  • starter threshold = 0.7
    Chosen in Section 4.5 without sensitivity analysis.
  • prior weight lambda = 5
    Chosen in Section 4.5 without sensitivity analysis.
  • top-k neighbors = 5
    Chosen in Section 4.5 without sensitivity analysis.
  • benchmark prior Acc_Bench = MMLU Pro per-tier accuracy
    The choice of which benchmark to use for the prior is arbitrary; using task-specific benchmarks changes results (Appendix B.1).
assumptions (5)
  • domain assumption If a given LLM tier is correct, any higher tier model also generates correct answers.
    Section 3.4.1; violated by performance reversals shown in Section 5.5.
  • domain assumption If the judge evaluates the response as valid, the answer is labeled as correct.
    Section 3.4.1; judge has imperfect F1 (Table 2), so labels are noisy.
  • domain assumption If a lower-tier generates the same answer as the valid answer, that tier is labeled as correct; otherwise incorrect.
    Section 3.4.1; only valid for closed-form QA with exact answers.
  • domain assumption Top-k similar questions by cosine similarity of embeddings estimate per-question difficulty.
    Section 3.4.2; no validation that embedding similarity correlates with tier difficulty.
  • domain assumption Past inference records stored in History are representative of future questions.
    Section 3; cold-start analysis (Table 3) shows early data is less useful, so this is only asymptotically true.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models." pith.science (2026). https://pith.science/paper/BLNWZLZZ

@misc{pith2026250520921,
  author       = {Pith},
  title        = {Pith review of: Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLNWZLZZ}},
  note         = {Machine review of arXiv:2505.20921}
}
read the original abstract

LLM providers typically offer multiple LLM tiers, varying in performance and price. As NLP tasks become more complex and modularized, selecting the suitable LLM tier for each subtask is a key challenge to balance between cost and performance. To address the problem, we introduce LLM Automatic Transmission (LLM-AT) framework that automatically selects LLM tiers without training. LLM-AT consists of Starter, Generator, and Judge. The starter selects the initial LLM tier expected to solve the given question, the generator produces a response using the LLM of the selected tier, and the judge evaluates the validity of the response. If the response is invalid, LLM-AT iteratively upgrades to a higher-tier model, generates a new response, and re-evaluates until a valid response is obtained. Additionally, we propose accuracy estimator, which enables the suitable initial LLM tier selection without training. Given an input question, accuracy estimator estimates the expected accuracy of each LLM tier by computing the valid response rate across top-k similar queries from past inference records. Experiments demonstrate that LLM-AT achieves superior performance while reducing costs, making it a practical solution for real-world applications.

Figures

Figures reproduced from arXiv: 2505.20921 by the authors.

Figure 1
Figure 1. An example of the LLM-AT process with an input question. ‘S’, ‘G’, and ‘J’ indicate the Starter, Generator, and Judge, respectively. are solved by dividing them into multiple sub￾tasks, each requiring different levels of reasoning or generation capabilities. For instance, the Tree of Thoughts approach (Yao et al., 2024) often solves a single complex task with more than tens of LLM calls. This introduces a new challe… view at source ↗
Figure 2
Figure 2. An overview of LLM-AT Framework. sume an environment where past interactions can be stored and utilized. It is a realistic scenario widely adopted in actual LLM-based chatbots such as ChatGPT (OpenAI, 2022), Claude (Anthropic, 2023) and DeepSeek (DeepSeek, 2025) etc. 3.1 Overview [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Main results. The marker shapes represent [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distribution of the estimated accuracy. slightly lower performance than the single o1. For MCQA, while the single o1 performs the best, it costs 88.01% more than LLM-AT ($59.52 vs. $7.14). Some results of LLM-AT show a higher accuracy than the baselines with a lower co…
Figure 5
Figure 5. Figure 5: Tiers selected by LLM-AT in MATH based on question difficulty. level questions, demonstrating that LLM-AT opti￾mizes the trade-offs between accuracy and cost as well as accuracy and execution time. Limitations In this study, we focus on QA tasks with clearly defined an…
Figure 6
Figure 6. Figure 6: Results when using benchmark performance [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Ablation study. The left graph shows the ac [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Tiers selected by LLM-AT in MCQA based on question difficulty. Level 1 Level 2 Level 3 Level 4 Level 5 0 10 20 30 40 50 60 70 80 Accuracy 46.67 36.36 37.50 27.27 0.00 72.94 62.02 48.61 44.07 48.98 Abstain Accuracy Non-Abstain Accuracy Abstain Ratio 0 5 10 15 20 25 Abst…
Figure 9
Figure 9. Figure 9: Performance of questions from abstained vs. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 10 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al. 2024. Automix: Automatically mixing language models. Advances in Neural Information Processing Systems, 37:131000--131034

    work page 2024

  4. [4]

    Anthropic. 2023. https://www.anthropic.com/claude Meet claude . https://www.anthropic.com/claude

    work page 2023

  5. [5]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024 a . https://openreview.net/forum?id=cSimKw5p6R Frugal GPT : How to use large language models while reducing cost and improving performance . Transactions on Machine Learning Research

    work page 2024

  6. [6]

    Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024 b . https://openreview.net/forum?id=7RQvjayHrM Router DC : Query-based router by dual contrastive learning for assembling large language models . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  7. [7]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research

    2023

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

Show all 31 references
  1. [9]

    DeepSeek. 2025. https://www.deepseek.com/ Deepseek into the unknown . https://www.deepseek.com/

    2025

  2. [10]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor R \"u hle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. https://openreview.net/forum?id=02f3mUtqnM Hybrid LLM : Cost-efficient and quality-aware query routing . In The Twelfth Internatio...

    2024

  3. [11]

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024. https://aclanthology.org/2024.acl-long.786 Don ' t hallucinate, abstain: Identifying LLM knowledge gaps via multi- LLM collaboration . In Proceedings of the 62nd Annual Meeting...

    2024

  4. [12]

    Dayuan Fu, Biqing Qi, Yihuai Gao, Che Jiang, Guanting Dong, and Bowen Zhou. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.38 MSI -agent: Incorporating multi-scale insight into embodied agents for superior planning and decision-making . In Proceedings of the 2024 Conference...

    2024 doi

  5. [13]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. https://openreview.net/forum?id=Sx038qxjek CRITIC : Large language models can self-correct with tool-interactive critiquing . In The Twelfth International Conference on Learning Rep...

    2024

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021 a . Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  7. [15]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  8. [16]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  9. [17]

    Woohwan Jung, Younghoon Kim, and Kyuseok Shim. 2019. Crowdsourced truth discovery in the presence of hierarchies for knowledge fusion. Advances in Database Technology-EDBT 2019, pages 205--216

    2019

  10. [18]

    Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.7 LLM -based agent society investigation: Collaboration and confrontation in avalon gameplay . In Proceedings of the 20...

    2024 doi

  11. [19]

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. https://doi.org/10.18653/v1/2024.naacl-long.109 Routing to the expert: Efficient reward-guided ensemble of large language models . In Proceedings of the 2024 Conference of the North...

    2024 doi

  12. [20]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://openrev...

    2023

  13. [21]

    Kaushal Kumar Maurya, KV Srivatsa, and Ekaterina Kochmar. 2024. Selectllm: Query-aware efficient selection algorithm for large language models. arXiv preprint arXiv:2408.08545

    2024 arXiv

  14. [22]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  15. [23]

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665

    2024 arXiv

  16. [24]

    OpenAI. 2022. https://openai.com/chatgpt/overview/ Chatgpt. get answers. find inspiration. be more productive

    2022

  17. [25]

    OpenAI. 2025. https://platform.openai.com/docs/guides/reasoning/ Reasoning models

    2025

  18. [26]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  19. [27]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    2024 arXiv

  20. [28]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  21. [29]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  22. [30]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  23. [31]

    Chang Zong, Yuchen Yan, Weiming Lu, Jian Shao, Yongfeng Huang, Heng Chang, and Yueting Zhuang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.101 Triad: A framework leveraging a multi-role LLM -based agent to solve knowledge base question answering . In Proceedings of the 2...

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.