Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering

Matt Thomson; Michael J. Zellinger

REVIEW 4 major objections 5 minor 35 references

Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counting a reasoning model's thinking tokens gives a black-box uncertainty signal: deferring the longest traces to a human cuts Qwen3's error on difficult MATH from about 3% to below 1%, and a fast front model saves about 40% latency.

desk verdict Practical systems paper with a real latency-drag finding, but its headline error reductions assume a perfect human oracle and in-sample thresholds. read the letter →

arxiv 2507.14406 v1 pith:RSMNFXCL submitted 2025-07-18 cs.AI cs.LGcs.SYeess.SY

Michael J. Zellinger , Matt Thomson This is my paper

classification cs.AIcs.LGcs.SYeess.SY

keywords reasoningLLMsselectivepredictionhuman-in-the-loopsystemsuncertaintyestimationtracelengthlatencydragmodelcascadesMATHbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that two practical failings of today's reasoning models—occasional wrong answers and long response times—can be managed from outside the model, using only the text the model produces. The first finding is that the number of tokens spent thinking is a usable uncertainty signal: deferring a reasoning model's longest-traced queries to a human expert cuts Qwen3 235B-A22B's error on difficult MATH problems from about 3% to below 1% while deferring only 7.5% of queries. The second is a cascade design, "Fail Fast, or Ask," in which a fast non-reasoning model answers easy questions and routes the rest, yielding roughly 40% lower latency and about 50% lower cost for DeepSeek R1 while keeping the accuracy-rejection area above 90%. If correct, these results mean a systems engineer can reduce reasoning-model error and latency or cost using only API-visible signals.

What carries the argument

The central mechanism is trace-length thresholding: count the output tokens a reasoning model emits, choose a cutoff at a quantile of the empirical token distribution, and defer every query above that cutoff to a human expert. The paper couples this with the P(True) strategy for the faster non-reasoning model, in which the model is asked whether its own proposed answer is correct and the resulting probability drives the routing decision $\pi_{\mathrm{nr}}$: fail fast to the human when the probability is low, pass to the reasoning model in the middle, and respond directly when the probability is high. A utilization rate $u$ fixes the fraction of queries that never reach the reasoning model. The paper also identifies "latency drag": because long reasoning traces mark difficult queries, the queries a fast front model passes onward are exactly the slow ones, so the realized latency savings fall below the naive weighted average in the paper's Equation (2).

What would settle it

Give the 7.5% longest-trace queries from a fresh set of MATH 5/5 questions to a real human expert and measure end-to-end accuracy; if the expert's error rate on those deferred questions is not near zero, the reported drop from about 3% to below 1% is not attainable.

Watch

Extended reading notes

Core claim

The paper's central claim is that the length of a reasoning model's chain of thought is a cheap, black-box signal of when it will be wrong. On 500 maximum-difficulty MATH problems, rejecting queries whose output-token count exceeds a threshold lowers Qwen3 235B-A22B's conditional error from about 2.8% to below 1% with only a 7.5% rejection rate, and raises DeepSeek R1's accuracy from about 94% to 97%. The "Fail Fast, or Ask" cascade adds a fast non-reasoning model in front: at 60% utilization it keeps the area under the accuracy-rejection curve above 0.90 while cutting latency by about 40% and cost by about 46% for DeepSeek R1. The paper further claims these gains come from black-box systems engineering alone, and reports that OpenAI o3 does not exhibit the trace-length error signal.

Load-bearing premise

The human expert is assumed to answer every deferred query instantly and perfectly, so the reported error reductions hold only if real human answers are always correct.

Editorial extensions

If this is right

Deferral thresholds can be set from the empirical distribution of output tokens alone, with no retraining and no access to model internals.
At 60% utilization of the fast front model, DeepSeek R1 keeps 90+% area under the accuracy-rejection curve while cutting latency by roughly 40% and cost by about 46%.
Because conditional accuracy, not raw accuracy, is the right metric, deployment decisions should be made from accuracy-rejection curves over the operating rejection rate.
OpenAI o3, which shows no trace-length error signal, would need a different uncertainty estimate to participate in this deferral scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: Trace length is most likely a proxy for problem difficulty, so other difficulty-correlated signals—such as token-level entropy or backtracking counts—could substitute for it and extend the method to models with different thinking styles.
Inference: If human experts share the model's sense of difficulty, the omniscient-oracle assumption is the main threat: the queries a model gets wrong may be precisely the ones that also trip up human solvers, making the reported error reductions upper bounds.
Inference: Latency drag suggests a learned or adaptive router that conditions on predicted reasoning latency as well as confidence could recover some of the lost latency savings without sacrificing accuracy.
Inference: The black-box nature of the approach makes it directly deployable as an online control policy: a system can log trace lengths and verified outcomes and continuously re-estimate the deferral threshold.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, worth a look if you care about deploying reasoning LLMs in risk-sensitive settings. The genuinely new pieces are the two-stage Ask/Fail-Fast-or-Ask configuration and the latency-drag measurement—the observation that fronting a reasoning model with a faster model removes easy queries and shifts the reasoning model's latency distribution upward. That effect is real, quantified on MATH, and it explains why naive cascade latency models overestimate savings. The paper also deserves credit for honesty: Section 3 states upfront that the human is an instant, perfect oracle, and Section 6 explicitly discusses accuracy drag and the need to screen experts on difficult queries. The trace-length uncertainty signal is not new—Ballon et al. and Shojaee et al. already established the correlation—but the paper applies it cleanly in a deferral system and documents that it fails for o3, which is a useful negative result.

The soft spots are real but not fatal to the core idea. First, the accuracy-rejection curves are computed on the same 500 MATH questions used to set the token thresholds. No held-out split, no confidence intervals; at n=500, moving from 3% to 0.5% error is a handful of errors. The paper needs a calibration/test split or at least bootstrap intervals. Second, the headline error reductions assume an omniscient human. The stress-test math holds up: at 7.5% deferral with model error near 0.5%, hitting system error below 1% requires human error below roughly 7%; for DeepSeek at 20% deferral, human error below 3%. Those are strong assumptions, and the reported numbers are upper bounds on achievable system accuracy. The paper acknowledges this, but the abstract presents the oracle-conditional numbers as the result. Third, P(True) validity is imported from the authors' own prior work; that is acceptable and cited, but it is a dependency worth flagging.

Who this is for: practitioners building human-in-the-loop pipelines for reasoning LLMs, and researchers working on selective prediction for LLMs. It deserves a serious referee, but the revision should include separate calibration and evaluation splits and a human-error sensitivity analysis, or else explicitly reframe the claims as oracle-conditional. The latency-drag analysis alone is a publishable contribution.

Referee Report

4 major / 5 minor

Summary. The paper proposes two human-in-the-loop systems for reasoning LLMs: "Ask," in which a reasoning model defers queries with long reasoning traces to a human expert, and "Fail Fast, or Ask," in which a fast non-reasoning model fronts the reasoning model and may respond, pass the query to the reasoning model, or defer directly to the human. Experiments on 500 difficult MATH problems (difficulty 5/5, numeric answers) with Qwen3 235B-A22B, DeepSeek R1, and OpenAI o3 as the reasoning model and Llama3.1 405B as the non-reasoning model report that trace-length-based deferral lowers Qwen3 error from about 2.8% to 0.5% at 7.5% deferral and DeepSeek R1 error from 5.8% to 3% at 20% deferral, and that fronting with the non-reasoning model yields roughly 40% latency and 50% cost savings at 60% utilization while keeping AUARC above 90%. The paper also identifies and analyzes 'latency drag,' the increase in the reasoning model's conditional latency when only difficult queries are passed to it.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it identifies a black-box signal (reasoning-trace length) that can drive selective prediction for reasoning LLMs, and it documents a non-obvious systems effect (latency drag) that matters for any cascade that routes queries by difficulty. The systems are described without requiring model-internal access, which is appropriate for API-based deployment. The paper is also commendably explicit about its simplifying assumption that the human expert is an omniscient, instantaneous oracle, and it discusses the 'accuracy drag' risk in Section 6. The main caveat is that the headline error-rate and AUARC numbers are computed in-sample on the same 500 queries used to set the deferral thresholds, and the error reductions are upper bounds under the oracle assumption; the significance of the results therefore depends on the extent to which these limitations are addressed in revision.

major comments (4)

[Section 3 and Section 5.2] The deferral thresholds are quantiles of the empirical output-token distribution of the same 500 queries on which accuracy is then measured. Section 3 states that to reject 10% of queries the threshold is set at the 90% quantile of its empirical distribution, and Section 5.2 then reports conditional accuracy as a function of thresholds on that same set. This in-sample protocol can overstate the accuracy-rejection curve and AUARC; the paper should either report results from a held-out split or use cross-validation to estimate thresholds, and should state explicitly whether any reported number uses a threshold chosen outside the evaluation set.
[Section 3 (Human Expert) and Conclusion] The reported error reductions (Qwen3 from 2.8% to 0.5%, DeepSeek R1 from 5.8% to 3%) treat deferral as a correct rejection because the human expert is assumed to answer 'instantly and perfectly.' Under this assumption the numbers are conditional upper bounds, not achievable system error rates. The paper acknowledges this in Section 6 and proposes expert screening, but the abstract and conclusion state the oracle-based numbers without the qualification. The authors should add a sensitivity analysis or an explicit bound showing how the error-reduction claims degrade under a human error rate e_h on deferred queries, e.g., system error = (1-r)e_m + r e_h, and should rephrase the headline claims as oracle-conditional.
[Section 5, n=500 and Figure 4] The error-rate differences driving the central claim are small in absolute error counts: on n=500, a change from 2.8% to 0.5% corresponds to roughly 14 errors to 2-3 errors, and the DeepSeek change from 5.8% to 3% corresponds to roughly 29 to 15 errors. No confidence intervals, standard errors, or significance tests are reported for the accuracy-rejection curves or for the AUARC differences in Table 2. The paper should report binomial or bootstrap confidence intervals for the key error-rate and AUARC comparisons, since at this sample size several of the reported differences may be consistent with sampling noise.
[Section 3 and Section 5.3] The configuration of the 'Fail Fast, or Ask' system depends on thresholds c_fail_fast and c_pass for the non-reasoning model's P(True) confidence, and on the utilization rate u, but the paper does not specify how these thresholds are chosen or validated. If they are selected on the same 500-query evaluation set, the latency, cost, and AUARC figures in Table 2 inherit the same in-sample optimism as the reasoning-model thresholds. Please state the threshold-selection procedure and whether the P(True) calibration from Zellinger and Thomson (2025) is reused as-is or re-fit on this dataset, and provide an out-of-sample evaluation if re-fitting occurs.

minor comments (5)

[Abstract and Section 5.3/Table 2] The abstract claims 'around 40% latency reduction and about 50% cost savings,' but Table 2 reports 36.5-37.8% latency reduction and 45.8% cost savings for DeepSeek R1 at 60% utilization; please reconcile these numbers and state which configuration is being cited.
[Section 1] In the description of the Fail Fast, or Ask system, the sentence 'This model may respond directly to the query, defer it to Mnr, or send it to the human expert' appears to say that the non-reasoning model defers to itself; it should presumably read 'defer it to Mr.'
[Section 4 and Figure 2] The model name is inconsistently written as 'Qwen3 225B-A22' in the Figure 2 caption and as 'Qwen3 235B-A22B' elsewhere; please unify the spelling to match the reported model.
[Section 5.2 and Figure 3] The local linear regression curves in Figure 3 are presented without specifying the bandwidth or number of bootstraps used for the ±1σ bands; adding this information would make the figure reproducible.
[Section 5.3 and Figure 5] The text says the error curves 'decrease exponentially with increasing rejection rate,' but the figure uses a log-scale y-axis; please clarify whether the decay is exponential in the raw error rate or merely appears linear on the log scale, or rephrase to avoid overstating the functional form.

Circularity Check

2 steps flagged · score 4.0 of 10

Accuracy-rejection curves are in-sample quantile summaries and the P(True) premise rests on the same authors' prior work on the same 500-question set; the core token-length correlation still has independent empirical content.

fitted input called prediction [Section 3, 'Predicting Reasoning Model Errors'; Section 5.2, Eq. (3) and Figure 4]
"For example, to reject 10% of queries (and defer them to the human expert), we set the output token threshold at the 90% quantile of its empirical distribution. ... Accuracy = E[Correct | # Output Tokens ≤ T] (3) for different thresholds T of the number of output tokens. We consider thresholds yielding rejection rates from 0% to 20% (x axis)."

The threshold T is defined as a quantile of the same # Output Tokens distribution used to compute E[Correct | # Output Tokens ≤ T]. Hence the reported error reduction (e.g., 3% to <1% at 7.5% deferral) is a within-sample conditional summary of the same 500 queries: choosing T by quantile fixes the rejection rate and re-sorts the same examples, so no held-out prediction is made. The empirical content is not zero — o3's flat curve shows the correlation is not definitional — but the headline 'prediction' double-uses one dataset for threshold setting and evaluation.
self citation load bearing [Section 3, 'Predicting Non-Reasoning Model Errors'; Section 5, data paragraph]
"To predict the errors of Mnr, we estimate its confidence using the P(True) strategy of Kadavath et al. (2022). Prior work shows that this methodology yields good performance for Llama3.1 405B (Zellinger and Thomson, 2025). ... We use the same data as Zellinger and Thomson (2025), which is filtered for questions with numeric answers."

The only cited basis for choosing P(True) on Llama3.1 405B is a paper by the present authors, and the present experiments reuse that paper's exact 500-question MATH set. The system's accuracy therefore imports a design premise (P(True) calibrates this model on this data) from a self-citation that shares both model and dataset, rather than from an independent or newly validated source. This is load-bearing for the Fail Fast, or Ask curves, although the latency and cost trade-offs themselves are measured in this paper.

full rationale

The central claims are not constructed to equal their inputs: the token-length/error correlation is an empirical finding (indeed it fails for o3), the latency-drag effect is measured rather than assumed, and the cost/latency savings come from API traces. However, two aspects raise the circularity score. First, the selective-precision curves are in-sample quantile summaries: the threshold is the quantile of the empirical output-token distribution, and the accuracy curve is the conditional accuracy on that same distribution, so the abstract's error-rate reductions are descriptive summaries rather than out-of-sample predictions. Second, the non-reasoning model's P(True) confidence premise is justified by a self-citation to the same authors' prior work, and that prior work underlies the same 500-question MATH set reused here. These are not fatal — the token-length result has independent empirical content and the system measurements are real — but they make the paper's framing somewhat more self-supporting than a fully out-of-sample evaluation would be. The perfect-human-oracle assumption is a correctness limitation, not a circularity, and is not scored here.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on three uncharged premises: perfect human answers, validity of the P(True) signal from same-author prior work, and stability of the empirical trace-length distribution across deployment. The first is acknowledged, the second is imported, the third is untested because thresholds and evaluation share the same 500 questions.

free parameters (3)

output-token rejection threshold T = not reported (set to empirical quantile, e.g., 90th percentile)
Used to decide which queries are deferred to the human. Computed from the empirical distribution of reasoning-trace lengths on the same 500 questions on which accuracy is then measured.
P(True) confidence thresholds c_fail_fast and c_pass for Llama3.1 405B = not reported (calibrated to target utilization and rejection rates)
Determine whether the non-reasoning model responds, passes to the reasoning model, or fails fast. Chosen to realize the reported utilization and rejection rates on the evaluation set.
non-reasoning model utilization rate u = 0.50, 0.60, 0.75
System design choice controlling how often the fast model acts without the reasoning model. Swept to show trade-offs, not fitted to maximize accuracy.

assumptions (3)

domain assumption The human expert is an omniscient, instantaneous oracle.
Section 3 'Human Expert': 'We assume that the human expert can instantly and perfectly answer each query.' Error reductions are conditional on this being true.
domain assumption P(True) confidence estimates of Llama3.1 405B reliably rank difficulty on MATH.
Used to route queries in the Fail Fast, or Ask system. Validity is imported from the same authors' prior work (Zellinger and Thomson 2025) rather than re-validated with independent evidence.
domain assumption The empirical distribution of reasoning-trace lengths on these 500 questions is representative of deployment conditions.
Thresholds are empirical quantiles of the evaluation set itself; the paper does not test whether the threshold transfers to new questions or shifted distributions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering." pith.science (2026). https://pith.science/paper/RSMNFXCL

@misc{pith2026250714406,
  author       = {Pith},
  title        = {Pith review of: Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSMNFXCL}},
  note         = {Machine review of arXiv:2507.14406}
}

read the original abstract

State-of-the-art reasoning LLMs are powerful problem solvers, but they still occasionally make mistakes. However, adopting AI models in risk-sensitive domains often requires error rates near 0%. To address this gap, we propose collaboration between a reasoning model and a human expert who resolves queries the model cannot confidently answer. We find that quantifying the uncertainty of a reasoning model through the length of its reasoning trace yields an effective basis for deferral to a human, e.g., cutting the error rate of Qwen3 235B-A22B on difficult MATH problems from 3% to less than 1% when deferring 7.5% of queries. However, the high latency of reasoning models still makes them challenging to deploy on use cases with high query volume. To address this challenge, we explore fronting a reasoning model with a large non-reasoning model. We call this modified human-in-the-loop system "Fail Fast, or Ask", since the non-reasoning model may defer difficult queries to the human expert directly ("failing fast"), without incurring the reasoning model's higher latency. We show that this approach yields around 40% latency reduction and about 50% cost savings for DeepSeek R1 while maintaining 90+% area under the accuracy-rejection curve. However, we observe that latency savings are lower than expected because of "latency drag", the phenomenon that processing easier queries with a non-reasoning model pushes the reasoning model's latency distribution towards longer latencies. Broadly, our results suggest that the deficiencies of state-of-the-art reasoning models -- nontrivial error rates and high latency -- can be substantially mitigated through black-box systems engineering, without requiring access to LLM internals.

Figures

Figures reproduced from arXiv: 2507.14406 by the authors.

**Figure 2.** Fronting the reasoning model Mr with a non-reasoning model Mnr increases the reasoning model’s average latency (white area), leading to lower latency savings than expected. Fundamentally, this latency drag results from the negative correlation between the reasoning model’s latency (y axis) and the non-reasoning model’s confidence (x axis) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Local linear regression (±1σ) of the reasoning models’ correctness vs the number of output tokens shows that correctness decreases when reasoning traces are long. 5.2 How low does selective prediction push the error rates of reasoning models? Since the length of reasoning models’ thinking traces correlates with decreased accuracy, queries leading to long reasoning traces carry a high risk of error. How much can we r… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Abstaining from answering difficult queries based on the length of the reasoning [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Different degrees of non-reasoning model utilization (% values annotated in color) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 30 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

B., and Kulesza, T

Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. (2014). Power to the people: The role of humans in interactive machine learning. AI Magazine , 35(4):105--120

work page 2014
[3]

and Mitchell, T

Azaria, A. and Mitchell, T. (2023). The internal state of an llm knows when it's lying

work page 2023
[4]

Ballon, M., Algaba, A., and Ginis, V. (2025). The relationship between reasoning and performance in large language models -- o3 (mini) thinks harder, not longer

work page 2025
[5]

Chen, L., Zaharia, M., and Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance

work page 2023
[6]

DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

work page 2025
[7]

Z., Creswell, J., Shetty, S., Sieniek, M., Spitz, T., Corrado, G., Kohli, P., Cemgil, T., and Karthikesalingam, A

Dvijotham, K., Winkens, J., Barsbey, M., Ghaisas, S., Stanforth, R., Pawlowski, N., Strachan, P., Ahmed, Z., Azizi, S., Bachrach, Y., Culp, L., Daswani, M., Freyberg, J., Kelly, C., Kiraly, A., Kohlberger, T., McKinney, S., Mustafa, B., Natarajan, V., Geras, K., Witowski, J., Qin, Z. Z., Creswell, J., Shetty, S., Sieniek, M., Spitz, T., Corrado, G., Kohli...

work page 2023
[8]

and Wiener, Y

El-Yaniv, R. and Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11:1605--1641

work page 2010

Show all 35 references

[9]

and van der Schaar, M

Fanconi, C. and van der Schaar, M. (2025). Towards a cascaded llm framework for cost-effective human-ai decision-making

2025
[10]

Farquhar, S., Kossen, J., Kuhn, L., et al. (2024). Detecting hallucinations in large language models using semantic entropy. Nature , 630:625--630

2024
[11]

and El-Yaniv, R

Geifman, Y. and El-Yaniv, R. (2017). Selective classification for deep neural networks

2017
[12]

The L lama 3 H erd of M odels

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

2024
[13]

Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. NeurIPS

2021
[14]

Jin, M., Yu, Q., Shu, D., Zhao, H., Hua, W., Meng, Y., Zhang, Y., and Du, M. (2024). The impact of reasoning step length on large language models. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024 , pages ...

2024
[15]

Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobs...

2022
[16]

Kamar, E. (2016). Directions in hybrid intelligence: Complementing AI systems with human intelligence. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16) , pages 4070--4073, New York, NY, USA. IJCAI Organization

2016
[17]

and Longbotham, R

Kohavi, R. and Longbotham, R. (2007). Online experiments: Lessons learned. Computer , 40(9):103--105

2007
[18]

Lawsen, A. (2025). Comment on the illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity

2025
[19]

Madras, D., Pitassi, T., and Zemel, R. (2018). Predict responsibly: Improving fairness and accuracy by learning to defer. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , vo...

2018
[20]

Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

2023
[21]

L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., and Hashimoto, T

Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., and Hashimoto, T. (2025). s1: Simple test-time scaling

2025
[22]

Openai o1 system card

OpenAI (2024). Openai o1 system card

2024
[23]

Pearl, J. (2009). Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2 edition

2009
[24]

Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Lakshminarayanan, B., and Liu, P. J. (2023). Out-of-distribution detection and selective generation for conditional language models

2023
[25]

Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy

2020
[26]

Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., and Farajtabar, M. (2025). The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity

2025
[27]

Strong, J., Men, Q., and Noble, A. (2025). Trustworthy and practical ai for healthcare: A guided deferral system with large language models

2025
[28]

Varshney, N., Mishra, S., and Baral, C. (2022). Investigating selective prediction approaches across several tasks in IID , OOD , and adversarial settings. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Findings of the Association for Computational Linguistics: ACL...

2022
[29]

A., Moss, E., Raffa, G., and Nachman, L

Watkins, E. A., Moss, E., Raffa, G., and Nachman, L. (2025). What's so human about human-ai collaboration, anyway? generative ai and human-computer interaction

2025
[30]

Xin, J., Tang, R., Yu, Y., and Lin, J. (2021). The art of abstention: Selective prediction and error regularization for natural language processing. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computatio...

2021
[31]

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., ...

2025
[32]

Zellinger, M. J. and Thomson, M. (2025). Economic evaluation of llms

2025
[33]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[34]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[35]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

B., and Kulesza, T

Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. (2014). Power to the people: The role of humans in interactive machine learning. AI Magazine , 35(4):105--120

work page 2014

[3] [3]

and Mitchell, T

Azaria, A. and Mitchell, T. (2023). The internal state of an llm knows when it's lying

work page 2023

[4] [4]

Ballon, M., Algaba, A., and Ginis, V. (2025). The relationship between reasoning and performance in large language models -- o3 (mini) thinks harder, not longer

work page 2025

[5] [5]

Chen, L., Zaharia, M., and Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance

work page 2023

[6] [6]

DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

work page 2025

[7] [7]

Z., Creswell, J., Shetty, S., Sieniek, M., Spitz, T., Corrado, G., Kohli, P., Cemgil, T., and Karthikesalingam, A

Dvijotham, K., Winkens, J., Barsbey, M., Ghaisas, S., Stanforth, R., Pawlowski, N., Strachan, P., Ahmed, Z., Azizi, S., Bachrach, Y., Culp, L., Daswani, M., Freyberg, J., Kelly, C., Kiraly, A., Kohlberger, T., McKinney, S., Mustafa, B., Natarajan, V., Geras, K., Witowski, J., Qin, Z. Z., Creswell, J., Shetty, S., Sieniek, M., Spitz, T., Corrado, G., Kohli...

work page 2023

[8] [8]

and Wiener, Y

El-Yaniv, R. and Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11:1605--1641

work page 2010