Artificial Intelligence · arXiv:2610.18579 v1 · submitted 13 Oct 2026 · 77 references
Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops
The paper says
“System One models choose an agent's next action by scoring a supplied set of options in a single pass.”
more of the abstract
This makes them fast and cheap, but when they choose wrongly there is nothing to inspect: did the model misread what the agent has seen, or read it correctly and still act wrongly? We answer behaviorally. On a frozen benchmark of agent decision states from simulated environments, customer-service policies, tool use, web navigation and coding, we pair every next-action question with factual probes about the same state, each answered independently. Across seventeen fast systems and a reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form. These errors are not noise; they gather on the same states across systems of every kind. Where the probed facts provably determine the action, models that read every fact still fail to combine them, and their own answers, composed by code, repair the failure. On real states, what is missing is often not a fact but the next sub-goal.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Careful, inventive study of where System One agents fail — the natural-state headline overreaches, but the composition-ladder results and the paired-measurement apparatus are solid and deserve review. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 77 references say.
The paper says, at §5.1, §2.4, §10, Table 2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.90):
“In finite environ- ments it is exhaustive; in natural data, absence of a counterexample does not prove sufficiency.”
Pith says: Objection 1 of 3 · major
The paper’s central claim, “most wrong actions follow correct reading” (abstract; §1, first bullet), is carried by the U1D0 shares of Table 2 and Fig. 3b, which are computed on the natural paired states; by the paper’s own criteria those shares do not license the diagnosis.
more of objection 1
The authors acknowledge in §2.4 that in natural data “absence of a counterexample does not prove sufficiency” and in §10 that “on natural states the probes are not sufficient for the decision, so a facts-right, action-wrong outcome mixes decision failures with facts we did not ask about.” A U1D0 outcome on such states means every measured fact was answered correctly, not that the decision-relevant facts were read correctly; an unprobed decision-relevant fact being missed produces the identical observable outcome. The §6.1 and §6.3 studies establish composition and sub-goal failures only on their covered domains and do not rescue the cross-domain reading of Table 2. Because the size of this share is the central contribution, the abstract and the first two bullets of §1 should either carry the qualifier “measured” at every occurrence or, preferably, the paper should also run the Eq. (1) check on the 13,678 ALFWorld/BabyAI-Text/TextWorld paired states, where the authors define the check as exhaustive, and report the headline shares recomputed on the states that pass.
The paper says, at §5.1, Fig. 3b
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The claim that “the more accurate a system, the more of its errors take this form” is partly a base-rate artifact. Writing the share as P(U=1|D=0) = P(D=0|U=1)·P(U=1) / (P(D=0|U=1)·P(U=1) + P(D=0|U=0)·P(U=0)), the share increases with U even when the conditional error rates are held fixed: with P(D=0|U=1)=0.10 and P(D=0|U=0)=0.30, the share grows from about 33% at U=0.60 to 84% at U=0.94.
more of objection 2
Across Table 2, U rises from 0.600 (Kev-9B) to 0.943 (GPT-6.1), mechanically pushing error mass into the U=1 cell, and Kev-9B (U=0.600, share 49.2%) is a visible non-monotonicity in Fig. 3b. The causal sentence “Stronger systems mainly stop misreading; they do not stop choosing wrongly after reading correctly” therefore needs a control, for example a null that preserves each system’s conditional error rates while matching its U, or a comparison restricted to states with U=1 for all compared systems. The conditional statistic P(D=1|U=1) (10–21% residual error for the strongest systems) is the cleaner evidence and should carry the interpretive weight.
The paper says, at §6.3, §F
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
In §F, the next sub-goal is defined as “the set of categories of the gold actions,” so the sub-goal block is a coarsening of the gold label itself, whereas the fact blocks are properties of the state. The result that the sub-goal block adds 16.3 points for JEV while decision-oriented facts add at most 5.3 therefore measures, in part, how …
more of objection 3
… well the model exploits a near-answer hint, and the abstract’s conclusion that “what is missing is often not a fact but the next sub-goal” inherits that confound. The own-answer condition (JEV’s own sub-goal is correct in only 66.8% of states and adds 6.4 points) supports the planning-failure reading, and the authors should be credited for running it; nevertheless, the interpretation should state explicitly that the sub-goal block is informationally closer to the label than any fact block is, and ideally include a control block that is equally label-like without naming the action category, such as the identity of the next object or receptacle to act on.
4 minor objections
The paper says, at §1, §5.1
Pith says: minor objection 1 of 4
The abstract and the first bullet of §1 say “seventeen fast systems,” but the number of systems contributing to each analysis varies under the coverage rule: Fig. 3b omits three systems with accuracy at or below 0.5, §5.2 pools 14 competent systems, and §6.1 covers 16; a per-figure statement of the included systems would make the subset structure transparent.
The paper says, at Fig. 4
Pith says: minor objection 2 of 4
In Fig. 4, several near-zero cells for Laya, Quyet-1.0-Large and CLM-8B reflect refusals and input limits counted as wrong rather than decisions; the caption should state this, as Table 2’s coverage column does.
show 2 more
The paper says, at §5.1, §5.2
Pith says: minor objection 3 of 4
The exact-lookahead result of §5.2 — that 5–42% of measured simulator errors are also optimal, and for GPT-6.1 up to 96% of its TextWorld errors after correct reading are not errors — materially qualifies the §5.1 shares and should be cross-referenced where the 73–85% figure is first stated.
The paper says, at §2.4
Pith says: minor objection 4 of 4
The feasibility of the Eq. (1) check on natural subsets depends on how many states share a probe vector with goal, rules and remaining context held fixed; reporting the group-size distribution of the probe-vector partition would make the sufficiency discussion in §2.4 and §10 concrete.
What would settle it
Run the §2.4 criterion exhaustively on the simulator subset of the natural paired benchmark (ALFWorld 9,097, BabyAI-Text 2,181, TextWorld 2,400 states): for each U1D0 state, enumerate all finite simulator states with the same goal, rules, and benchmark-probe vector, and test whether the gold-set intersection is nonempty. Recompute the §5.1 shares after removing or separately reporting U1D0 states whose probe group fails the test. If the 73–85% shares survive on the remaining states, the 'correct reading' reading of Table 2 is supported on the simulator majority; if they fall materially, the abstract and §5.1 must be re-scoped to 'all measured facts correct' rather than 'correct reading.'
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a careful and methodologically inventive paper, and I'd send it to referees. My main caveat is about packaging, not method.
What's actually new: the paired probe/action measurement itself — independent factual probes on the identical state through a choice-only interface where no text or hidden states are exposed — plus the model-free sufficiency criterion (Eq. 1), which is the right way to ask whether a probe set can license a decision-failure diagnosis at all. The composition ladder in Section 6.1 is the strongest part: probes sufficient by construction, gold verified by enumeration, distractors and overwritten facts as controls, and the rewrite experiments genuinely separate consolidation failure from cue capture. That is reusable methodology.
The empirical hygiene is unusually careful: hash-sealed frozen benchmark, two independent label audits (the ALFWorld co-located-receptacle correction is a real find — the models were answering the text, not the simulator label), exact-lookahead regret analysis, cluster-bootstrapped intervals, confirmations from new seeds. The 141 states that defeat all fourteen competent systems is striking and not a measurement artifact.
The soft spots align with the paper's own Limitations, plus one it doesn't state. First, the headline U1D0 shares in Table 2 rest on natural states where the authors admit the probes are not sufficient for the decision, so "most wrong actions follow correct reading" in the abstract outruns what the data establish. The stress-test is right that this could have been narrowed: the Eq. 1 check is feasible on the 13,678 simulator states and is not reported there. Second, the sub-goal block in Section 6.3 is derived from gold-action categories, so it leaks most of the answer and the +16 point effect for JEV is inflated; the model's own sub-goal accuracy partially rescues the mechanism story. Third, the growth of the U1D0 error share with accuracy (Fig. 3b) is partly mechanical — as measured-fact accuracy approaches 0.9, errors are forced into the U=1 cell by pigeonhole. The conditional P(D=1|U=1) columns (10–21% wrong actions even with every measured fact right) are artifact-free and carry the finding, but the paper should say the share measure is base-rate-sensitive. Also worth noting: most of GPT-6.1's "errors" are optimal under exact lookahead, which tempers the 85% figure.
This is for people building fast agent loops and evaluators of System One models. Recommendation: accept for peer review. The abstract needs to be brought in line with the paper's own Limitations, and the revision should run the sufficiency check on the simulator states and report base-rate-adjusted shares. The core diagnostic apparatus is sound.
The strongest objection
The headline claim that most wrong actions follow correct reading is carried by the U1D0 shares in §5.1: for the six most accurate systems, 73–85% of decision errors occur with every measured fact correct. Those shares are computed on the natural paired states, where the paper's own sufficiency criterion does not license the diagnosis. Eq. 1 in §2.4 requires the gold sets of all states sharing a probe vector to intersect, and §2.4 concedes that in natural data 'absence of a counterexample does not prove sufficiency'; §10 states that on natural states the probes are not sufficient for the decision. A U1D0 outcome there means only that all measured facts were answered correctly, not that the model read the decision-relevant facts correctly; the same outcome is consistent with an unprobed fact being missed. The growth of the U1D0 share with accuracy (Fig. 3b) is also partly a base-rate artifact: as U rises toward 0.85–0.94, errors are increasingly forced into the U=1 category regardless of their cause. §6.1 establishes composition failures only on generated states whose probes are sufficient by construction, and §6.3 does so only for ALFWorld sub-goals; neither rescues the cross-domain reading of Table 2. The §2.4 check is feasible for the 13,678 simulator states in the paired set, and the paper does not report running it there.
The load-bearing step is the inference from a prevalence statistic to a diagnosis: §5.1's 73–85% is interpreted as evidence that strong systems read correctly and then choose wrongly. The paper defines the condition under which that inference is valid (Eq. 1) and then concedes that it is not established on the natural states where the statistic is computed. This is not a side caveat: the abstract states the conclusion without it, and the title names the diagnosis. I agree with the reader's weakest_assumption. The independent evidence in §6.1 and §6.3 is real, showing composition and sub-goal failures in the domains they cover, but it does not establish the cross-domain prevalence. The sub-goal block leakage identified by the reader is secondary; it affects the §6.3 effect size, not the cross-domain prevalence. The proposed check would settle the matter for the simulator majority of the paired states: if the 73–85% shares survive after removing states whose probe groups violate Eq. 1, the headline is supported; if they drop, the abstract and §5.1 must be re-scoped. Because the paper discloses the limitation and has strong secondary evidence, I do not move the verdict; CONDITIONAL remains appropriate pending this check.
The referee's report
If the central claim holds, this is a genuinely useful contribution: it converts an unanswerable introspection question (did the model misread or misdecide?) into a falsifiable behavioral measurement, and it ships unusually careful apparatus — a hash-sealed frozen manifest, closed-world enumeration of admissible actions, gold sets checked by breadth-first search, a model-free sufficiency criterion verified by enumeration on the generated states, exact-lookahead regret analysis showing that 5–42% of measured simulator errors are actually optimal, two label audits whose exclusion makes the headline estimate conservative, cluster-bootstrapped intervals, and confirmation cohorts from fresh seeds. The §6.1 composition ladder is the strongest part of the paper: it cleanly separates reading from composing on states where the diagnosis is licensed. The main gap is that the cross-domain headline in the abstract rests on natural-state U1D0 shares that the paper’s own criterion does not license, so the size and scope of the headline claim are larger than the evidence; the diagnostic instrument, however, remains valuable independently of that gap.
Pith's reading of the paper (a machine's summary, not the authors')
Fast 'System One' models pick an agent's next action in a single pass and leave nothing to inspect when they err, so it was unknown whether they misread the agent's state or read it correctly and still chose badly. This paper separates the two behaviorally by pairing every next-action question on a frozen 19,663-state benchmark with independent factual probes about the same state, across simulated household tasks, customer-service policies, tool use, web navigation and coding. Across seventeen fast systems and a reasoning model the answer is consistent: most wrong actions follow correct reading — for the six most accurate systems, 73-85% of decision errors occur on states where every measured fact was answered correctly, and the share grows with accuracy. On generated states whose probes provably determine the action, models that read every fact and decide perfectly from the facts alone lose 20-45 points when they must do both from a trace, while composing the action in code from the models' own answers reaches 0.96-1.00; on real household states the missing piece is often the next sub-goal, not any fact. The upshot is a diagnostic instrument that tells practitioners whether a failure calls for a better state representation or for a composed decision — and for fast models, the second is the usual case.
What carries the argument. The load-bearing instrument is the paired measurement: on each decision state the model answers, in isolation, the next-action question and closed factual probes about the same state (last-action outcome, location, inventory, permissions, evidence). Crossing 'every measured fact correct' ($U{=}1$) with 'action correct' ($D{=}1$) gives four outcomes, of which the cell $U{=}1, D{=}0$ — facts right, action wrong — is the diagnostic target. Because that cell licenses less than it seems, the paper adds a model-free sufficiency criterion (Eq. 1): a probe set can support a decision-failure diagnosis on a group of states only if states sharing the same probe-answer vector have non-empty intersection of their gold action sets; on finite, enumerable environments the check is exhaustive, and on natural data it is at best evidence, a gap the authors acknowledge. The third instrument is the composition ladder: 1,200 generated decision-list states where enumeration verifies that four rule facts determine a unique gold action, so that $U{=}1, D{=}0$ there can only mean the model read the facts and failed to use them. The ladder's rewrites (neutral wording, evidence placement, distractor removal, self-supplied summaries) separate two mechanisms behind the same outcome — consolidation of scattered evidence versus capture by irrelevant cues — and the decision-oriented probe study on 644 ALFWorld states adds sub-goal selection as a third.
Core claim. The paper claims that when a System One model picks the wrong next action, the error is usually not a failure to understand what the agent has seen. On the same 19,663 paired test states, the six most accurate systems answer every factual probe correctly on 73-85% of the states where they choose wrongly, and conditional on all measured facts being correct the action is still wrong in 10-21% of states; the converse also holds — with a fact wrong, the action is right 62-70% of the time — so neither direction of the pairing is a proxy for the other. The errors are not noise: all 14 competent fast systems fail with every fact right on the same 141 states, where chance predicts none, and each system decides correctly on 74-94% of states that no other system fails but on only 1-16% of states that all others fail. Where probe sufficiency is guaranteed by construction, models read the rule facts almost perfectly (97-100%) and several decide perfectly from the facts alone, yet lose 20-45 points from the raw trace; appending their own separately-asked fact answers recovers 15-28 points, and composing the action in code from those answers reaches 0.96-1.00. On real ALFWorld states, decision-oriented facts add at most 5.3 points for JEV while stating the next sub-goal adds 16.3, and 41% of JEV's errors occur with every fact and the sub-goal answered correctly in separate requests — the model can state the sub-goal it then fails to pursue — so the residual errors are errors of planning and composition, not of reading.
The premise it rests on. The claim that most wrong actions follow correct reading assumes that the factual probes on real states cover the facts the decision actually requires; the authors state directly that on natural states 'absence of a counterexample does not prove sufficiency' and that the probes are not sufficient for the decision there, so a facts-right/action-wrong outcome on those states could in part be a comprehension failure the probes never asked about.
If the paper is right (Pith's inferences, not the authors')
For any workflow built on a System One model, the paired outcomes tell where to intervene: errors after misread facts point to state representation or context budget, while a dominant facts-right/action-wrong share points to the decision itself.
When the decision-relevant facts and rules are explicit, the strongest available remedy is to ask the facts as narrow parallel questions and compose the action in code, which took every system tried to 0.96-1.00; a confidence cascade keeps most of the gain (JEV at 0.932) with a second request on only 38% of states.
In open-ended tasks the missing ingredient is the next sub-goal, which facts do not supply; on ALFWorld, stating the sub-goal added 16 points for JEV where any block of facts added at most 5, so those errors are a planning problem rather than a reading problem.
Accuracy alone is a poor summary of these models: two systems with similar paired accuracy fail for different reasons (consolidation for one, cue capture for another), a perfect action score on coding traces hid a twelve-point difference in reading, and the most accurate fast system is among the least calibrated (ECE 19.1%).
The 141 states that defeat every competent fast system form a sharper benchmark than uniformly weighted states; even the reasoning reference resolves only 40% of them, so those states are where deliberation actually buys something.
Editorial inference: if the reading-acting dissociation is a general property of single-pass choice models, the composition failure is not a long-context or distance problem — the paper's own ladder shows no loss over up to 15 intervening steps — but a consolidation problem: evidence scattered over a trace is not integrated before rule application. A testable extension is to measure whether the ga
Editorial inference: the paper's sufficiency criterion implies a benchmark-design principle it does not fully state — natural-state diagnostics should be stratified by whether probe sets pass the Eq. 1 intersection check; the headline 73-85% shares on natural states mix decision failures with unasked facts, which the authors concede, and only the generated ladder separates the two cleanly.
Editorial inference: the taxonomy the paper uncovers — consolidation failure, cue capture, and sub-goal selection — is a candidate classification of failure modes for choice-only models generally; a natural next test, which the paper leaves out of scope, is whether the same taxonomy reproduces in closed-loop execution where a wrong action changes the next state.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
System One models choose an agent's next action by scoring a supplied set of options in a single pass. This makes them fast and cheap, but when they choose wrongly there is nothing to inspect: did the model misread what the agent has seen, or read it correctly and still act wrongly? We answer behaviorally. On a frozen benchmark of agent decision states from simulated environments, customer-service policies, tool use, web navigation and coding, we pair every next-action question with factual probes about the same state, each answered independently. Across seventeen fast systems and a reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form. These errors are not noise; they gather on the same states across systems of every kind. Where the probed facts provably determine the action, models that read every fact still fail to combine them, and their own answers, composed by code, repair the failure. On real states, what is missing is often not a fact but the next sub-goal.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 1.00
“Across seventeen fast systems and a reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form.”
How the main claim is stated: asserted 0.96
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.53 · human participants 0.04 · controlled comparison 0.21 · randomized assignment 0.02 · states quantitative result 0.40 · compares with prior results 0.34 · mentions code or data availability 0.07
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| System One models choose an agent's next action by scoring a supplied set of options in a single pass. | background 0.99 | 0.08 | 0.02 | asserted 1.00 |
| This makes them fast and cheap, but when they choose wrongly there is nothing to inspect: did the model misread what the agent has seen, or read it correctly and still act wrongly? We answer behaviorally. | gap 0.70 | 0.25 | 0.08 | asserted 0.97 |
| On a frozen benchmark of agent decision states from simulated environments, customer-service policies, tool use, web navigation and coding, we pair every next-action question with factual probes about the same state, each answered independently. | method 0.99 | 0.94 | 0.03 | asserted 0.95 |
| Across seventeen fast systems and a reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form. | result 1.00 | 0.94 | 0.86 | asserted 0.96 |
| These errors are not noise; they gather on the same states across systems of every kind. | result 0.81 | 0.91 | 0.12 | asserted 1.00 |
| Where the probed facts provably determine the action, models that read every fact still fail to combine them, and their own answers, composed by code, repair the failure. | result 0.80 | 0.86 | 0.18 | asserted 1.00 |
| On real states, what is missing is often not a fact but the next sub-goal. | result 0.63 | 0.80 | 0.12 | asserted 0.85 |
What kind of paper the classifier reports this is: empirical study 0.99 · evidence from observation or experiment 0.76
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 5,608 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (11)
Figures are the authors', reproduced from the paper.
References (77)
- [1] Shridhar, Mohit and Yuan, Xingdi and C. International Conference on Learning Representations , year =. 2010.03768 , archivePrefix = on PithPith's reviewarXiv
- [2] Chevalier-Boisvert, Maxime and Bahdanau, Dzmitry and Lahlou, Salem and Willems, Lucas and Saharia, Chitwan and Nguyen, Thien Huu and Bengio, Yoshua , booktitle =. 2019 , url =. 1810.08272 , archivePrefix = on PithPith's reviewarXiv
- [3] Paglieri, Davide and Cupia. International Conference on Learning Representations , year =. 2411.13543 , archivePrefix = on PithPith's reviewarXiv
- [4] Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , url = on Pith
- [5] Cui, Christopher Zhang and Yuan, Xingdi and Xiao, Ziang and Ammanabrolu, Prithviraj and C. 2025 , eprint = on Pith
- [6] -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author =. International Conference on Learning Representations , year =. 2406.12045 , archivePrefix = on PithPith's reviewarXiv
- [7] ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author =. Proceedings of the 43rd International Conference on Machine Learning , series =. 2026 , url =. 2506.07982 , archivePrefix = on PithPith's reviewarXiv
- [8] Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Sam and Wang, Boshi and Sun, Huan and Su, Yu , booktitle =. 2023 , doi = on Pith
- [9] Ma, Chang and Zhang, Junlei and Zhu, Zhihao and Yang, Cheng and Yang, Yujiu and Jin, Yaohui and Lan, Zhenzhong and Kong, Lingpeng and He, Junxian , booktitle =. 2024 , doi = on Pith
- [10] Xia, Yu and Fan, Jingru and Chen, Weize and Yan, Siyu and Cong, Xin and Zhang, Zhong and Lu, Yaxi and Lin, Yankai and Liu, Zhiyuan and Sun, Maosong , editor =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , mon... on Pithdoi:10.18653/v1/2025.acl-long.945
- [11] Xu, Zhangchen and Soria, Adriana Meza and Tan, Shawn and Roy, Anurag and Agrawal, Ashish Sunil and Poovendran, Radha and Panda, Rameswar , booktitle =. 2026 , url =. 2510.01179 , archivePrefix = on PitharXiv
- [12] Shi, Zhengliang and Wang, Yuhan and Yan, Lingyong and Ren, Pengjie and Wang, Shuaiqiang and Yin, Dawei and Ren, Zhaochun , editor =. Retrieval Models Aren. Findings of the Association for Computational Linguistics:. 2025 , address =. doi:10.18653/v1/2025.findings-acl.1258 , pages = on Pithdoi:10.18653/v1/2025.findings-acl.1258
- [13] Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle =. The. 2025 , url = on Pith
- [14] Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi , editor =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , month = apr, year =. d... on Pithdoi:10.18653/v1/2025.naacl-long.174
- [15] L. Second Conference on Language Modeling (. 2025 , url =. 2504.08942 , archivePrefix = on PitharXiv
- [16] Yuan, Tongxin and He, Zhiwei and Dong, Lingzhong and Wang, Yiming and Zhao, Ruijie and Xia, Tian and Xu, Lizhen and Zhou, Binglin and Li, Fangqi and Zhang, Zhuosheng and Wang, Rui and Liu, Gongshen , editor =. Findings of the Association for Computational Linguistics:. 2024 , ... on Pithdoi:10.18653/v1/2024.findings-emnlp.79
- [17] Zhang, Shaokun and Yin, Ming and Zhang, Jieyu and Liu, Jiale and Han, Zhiguang and Zhang, Jingyang and Li, Beibin and Wang, Chi and Wang, Huazheng and Chen, Yiran and Wu, Qingyun , booktitle =. Which Agent Causes Task Failures and When? On Automated Failure Attribution of. 202... on PithPith's reviewarXiv
- [18] Liu, Jiale and Xi, Huajun and Zhang, Shaokun and Zeng, Yifan and Yue, Tianwei and Wang, Chi and Kang, Jian and Wu, Qingyun and Wang, Huazheng , year =. 2607.09996 , archivePrefix = on PithPith's reviewarXiv
- [19] Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation , author =. 2026 , eprint = on Pith
- [20] Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction , author =. Findings of the Association for Computational Linguistics:. 2024 , address =. doi:10.18653/v1/2024.findings-emnlp.905 , pages = on Pithdoi:10.18653/v1/2024.findings-emnlp.905
show 57 more
- [21] Sobotka, Jan and Karabag, Mustafa O. and Topcu, Ufuk , year =. Why Do. 2605.00226 , archivePrefix = on PithPith's reviewarXiv
- [22] Deb, Sagar and Krishnan, Ashwanth , year =. 2607.13618 , archivePrefix = on PithPith's reviewarXiv
- [23] A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents , author =. Proceedings of the 43rd International Conference on Machine Learning , series =. 2026 , url =. 2602.08964 , archivePrefix = on PithPith's reviewarXiv
- [24] Memory Control Signals Emerge Before Action in Long Horizon Agents , author =. 2026 , eprint = on Pith
- [25] Wang, Mingxuan and Chen, Hongyue and Guo, Yinglong and Luo, Fei and Ning, Chao and Wang, Bo and Yao, Guorun and Ma, Yanbiao and Han, Jungong , year =. 2609.27298 , archivePrefix = on PithPith's reviewarXiv
- [26] Schmied, Thomas and Bornschein, J. International Conference on Learning Representations , year =. 2504.16078 , archivePrefix = on PithPith's reviewarXiv
- [27] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =. 2210.03629 , archivePrefix = on PithPith's reviewarXiv
- [28] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , editor =. 2022 , doi = on Pith
- [29] Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting , author =. Advances in Neural Information Processing Systems , editor =. 2023 , doi = on Pith
- [30] Measuring Faithfulness in Chain-of-Thought Reasoning , author =. 2023 , eprint = on Pith
- [31] Thinking, Fast and Slow , author = on Pith
- [32] Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer , editor =. Beyond Accuracy: Behavioral Testing of. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , month = jul, year =. doi:10.18653/v1/2020.acl-main.442 , pages = on Pithdoi:10.18653/v1/2020.acl-main.442
- [33] Probing Classifiers: Promises, Shortcomings, and Advances , author =. Computational Linguistics , volume =. 2022 , address =. doi:10.1162/coli_a_00422 , pages = on PithPith's reviewdoi:10.1162/coli_a_00422
- [34] On Calibration of Modern Neural Networks , author =. Proceedings of the 34th International Conference on Machine Learning , series =. 2017 , url = on Pith
- [35] Language Models (Mostly) Know What They Know , author =. 2022 , eprint = on Pith
- [36] Decision Models , author =. 2026 , url = on Pith
- [37] Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making , author =. 2026 , url = on Pith
- [38] Palmer, Jared , howpublished =. 2026 , url = on Pith
- [39] Laya , author =. 2026 , url = on Pith
- [40] Cygnet Recipe: Typed Decisions from Frozen. 2026 , url = on Pith
- [41] Clef , author =. 2026 , url = on Pith
- [42] 2026 , howpublished = on Pith
- [43] Ravichander, Abhilasha and Gardner, Matt and Marasovi. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , month = dec, year =. doi:10.18653/v1/2022.emnlp-main.598 , pages = on Pithdoi:10.18653/v1/2022.emnlp-main.598
- [44] She, Jingyuan S. and Potts, Christopher and Bowman, Samuel R. and Geiger, Atticus , editor =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , month = jul, year =. doi:10.18653/v1/2023.acl-short.154 , pages = on Pithdoi:10.18653/v1/2023.acl-short.154
- [45] Tafjord, Oyvind and Dalvi, Bhavana and Clark, Peter , editor =. Findings of the Association for Computational Linguistics:. 2021 , address =. doi:10.18653/v1/2021.findings-acl.317 , pages = on Pithdoi:10.18653/v1/2021.findings-acl.317
- [46] Sinha, Koustuv and Sodhani, Shagun and Dong, Jin and Pineau, Joelle and Hamilton, William L. , editor =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (. 2019 , a... on Pithdoi:10.18653/v1/d19-1458
- [47] Trivedi, Harsh and Balasubramanian, Niranjan and Khot, Tushar and Sabharwal, Ashish , journal =. 2022 , address =. doi:10.1162/tacl_a_00475 , pages = on Pithdoi:10.1162/tacl_a_00475
- [48] Kuratov, Yuri and Bulatov, Aydar and Anokhin, Petr and Rodkin, Ivan and Sorokin, Dmitry and Sorokin, Artyom and Burtsev, Mikhail , booktitle =. 2024 , doi = on Pith
- [49] An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction , author =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (. 2019 , address =. doi:10.... on Pithdoi:10.18653/v1/d19-1131
- [50] FitzGerald, Jack and Hench, Christopher and Peris, Charith and Mackie, Scott and Rottmann, Kay and Sanchez, Ana and Nash, Aaron and Urbach, Liam and Kakarala, Vishesh and Singh, Richa and Ranganath, Swetha and Crist, Laurie and Britan, Misha and Leeuwis, Wouter and Tur, Gokhan... on Pithdoi:10.18653/v1/2023.acl-long.235
- [51] Jin, Zhijing and Chen, Yuen and Leeb, Felix and Gresele, Luigi and Kamal, Ojasv and Lyu, Zhiheng and Blin, Kevin and Gonzalez Adauto, Fernando and Kleiman-Weiner, Max and Sachan, Mrinmaya and Sch. Advances in Neural Information Processing Systems , editor =. 2023 , doi = on Pith
- [52] Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ... on Pithdoi:10.18653/v1/2024.naacl-long.102
- [53] Kazemi, Mehran and Yuan, Quan and Bhatia, Deepti and Kim, Najoung and Xu, Xin and Imbrasaite, Vaiva and Ramachandran, Deepak , booktitle =. 2023 , doi = on Pith
- [54] Mirzadeh, Iman and Alizadeh, Keivan and Shahrokhi, Hooman and Tuzel, Oncel and Bengio, Samy and Farajtabar, Mehrdad , booktitle =. 2025 , url =. 2410.05229 , archivePrefix = on PithPith's reviewarXiv
- [55] Pang, Richard Yuanzhe and Parrish, Alicia and Joshi, Nitish and Nangia, Nikita and Phang, Jason and Chen, Angelica and Padmakumar, Vishakh and Ma, Johnny and Thompson, Jana and He, He and Bowman, Samuel R. , editor =. Proceedings of the 2022 Conference of the North American Ch... on Pithdoi:10.18653/v1/2022.naacl-main.391
- [56] Rajpurkar, Pranav and Jia, Robin and Liang, Percy , editor =. Know What You Don. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , month = jul, year =. doi:10.18653/v1/P18-2124 , pages = on Pithdoi:10.18653/v1/p18-2124
- [57] Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset , author =. Proceedings of the. 2020 , doi = on Pith
- [58] Louis, Annie and Roth, Dan and Radlinski, Filip , editor =. ``. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (. 2020 , address =. doi:10.18653/v1/2020.emnlp-main.601 , pages = on Pithdoi:10.18653/v1/2020.emnlp-main.601
- [59] On Memorization of Large Language Models in Logical Reasoning , author =. 2024 , eprint = on Pith
- [60] Ming, Yifei and Purushwalkam, Senthil and Pandit, Shrey and Ke, Zixuan and Nguyen, Xuan-Phi and Xiong, Caiming and Joty, Shafiq , booktitle =. 2025 , url =. 2410.03727 , archivePrefix = on PithPith's reviewarXiv
- [61] Kwan, Wai-Chung and Zeng, Xingshan and Jiang, Yuxin and Wang, Yufei and Li, Liangyou and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wong, Kam-Fai , editor =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , month = nov, year =. doi:10... on Pithdoi:10.18653/v1/2024.emnlp-main.1124
- [62] Ruis, Laura and Khan, Akbir and Biderman, Stella and Hooker, Sara and Rockt. The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by. Advances in Neural Information Processing Systems , editor =. 2023 , doi = on Pith
- [63] Measuring and Narrowing the Compositionality Gap in Language Models , author =. Findings of the Association for Computational Linguistics: EMNLP 2023 , year =. doi:10.18653/v1/2023.findings-emnlp.378 , url = on Pithdoi:10.18653/v1/2023.findings-emnlp.378
- [64] Large Language Models Can Be Easily Distracted by Irrelevant Context , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , publisher = on Pith
- [65] Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =. 2024 , doi = on Pith
- [66] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models , author =. International Conference on Learning Representations , year = on Pith
- [67] Decomposed Prompting: A Modular Approach for Solving Complex Tasks , author =. International Conference on Learning Representations , year = on Pith
- [68] Gao, Luyu and Madaan, Aman and Zhou, Shuyan and Alon, Uri and Liu, Pengfei and Yang, Yiming and Callan, Jamie and Neubig, Graham , booktitle =. 2023 , publisher = on Pith
- [69] On the Planning Abilities of Large Language Models -- A Critical Investigation , author =. Advances in Neural Information Processing Systems , volume =. 2023 , pages = on Pith
- [70] Kambhampati, Subbarao and Valmeekam, Karthik and Guan, Lin and Verma, Mudit and Stechly, Kaya and Bhambri, Siddhant and Saldyt, Lucas and Murthy, Anil , booktitle =. Position:. 2024 , publisher = on Pith
- [71] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks , author =. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year = on Pith
- [72] Chen, Lingjiao and Zaharia, Matei and Zou, James , journal =. 2024 , url =. 2305.05176 , archivePrefix = on PithPith's reviewarXiv
- [73] Decisions. 2026 , url = on Pith
- [74] Eikos: Open, Calibrated, Single-Pass Typed-Decision Models for Finance and Trading , author =. 2026 , url = on Pith
- [75] decider-chat-gemma4-31b:. 2026 , url = on Pith
- [76] Nguyen, Chinh , howpublished =. 2026 , url = on Pith
- [77] Li, Yubo and Miao, Yidi and Krishnan, Ramayya and Padman, Rema , year =. 2609.26550 , archivePrefix = on PithPith's reviewarXiv
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment