{"id":"b2ba6f47-8a51-413a-bcc4-c70bac794325","arxiv_id":"2411.12828","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On the OEDD benchmark, LLM agents select the worse of two actions below chance when the correct choice requires combining two earlier facts and ignoring a recent distractor in contexts over 1,615 tokens.","lead":"A new benchmark called OEDD tests whether AI agents can make the right choice from a long history of past experiences while a distracting irrelevant fact is present. In the hardest conditions, three top AI models picked the worse option more often than a random guess.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'worse than random' claim may be an artifact of fixed option order: the paper never reports counterbalancing the position of the correct versus distractor action, and the only rendered example places the correct action consistently at A.","rationale":"The reader's weakest assumption concerns the correctness of the ground-truth 'better action' labels. That is a legitimate validity threat, but the paper's annotation process was explicitly designed to address it, and the single rendered example is not obviously ambiguous. The more pressing threat to the headline claim is the complete absence of any reported control for option order. Because the paper reshuffles only historical episodes and never mentions randomizing the two options in the final episode, the fixed A/B arrangement in the prompt (with the correct action placed first in the example) leaves open a direct alternative explanation: a position preference could produce below-chance accuracy that has nothing to do with two-premise reasoning or red-herring distraction. This is precisely the kind of confound that a careful empirical study must pre-register or at least counterbalance, especially when the claim is explicitly about being 'worse than random.' The significance-test omission identified by the reader is real and should also be fixed, but it is not the most load-bearing issue: adding a t-test against 0.5 would still not establish that the below-chance result is caused by the intended cognitive failure. Therefore the verdict should remain conditional, but the required conditions should include demonstration that option order is balanced or that the effect persists under swapped order. I partially agree with the reader because the missing significance test is part of the same family of baseline-rigor concerns, but I do not see ground-truth ambiguity as the primary risk to the central claim.","tokens_in":9424,"tokens_out":8020,"duration_ms":81877,"concrete_test":"For each of the 16 OEDD scenarios in the 2P+RH Medium and Long conditions, swap the order of the two options in the current observation (so the original correct action becomes B and the distractor action becomes A), while keeping all historical episodes, prompt templates, and sampling settings identical. Re-run the same protocol on GPT-4o and Gemini 1.5 Pro. If accuracy follows the swapped position (i.e., the model now picks the originally distractor action more often), the below-chance effect is caused or inflated by option-order bias, and the headline claim as stated is unsupported. If accuracy remains similarly below 0.5 regardless of position, the position-bias confound is rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that sub-0.5 accuracy reflects a reasoning failure under the OEDD manipulation (long context, two disparate premises, red herring). But the experimental protocol restores only the order of historical episodes, not the order of the two options in the final episode (Section 4: 'we reshuffled the order of the historical episodes preceding the final episode before every sample was taken'). The prompt template in Appendix A.3 renders options in a fixed list, and the full example in Appendix A.5 shows the correct action as A and the red-herring-paired action as B. If the correct action is consistently placed in the same position across the 16 scenarios, any systematic LLM position preference (e.g., a bias toward the last listed option) would mechanically produce accuracy below 0.5, entirely independent of the intended reasoning load. This is not a statistical nuance: it changes the meaning of the random baseline. A model with a 0.6 preference for position B would score 0.4 when the correct answer is always A, looking 'worse than random' while actually following a fixed heuristic. The authors do not mention randomizing or counterbalancing option positions across tests or samples, so this alternative explanation is not ruled out. The missing direct significance test against 0.5 (noted by the reader) is secondary: even a significant below-chance p-value would not distinguish a reasoning deficit from a position bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OEDD (Operationalize Experience Despite Distraction), a corpus of 16 human-annotator-validated agent scenarios in which an LLM must choose between two actions using information from two disparate premises in long pre-scripted interaction histories, while ignoring a distracting red herring. The authors evaluate GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro with a zero-shot chain-of-thought prompt and report that, under the hardest conditions (two premises plus a red herring and long context), all models perform worse than random choice. The corpus and code are publicly available.","tokens_in":9677,"tokens_out":3834,"duration_ms":37152,"significance":"If the reported effect is genuine, the OEDD benchmark is a valuable contribution that exposes a surprising and practically relevant failure mode of LLM agents: the inability to integrate two disparate facts in long contexts despite a trivial distractor. The authors have taken reasonable steps to ensure ground-truth validity through human annotation, and the public release of code and corpus supports reproducibility. The design of shuffleable, temporally independent episodes is a useful methodological idea. However, the paper's central claim of below-random accuracy is not yet adequately supported because the experimental protocol does not control for option-position bias, and no direct statistical test against the 0.5 chance baseline is reported.","major_comments":[{"comment":"The experimental protocol never mentions counterbalancing or randomizing the order of the two current action options. The only fully rendered example in Appendix A.5 places the correct action at position A, and the prompt template renders options in a fixed list. If the correct action is consistently at the same position across the 16 scenarios, a model with any systematic position preference (e.g., a bias toward the last-listed option) would mechanically exhibit accuracy below 0.5, independent of the intended reasoning load. The paper claims that reshuffling only the historical episode order controls for order effects, but this does not address the order of options in the final decision step. To support the 'worse than random' claim, the authors must either randomize or counterbalance option positions across samples, or report accuracy separately for each option position and show that the below-chance result is not driven by a position artifact.","section":"§4, A.3, A.5"},{"comment":"The abstract and Discussion assert that the models 'perform worse than random choice,' but the paper never reports a statistical comparison of the estimated probabilities against the 0.5 chance baseline. Figure 2 shows only color-coded matrices without numeric accuracy values or confidence intervals, and Table 3 reports pairwise comparisons between setups (e.g., 1P vs. 2P) rather than one-sample tests against chance. Because the central claim is explicitly about being worse than random, the authors should provide per-configuration accuracy estimates with confidence intervals and perform a direct test of the null hypothesis that the true selection probability equals 0.5 (e.g., a one-sample binomial test or a bootstrap interval). Without this, the headline conclusion is not formally supported even setting aside the option-order confound.","section":"§4, Figure 2, §5"},{"comment":"The abstract states that 'all LLMs perform worse than random choice' under the specified conditions, but the Discussion only explicitly states this for GPT-4o and Gemini 1.5 Pro, while GPT-3.5 Turbo is described as 'significantly worse than GPT-4o and Gemini 1.5' without an explicit statement that it falls below chance. If the authors intend the 'all LLMs' claim, they must document the GPT-3.5 result separately and show that it is below 0.5 with statistical significance. If not, the abstract should be revised to match the reported results.","section":"Abstract vs. §5"}],"minor_comments":[{"comment":"The sentence 'During an episode, the agent may or may not be be exposed to one of the constituents' contains a typo ('be be').","section":"§3"},{"comment":"The example prompt contains a misspelling: 'restuarant' should be 'restaurant'.","section":"A.5"},{"comment":"Several reference errors need correction: 'Jian V ora' should be 'Jian Vora' (Huang et al.), 'Anto P V au2' appears to contain a garbled author name (Srivastava et al.), and 'V oyager' should be 'Voyager' (Wang et al., 2023).","section":"References"},{"comment":"The color-coded heatmap would be more informative if numeric accuracy values and confidence intervals were overlaid, and if the colorbar were explicitly labeled with the scale.","section":"Figure 2"},{"comment":"The claim 'To our knowledge at the time of submission, we are the first to do this' is somewhat strong given the rapid evolution of the field; consider softening to 'we are not aware of prior work' and adding the date of the literature search.","section":"§2"},{"comment":"The table reports 36 pairwise significance tests without any multiple-comparison correction. While the p-values are generally very small, the authors should mention this issue or apply a correction such as the Benjamini-Hochberg procedure.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The option-order confound is the critical issue: if the correct action is consistently placed at the same list position, the below-chance result may say more about LLM position biases than about the OEDD manipulation. I would ask the authors to re-run the evaluation with randomized option order (or at least report per-position accuracy) and to add direct significance tests against the 0.5 baseline. The corpus itself appears useful and the public code is a strength, so the paper could become acceptable after these additions. The abstract should also be made consistent with the Discussion regarding which models are claimed to be below random."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe OEDD corpus is a genuinely new resource: 16 human-validated agent scenarios with controlled variation in reasoning complexity, context length, and distracting information, with code and data released. That part is solid and worth building on. The headline claim, however, is not supported as presented: 'worse than random' is asserted from point estimates without a significance test against 0.5, and the abstract's 'all LLMs' overstates what the discussion reports for GPT-4o and Gemini 1.5 Pro.\n\nThe bigger problem is a possible confound. The paper states that historical episode order was reshuffled before each sample, but never says the order of the two action options in the final episode was randomized or counterbalanced. The one fully rendered example puts the correct action at A. If that position is fixed across scenarios, any systematic model preference for the other position would mechanically produce accuracy below 0.5, regardless of reasoning load. That would make the 'below chance' result an artifact of position bias, not a demonstration that the OEDD manipulations cause the failure. Even adding a direct significance test against 0.5 would not fix this, because a significant below-chance p-value would still be ambiguous.\n\nI don't want to overstate the damage. The comparative results—2P+RH being harder than 1P, red herrings hurting performance—are likely less affected, since the same position structure would be present across conditions. And the corpus itself is a contribution: human validation, public code, and a clean separation of reasoning setups are all done well. The limitations section is honest about corpus size and potential biases.\n\nAs it stands, I would not cite the below-chance claim, but I would cite the corpus if I needed a controlled multi-hop agent benchmark. I think this deserves peer review, but the authors must be required to (1) report or add option-order randomization/counterbalancing, (2) perform direct tests against the chance baseline, and (3) align the abstract with the specific model results. With those, the paper could be a useful contribution.","headline":"Useful new benchmark, but the headline below-chance result is confounded by a missing option-order counterbalance and unsupported by a direct test against chance.","tokens_in":10214,"tokens_out":4175,"would_cite":false,"duration_ms":41245,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs score below random chance on decisions requiring two distant facts plus a red herring.","keywords":["LLM agents","OEDD corpus","multi-hop reasoning","red herring","chain-of-thought","action inference","context length","benchmark"],"falsifier":"Re-run the 2P+RH Medium and Long configurations on GPT-4o and Gemini 1.5 Pro with at least 200 samples per test at temperature 0.4 using the same prompt template; if either model's accuracy is statistically significantly above 0.5, the central claim of below-random performance is contradicted. Alternatively, have a fresh, blinded panel of annotators independently label the better action in all 16 scenarios without seeing the intended answers; if the majority disagree with the original labels, the result may reflect annotator disagreement rather than model failure.","tokens_in":9209,"feed_emoji":"🤖","tokens_out":3366,"duration_ms":36919,"temperature":0.7,"pith_summary":"This paper asks whether large language model agents can use accumulated experience in a prompt to make a single good action choice when the decisive reasoning requires combining two separate facts and ignoring a trivial distractor. To test this, the authors built the OEDD corpus, a set of 16 human-validated scenarios with pre-scripted agent histories, expanded into 192 configurations. They then evaluated three state-of-the-art models with a minimal chain-of-thought prompt. The central finding is that when the history is longer than about 1,615 tokens, the decisive premise requires combining two disparate facts, and a red herring appears just before the decision, all three models choose the better action less often than random guessing. The paper argues this reveals a persistent failure mode in current LLM agents.","feed_headline":"LLMs drop below random on long distracted decisions","feed_subtitle":"At over 1,615 tokens, a two-part premise plus a red herring makes GPT-4o, GPT-3.5, and Gemini choose wrong more often than chance.","key_machinery":"The OEDD corpus is the load-bearing object: each scenario is built from a set of constituent statements (facts, a derived decision-informing premise, and a red herring), validated by paid human annotators through a custom interface, then fleshed out into temporally independent episodes that can be reshuffled before a final decision episode. The setups are 1P, 2P, 1P+RH, and 2P+RH, and length bands are Short, Medium, and Long. The prompting strategy is a minimal zero-shot chain-of-thought template that asks the model to first produce a reasoning statement and then output the chosen action in JSON, with episodes rendered as observation/options/decision triples.","core_discovery":"The paper introduces OEDD, a corpus of agent scenarios where an action decision is correct only if the agent infers a premise from two disparate environment facts revealed earlier in separate, shuffleable episodes, while ignoring a deliberately planted red herring that appears in the final episode. Across four reasoning setups (single premise, two premise, each with or without a red herring) and three length bands, the authors measured the probability that GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro select the better of two actions. Their central claim is that in the two-premise plus red-herring setup at medium or long length (over 1,615 tokens), every tested model performs worse than random choice. They also report that the red herring consistently degrades performance in both one-premise and two-premise conditions, and that performance generally declines as context length grows.","pith_inferences":["The below-chance performance in the 2P+RH condition hints that these models may rely on recency-weighted pattern matching rather than genuine cumulative inference: the red herring, being newest and topically similar, overwrites the earlier premises. A direct test would be to place the red herring before the two premises and see if the effect diminishes.","If this failure generalizes to deployed agents, then long-context memory systems (browser logs, robot histories, conversational memory) could systematically push agents toward worse decisions exactly when they have the most evidence—a concrete safety concern for autonomous LLM agents.","Because the benchmark isolates a single inference step, a model that fails OEDD cannot be trusted on compositional long-horizon tasks even if it does well on multi-step benchmark suites, since those suites may allow the model to recover from a single bad step.","The corpus's emphasis on novel, environment-specific premises makes it a candidate for diagnosing whether models reason from the actual prompt content rather than from corpus-level priors; the authors' below-chance result is consistent with the model ignoring the two disparate facts entirely."],"forward_implications":["If the result is correct, current frontier LLMs cannot be relied upon to operationalize long accumulated experience for even a single action decision when the decisive inference spans two separated facts and a distractor follows.","The red herring effect is not marginal: it significantly reduces accuracy in all tested setups, suggesting that LLM agents are highly sensitive to recent, topic-overlapping but irrelevant information.","Context length itself degrades reasoning, so an agent with more history may make worse decisions, a direct challenge to the assumption that more context helps.","The OEDD corpus provides a controlled, publicly available benchmark for isolating single-step action inference from multi-step task completion, enabling future work on prompting, retrieval augmentation, and training strategies.","The annotator-validation protocol, including covertly planted violations, offers a reusable method for filtering ambiguous reasoning-test content."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting method that the paper adapts as its minimal zero-shot strategy.","marker":"Wei et al., 2023"},{"why":"Shows that large language models are zero-shot reasoners, grounding the paper's decision to use a minimal prompt without few-shot exemplars.","marker":"Kojima et al., 2022"},{"why":"Frames LLM agents and the expectation that they reason over accumulated experience, motivating the OEDD design.","marker":"Xi et al., 2023"}],"fun_headline_variants":["Long prompts plus red herring make LLMs worse than random","OEDD test: LLMs fail on distracted long-context choices","Over 1,615 tokens, red herrings drop LLM accuracy below chance","Red herrings in long prompts make LLMs worse than chance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground truth of which action is 'better' in each OEDD scenario is assumed to be correctly identified by the authors and validated by two annotators; if annotators missed ambiguities, the below-chance result could be an artifact of noisy labels rather than genuine reasoning failure.","fun_headline_variants_meta":{"raw":{"variants":["Long prompts plus red herring make LLMs worse than random","OEDD test: LLMs fail on distracted long-context choices","Over 1,615 tokens, red herrings drop LLM accuracy below chance","Red herrings in long prompts make LLMs worse than chance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001063,"raw_usage":{"total_tokens":4456,"prompt_tokens":941,"completion_tokens":3515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":3447}},"tokens_in":557,"tokens_out":3515,"duration_ms":25261,"temperature":1.0,"reasoning_tokens":3447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:08:32.684110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 2P+RH Medium and Long configurations on GPT-4o and Gemini 1.5 Pro with at least 200 samples per test at temperature 0.4 using the same prompt template; if either model's accuracy is statistically significantly above 0.5, the central claim of below-random performance is contradicted. Alternatively, have a fresh, blinded panel of annotators independently label the better action in all 16 scenarios without seeing the intended answers; if the majority disagree with the original labels, the result may reflect annotator disagreement rather than model failure.","supporting_citations":[],"review_version":1}