{"id":"87bb5e4d-1c41-432d-9fbf-803096c493e3","arxiv_id":"2609.39496","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Jev selects correct numeric answers in 99% of cases but rejects invalid menus only 7% of the time; a confidence threshold raises rejection to 79% at 97% selection.","lead":"Tests on Jev, a decision-making model that picks from given options, show it almost always selects the correct numeric answer when one exists (99%) but rejects a menu only 7% of the time when the right answer is absent, despite an explicit None option. A simple threshold on the model's own confidence scores recovers most rejection, to 79%, without retraining.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The 7% rejection figure and the 92-point gap are single-snapshot measurements from an unversioned proprietary endpoint; the one test that decides whether they reflect model behavior or API-state artifact is a rerun-stability and version-pinning check.","rationale":"I read the paper in good faith and find it internally consistent and unusually well controlled. The paired present/absent design, the three matched decision formulations (Menu Choice vs T/F Choice vs native Boolean differing only in a type field), the computation-removal control (supplying the result lifts rejection from 7% to 100%, b83), the depth ablation (collapse after two operations), the operator-structure results (Table 1), the distance bands (Figure 4c), and the generative Qwen control (100% rejection on the same candidate sets) together make a strong, self-consistent case that the bottleneck is real, is specific to arithmetic computation in the categorical interface, and is not caused by malformed candidates. All headline arithmetic checks out: balanced accuracy rises from (99+7)/2=53% to (97+79)/2=88%; purchase (94+24)/2=59% to (86+79)/2=82.5%. The calibration threshold is selected on a 20-case pilot and applied to an explicitly disjoint 100-case transfer set, so there is no leakage. The single genuinely load-bearing uncertainty is that every quantitative claim is a snapshot of an unversioned, proprietary, hosted API, with no determinism check, no version pinning, and no shipped outputs; if that API is nondeterministic or shifted mid-study, the exact 7% and the exact 79% recovery could be configurational artifacts. The consistency across ~40 independent samples makes a nondeterminism artifact unlikely to overturn the qualitative conclusion, which is why this is a reproducibility concern supporting the reader's CONDITIONAL verdict rather than a reason to reject. The reader's weakest_assumption identifies precisely this determinism/version-stability premise, so I agree; the reader's secondary distance-sensitivity concern I regard as a documented feature rather than a flaw. My recommendation is therefore UNCHANGED: the conditional verdict stands until a rerun-stability test is performed.","tokens_in":11724,"tokens_out":7580,"duration_ms":54386,"concrete_test":"Re-submit the frozen 100-case arithmetic suite (with activations frozen in identical payloads) twice in immediate succession and again on three separate days, recording the Jev endpoint and returned model/version identifier for every batch. Compute present/absent accuracy per rerun: if Menu Choice rejection moves by more than ~5 points across reruns, or if batch-to-batch variance exceeds the within-run sampling error, the 92-point headline gap must be re-verified against pinned versions. Additionally require that all numbers in Figure 3 and Table 2 be reproduced from saved per-case outputs collected under a single pinned API version before the conditional verdict is lifted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that selection accuracy is an incomplete reliability measure because Menu Choice rejects only 7% of answer-absent arithmetic menus while native Boolean verification hits 99% on the same cases — rests on point estimates from one proprietary, hosted, unversioned endpoint (Jev), with no code, saved responses, seeds, or commit shipped. The Limitations paragraph (b183) acknowledges API-only access to weights/internals but never addresses determinism or version stability. If the API performs request-level sampling (e.g., a top-level temperature) or if the deployed model changed during data collection, the 7% on 100 cases (95% CI roughly ±5 points) and the calibrated 79% recovery could be artifacts of a transient configuration rather than stable decision behavior. The internal evidence softens but does not eliminate this concern: the failure replicates across roughly 40 independent samples (7%, 4%, 2% rejection at two operations in Figure 4a; the 127/128 vs 14/128 inventory split), and its sharp task/operator/distance structure (lookup 100%, refund-policy and logical-eligibility 64/64, multiplication/division chains near-perfect, Table 1) would demand a remarkably selective disturbance to be explained by version drift. So nondeterminism is unlikely to overturn the qualitative finding, but it is the single assumption whose violation would invalidate the exact numbers the whole paper reports, and it is not tested anywhere. The reader's secondary concern (distance sensitivity, Figure 4c) is a documented scope condition — the paper shows rejection rises with distractor error and the repair targets exactly the near-distractor regime relevant to reconciliation — so it is not load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper reports an empirical study of Jev, a hosted typed-decision API that selects directly from predefined options. Using paired answer-present and answer-absent menus with programmatic ground truth, it finds 99% answer-present accuracy but only 7% correct rejection on two-operation arithmetic (a+b−c) despite an explicit NONE option; purchase totals show a 70-point gap (94% vs 24%). Native Boolean verification on the same answer-absent cases reaches 99%, while candidate-level categorical T/F Choice achieves only 27%, isolating the decision interface rather than the underlying computation. An error analysis shows the returned scores retain separation (T/F AUC .985 despite 88/200 exact-match menus), motivating a development-selected threshold (τ=.03) that raises held-out menu rejection from 7% to 79% while retaining 97% selection accuracy. Extensive ablations — computation depth, numerical magnitude, distractor distance, operator composition, 15 task families, rejection-label and position variants, and a Qwen generative control — characterize the effect as arithmetic-dependent and task-dependent.","tokens_in":12149,"tokens_out":8843,"duration_ms":59870,"significance":"If the finding holds, it is a useful cautionary result for decision workflows: high selection accuracy in the presence of an explicit rejection option is not a reliable guarantee of correct rejection on arithmetic-dependent menus, even when the model can verify candidate correctness in a Boolean interface. The study design is genuinely careful: programmatic ground truth, paired menus preserving facts and two distractors, matched payloads differing only in the API type field (verified programmatically), frozen thresholds evaluated on disjoint held-out sets, paired bootstrap CIs with a fixed seed, explicit denominators for small counts, and a control battery that includes an answer-supplied condition, a generative model on the same menus, and label/position perturbations. These strengths make the qualitative claim robust. The main residual risks concern the exact numbers, which come from a single unversioned proprietary endpoint, and the stability of the calibration threshold selected on only 20 development cases; both are addressable and neither overturns the central qualitative finding.","major_comments":[{"comment":"Every headline number — 7% rejection, the 92-point gap, the 99% Boolean contrast, and the calibrated 79% — is a point estimate from a hosted, unversioned proprietary endpoint, yet the manuscript never tests request-level determinism or pins the deployed model version. The Limitations paragraph (b183) acknowledges API-only access but stops short of addressing this. The internal evidence mitigates the risk substantially (the failure reproduces across roughly 40 independent samples, has a sharp task/operator/distance structure, and coexists with simultaneous Boolean success), so a wholesale artifact is implausible. Nevertheless, a rerun-stability check is cheap and decisive: re-submit a fixed sample of saved requests over several sessions and report response agreement together with the API version field. Without it, the precision of statements such as 'Menu present/absent accuracy falls from 100%/66% at one operation to 98%/4% at two' (Fig. 4a) rests on an untested stability assumption.","section":"§4.2; Limitations (b183)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The calibration protocol selects τ by grid search over 101 values on only 20 development problems per condition (40 cases) with a tie-breaking rule, and the paper reports a single selected τ per task/interface. The disjoint held-out evaluation is honest, but a sensitivity analysis is needed to show the headline 7→79 recovery is not knife-edge: report held-out present/absent accuracy for τ in a neighborhood of each selected value (e.g., .02–.04 for arithmetic). With only 20 absent dev cases, the dev objective is quantized in 5% steps, and the selected τ could sit at a boundary that the small dev sample chose arbitrarily; the held-out CI in Fig. 8b partially addresses this, but not the sensitivity of the result to τ itself.","section":"§5.2; Table 2; Fig. 8","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract and Fig. 1a present 7% without the distractor-distance caveat; §4.4 (Fig. 4c) shows menu rejection rises from 20% at distractor error 1–5 to 88% at 195–200. Recommend that the abstract mention the near-distractor construction {−2,+1,+3} or reference Fig. 4c.","section":"Abstract; Fig. 1a; §4.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"In the depth curve, clarify the depth-zero baseline explicitly; the text reports one and two operations but not zero, which §4.3 implies is 100% rejection when the result is supplied.","section":"§4.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"With n=10 per operator pattern, differences such as 1/10 versus 9/10 absent rejection carry wide intervals; add binomial CIs or label the table as exploratory.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"T/F Choice batches all three candidate questions in one request; a sentence on whether per-request batching introduces ordering or position effects beyond the menu format would strengthen the interface-isolation claim.","section":"§3.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The tie-breaking rule ('ties favor proximity to .5, then the smaller threshold') can move τ within flat regions of the dev objective; a one-sentence check of whether the reported τ values are robust to that rule would be useful.","section":"§5.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Consider releasing the saved request payloads and recorded scores (the paper already states scores are recorded and calibration is performed offline); this would also support the rerun-stability check in the first major comment.","section":"Reproducibility","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The headline is that Jev's Menu Choice picks the right answer 99% of the time but rejects a wrong-only menu 7% of the time on the same arithmetic — and the paper mostly earns that claim. The design is the strongest thing about it: programmatic ground truth, paired menus that differ only in coverage, payloads identical up to the type field, thresholds fit on 20 development cases and frozen on disjoint 100-case evaluation sets. The ablations actually do the work. Boolean verification hits 99% on the same absent menus, supplying the computed answer restores rejection to 100%, and the T/F Choice ablation at 27% shows the loss is interface-specific rather than a computation failure. The Qwen control handling the same candidate sets perfectly closes the 'the candidates are somehow poisoned' escape hatch. The failure also replicates: 127/128 versus 14/128 on contextual inventory, and it has a sharp operator/distance/task structure that no plausible request-level noise would produce. The claim that selection accuracy alone is an incomplete reliability measure is supported.\n\nThe soft spot is exactly what the stress test flags. All the percentages are a single snapshot of an unversioned hosted endpoint, and the Limitations paragraph (b183) concedes API-only access without ever addressing deterministic stability or version pinning. No code, saved responses, or commits ship. If the API samples internally or the deployed model changed mid-collection, the exact 7% and 79% move. I do not think that overturns the qualitative result — the failure's structure is too selective to be explained by noise or version drift — but the paper's numbers are not independently reproducible as written, and the fix (a rerun-stability and version-pinning check) is cheap.\n\nMinor concerns stay minor. The threshold is tuned on 20 cases, which is thin but legitimate because evaluation is disjoint and the threshold is frozen. Distance sensitivity (Figure 4c) is a documented scope condition, not a flaw. The 100-case denominators put roughly five points around the 7% figure.\n\nThis paper is for anyone building or deploying typed decision APIs in high-stakes workflows; the paired-coverage measurement design is itself the contribution worth taking. It deserves a serious referee — I would send it, conditional on a determinism/version-stability check and ideally shipped artifacts.","headline":"A well-controlled behavioral study showing that an explicit rejection option fails on arithmetic in a typed decision API — the qualitative result is robust, but the exact percentages rest on a determinism check the paper never runs.","tokens_in":13368,"tokens_out":2405,"duration_ms":32369},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A model that selects the right arithmetic answer 99% of the time correctly rejects a menu with no valid answer only 7% of the time, even with an explicit None option, and a score threshold lifts rejection to 79% without retraining.","keywords":["rejection","abstention","typed decision models","arithmetic reasoning","threshold calibration","menu choice","boolean verification","none-of-the-above"],"falsifier":"Re-query the hosted Jev API on the same 100 answer-absent arithmetic menus, verify that repeated requests return identical choices, and check whether the 7% correct-rejection figure reproduces exactly; separately, rebuild an absent menu with distractors far from the true answer (errors near 200 rather than 1–3) and observe whether Menu Choice rejection rises toward the calibrated 79% range, which would show that the near-distractor construction, rather than categorical rejection being impossible, drives the reported number.","tokens_in":13195,"feed_emoji":"🧮","tokens_out":3769,"duration_ms":25349,"temperature":0.7,"pith_summary":"The paper asks whether a model that reliably selects correct answers can also recognize when every supplied answer is wrong, and it finds the answer is no. On paired arithmetic menus, Jev picks the correct candidate 99% of the time when one exists but correctly rejects only 7% of answer-absent menus despite an explicit None option. Native Boolean verification on the same answer-absent cases reaches 99% exact match, so the bottleneck lives in the categorical decision interface rather than in the arithmetic itself. A development-selected threshold on the model's returned scores raises rejection from 7% to 79% while keeping 97% answer-present accuracy, without retraining or extra inference. The takeaway is that selection accuracy alone is an incomplete reliability measure for decision workflows.","feed_headline":"Right answers 99%. Saying 'none': 7%.","feed_subtitle":"A pick-the-answer AI almost never rejects a wrong set, yet a score threshold lifts rejection to 79%.","key_machinery":"The central object is the paired coverage test: each case has a three-candidate menu with exactly one correct candidate (present) or none (absent), evaluated through three matched decision interfaces — Menu Choice, candidate-level T/F Choice, and native Boolean — that differ only in the API question type. The load-bearing mechanism is the score-based rejection rule $\\hat d_\\tau(x,\\mathcal C)=\\emptyset$ if $p_\\emptyset \\geq \\tau$, else $\\arg\\max_i p_i$, which thresholds the probability assigned to the None option before selecting a candidate. This rule converts the retained ordering signal in the scores into a usable abstention decision and, together with the interface contrast, isolates computation failures from decision-interface failures.","core_discovery":"On paired answer-present and answer-absent arithmetic problems, Jev's Menu Choice achieves 99% accuracy when a correct candidate exists but only 7% correct rejection when none does, a 92-point paired gap. The same pattern appears in purchase totals (94% versus 24%) and extends to inventory, time calculation, and capacity rounding. Supplying the computed result directly restores Menu Choice rejection to 100%, and native Boolean verification rejects all three candidates on 99% of answer-absent cases, demonstrating that the model can evaluate numerical correctness. Yet categorical candidate-level T/F Choice, with matched payloads, reaches only 27% on the same absent cases, so decomposition alone does not repair the failure: the decision interface is consequential. The returned scores still separate valid from invalid menus (arithmetic AUC of .985), and a rejection threshold selected on separate development problems raises arithmetic rejection from 7% to 79% while retaining 97% answer-present accuracy.","pith_inferences":["The P(NONE) threshold acts as a second-order abstention signal: read the paper as showing that the absent/present distinction is recoverable from confidence, not from the argmax choice, which suggests cost-sensitive deployments should tune τ on the selection-versus-rejection trade rather than trust the default.","The 7% figure is conditional on the near-distractor design ({-2, +1, +3}), and rejection climbs steeply with distractor distance; real pipelines with wider error margins should expect better rejection than 7%.","Extending the paper's own framing, an explicit none-of-the-above label is an output string, not a guarantee of abstention behavior, so any model offering such an option should be validated on paired answer-absent sets before deployment.","If the hosted API's P(NONE) scores prove stable across versions, the calibrated threshold might transfer across tasks to give calibration-free rejection; testing that transfer is the natural next step the paper leaves open."],"forward_implications":["Selection and rejection should be evaluated jointly in decision workflows, since strong selection accuracy alone masks a rejection failure.","Switching the interface from categorical Choice to native Boolean verification, rather than retraining, already restores near-perfect rejection on the same answer-absent arithmetic.","Thresholding the model's own returned scores repairs rejection from 7% to 79% with only a 2-point selection loss, and requires no additional inference.","A model can rank candidates near-perfectly (AUC .985) while still failing exact-match rejection at the default .5 threshold, so ranking quality does not imply decision quality."],"supporting_citations":[{"why":"Supplies the recommendation to include an 'other' or 'none-of-the-above' option whose adequacy the paper tests.","marker":"(TypeSafe AI, 2026a)"},{"why":"Documents Jev's mathematical limitations and Boolean/Choice inconsistencies that motivate the interface hypothesis.","marker":"(TypeSafe AI, 2026b)"},{"why":"Provides the generative LLM control that solves the same menus perfectly, showing the candidate construction permits rejection.","marker":"(Qwen Team, 2026)"},{"why":"Establishes the background pattern of degradation under none-of-the-above questions that this study extends to typed decision APIs.","marker":"(Tam et al., 2025)"},{"why":"Frames uncertainty-based abstention, which the threshold calibration repair draws on.","marker":"(Tomani et al., 2024)"}],"fun_headline_variants":["Right answer 99% of the time, but rejects none just 7%","AI that picks answers can't say 'none' when right one's missing","A score threshold lifts AI's rejection rate from 7% to 79%","Jev nails right answers but fails to reject wrong sets — fix found"],"cache_read_input_tokens":0,"weakest_assumption_plain":"Every quantitative claim rests on the hosted Jev API returning deterministic, version-stable outputs, so the observed 92-point gap between Menu Choice and Boolean verification reflects the model's decision behavior rather than request-level nondeterminism or a mid-study API update; the 7% figure is also conditional on the near-distractor candidate design the authors themselves document.","fun_headline_variants_meta":{"raw":{"variants":["Right answer 99% of the time, but rejects none just 7%","AI that picks answers can't say 'none' when right one's missing","A score threshold lifts AI's rejection rate from 7% to 79%","Jev nails right answers but fails to reject wrong sets — fix found"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"cost":0.000164248,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000164248,"upstream_inference_prompt_cost":0.000103488,"upstream_inference_completions_cost":6.076e-05},"total_tokens":1161,"prompt_tokens":944,"completion_tokens":217,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":129}},"tokens_in":944,"tokens_out":217,"cost_source":"openrouter_usage","duration_ms":2945,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":129,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-01T15:56:42.318904+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-query the hosted Jev API on the same 100 answer-absent arithmetic menus, verify that repeated requests return identical choices, and check whether the 7% correct-rejection figure reproduces exactly; separately, rebuild an absent menu with distractors far from the true answer (errors near 200 rather than 1–3) and observe whether Menu Choice rejection rises toward the calibrated 79% range, which would show that the near-distractor construction, rather than categorical rejection being impossible, drives the reported number.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}