{"id":"be6963d2-b8a4-4c4d-b858-2f90cb876d46","arxiv_id":"2608.12836","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A decomposition-plus-ILP framework with contrastive atomic scoring and relative calibration substantially improves LLM accuracy on compound answer options, with the largest gains on NEITHER/NOR.","lead":"Large language models struggle when multiple-choice answers contain logical operators like AND, OR, or NEITHER/NOR, even when they judge the individual parts correctly. This paper introduces a framework that scores each atomic answer separately, then uses an integer linear program to combine the scores under the operator's constraints, improving Macro-F1 from about 48 to 77 on two benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central evidence pipeline rests on the unverified fidelity of LLM-generated contrastive hypotheses; a templated-hypothesis control would settle whether the reported gains depend on paraphrases that preserve atomic meaning.","rationale":"Reader's weakest assumption is the same one I would defend as most load-bearing. I considered a second candidate: the absence of a decomposition-without-ILP baseline. That is a real attribution gap for the specific claim that ILP, rather than decomposition alone, drives the gains, and it is part of the reader's conditional rationale. I do not elevate it to the primary concern because the paper's contribution is the full pipeline and the empirical comparison against direct prompting is still meaningful. In contrast, if the generated h+/h- are not faithful, the atomic scores in Equation (1) are not evidence about the gold-labeled atom, and the calibration layer cannot detect per-instance drift. The paper's own Limitations section concedes that errors in atomic evidence propagate; hypothesis drift is a mechanism for such errors that is not measured. The proposed templated-hypothesis control is cheap because the code and data are released. Independent support that should not be discounted includes the exact ILP constraints, the gold-atomic ceiling of 1.00, the calibration improvements, and the operator-conditional pattern of gains. The concern therefore does not overturn the reader's CONDITIONAL verdict; it sharpens the condition: report a fidelity control on hypothesis generation, or restrict the claim to model-authored paraphrases.","tokens_in":22573,"tokens_out":15067,"duration_ms":155944,"concrete_test":"Sample 200 instances per benchmark (stratified by operator) from the released test splits. For each unique atom, replace the LLM-generated h+/h- with two literal templates that instantiate the atomic string verbatim, e.g., H+: 'The statement \"<atomic>\" is supported by the context/question.' and H-: 'The statement \"<atomic>\" is not supported by the context/question.' Re-run the full pipeline (paired multiple-choice, relative calibration, ILP) with the same seeds and report atomic accuracy and compound Macro-F1. If the templated-hypothesis numbers stay within about 2 points of Tables 1 and 2, the drift concern is settled; if they drop substantially, the headline results depend on unvalidated model-authored paraphrases and should be re-reported with a fidelity check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 constructs h+/h- with the same LLM that later scores them. The prompt in Appendix E.1 says to use the atomic statement exactly as written, and the E.6 demonstrations are faithful, but nothing in the pipeline verifies that the produced h+ is logically equivalent to 'a satisfies C' on each instance. Equation (1) then reduces all atomic evidence to the relative token probability of the two model-authored paraphrases. Calibration (Section 3.5) is fit against gold atomic statuses on training instances; it can correct a stationary paraphrase bias, but it cannot repair per-instance semantic drift, for example an h+ that drops a modifier or reinterprets an open-ended question. Since the ILP in Section 3.6 only rearranges these already-corrupted scores, the entire 48.3 to 77.0 and 47.0 to 75.6 claim inherits the unverified equivalence. The comparison to direct prompting also becomes asymmetric: direct prompting shows the real atom inside the compound option, while structured inference may score a paraphrase of it. This is the weakest load-bearing link because the atomic evidence is the only content the logical layer receives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies multiple-choice questions whose answer options are compounds of two atomic answers joined by AND, OR, or NEITHER/NOR. It proposes a structured framework that deterministically decomposes each option into atoms, uses the same LLM to generate positive and negative contrastive hypotheses for each atom, scores them with paired multiple-choice log-probabilities, calibrates those scores with Platt, isotonic, or a new relative calibration method, and finally combines them with an operator-constrained integer linear program that enforces the truth tables of the operators and selects exactly one option. The framework is evaluated with Llama-3.1-8B-Instruct on LOGICAL-COMMONSENSEQA and on a new benchmark, LOGICAL-SATA, constructed from SATA-Bench. The reported macro-F1 gains over direct prompting are large, especially on NEITHER/NOR (e.g., 14.0 to 76.8 on LOGICAL-COMMONSENSEQA-HV with relative calibration).","tokens_in":22862,"tokens_out":4747,"duration_ms":49939,"significance":"If the main results hold, the paper makes a useful contribution by demonstrating that separating atomic evidence elicitation from logical composition can substantially reduce the compositionality gap on compound answer options, and by providing a new benchmark, LOGICAL-SATA, in this format. The framework is clearly described, the ILP constraints are exact encodings of the operator semantics, the code and datasets are promised publicly, and the comparison across three calibration schemes is systematic. The strength of the empirical claim, however, rests on two assumptions that are not directly tested: that the LLM-generated contrastive hypotheses faithfully preserve the meaning of each atomic answer and its negation, and that the ILP, rather than the decomposition or contrastive scoring alone, is responsible for the reported gains. These are load-bearing because the atomic evidence is the only information that reaches the logical composition layer.","major_comments":[{"comment":"The contrastive hypotheses h+ and h- are generated by the same LLM that later scores them, and the pipeline does not verify that the produced h+ is logically equivalent to 'a satisfies C' or that h- is its negation. The prompts in Appendix E.1 instruct the model to use the atomic statement exactly as written and the demonstrations in E.6 are faithful, but nothing checks per-instance equivalence; a generated h+ that drops a modifier or reinterprets an open-ended question would make the elicited scores evidence for a different proposition than the gold-labeled atom. Calibration (Section 3.5) can correct a stationary bias but cannot repair per-instance semantic drift, and the ILP in Section 3.6 only rearranges these already-corrupted scores. Because this assumption is load-bearing for the 48.3-to-77.0 and 47.0-to-75.6 claims, I ask for a control that uses templated or otherwise verified hypotheses (or an NLI/verification step on a sample of instances) to establish that the gains do not depend on paraphrase fidelity.","section":"Section 3.3, Eq. (1)"},{"comment":"The main comparison, direct prompting versus structured inference, changes three components at once: option decomposition, contrastive scoring, and ILP-constrained composition. There is no ablation that keeps the atomic evidence fixed and replaces the ILP with a natural-language composition step or with a simple deterministic rule-based combiner over the same calibrated scores. As a result, the paper does not establish that the operator-constrained ILP itself contributes to the observed improvement; the gains could come entirely from decomposition and contrastive scoring. Since the title and introduction emphasize 'operator-constrained ILP inference' as a central contribution, this ablation is needed to support the attribution of the reported gains to the ILP.","section":"Section 5.1, Tables 1 and 2"},{"comment":"Most structured-inference rows report a standard deviation of exactly 0.0, while direct-prompting rows show nonzero standard deviations. The paper states that all experiments use temperature 0.7 but does not explain why the structured-inference results have zero variance; presumably log-probability extraction is deterministic whereas direct prompting uses stochastic generation. This should be stated explicitly in the experimental settings, because a reader could otherwise suspect that the 0.0 values reflect an error in the averaging or in the random seeds.","section":"Section 5.1 and Tables 1-6"}],"minor_comments":[{"comment":"The sentence 'paired multiple-choice structured inference raises these to 75.1, and 71.9, respectively, and relative to 76.8 and 73.4' is missing a verb before 'relative' and should be rewritten, for example as 'and relative calibration raises them to 76.8 and 73.4.'","section":"Section 5.1, Paragraph on operator-level gains"},{"comment":"The feature vector for relative calibration includes rankC(a), but the ranking convention (ascending versus descending order, and how ties are broken) is not specified; please add this detail to Appendix A or to the main text.","section":"Section 3.5"},{"comment":"The representative demonstrations show only positive examples (Option A correct) for both benchmarks; adding a negative example or stating how demonstrations are balanced across labels would clarify whether the few-shot demonstrations could bias the model toward selecting A.","section":"Appendix E.6"},{"comment":"The appendix says all experiments use a random seed of 42 but does not describe how the five runs differ; please specify whether only decoding randomness varies or whether other components (calibration set subsampling, demonstration selection) also vary across runs.","section":"Appendix A"},{"comment":"For LOGICAL-SATA, the paper reports that 1,390 eligible source questions are reduced to 1,350, but the selection criterion for dropping the remaining 40 questions is not described; please state the criterion or note that they were dropped by random selection.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is interesting. My main concerns are experimental rather than conceptual: the missing control for hypothesis fidelity and the missing ILP ablation both affect how the central claim can be interpreted. The fact that LOGICAL-COMMONSENSEQA is the authors' own benchmark is not itself a problem, but it would be reassuring if the authors acknowledged any possible shared construction artifacts with the framework and if at least one of the two benchmark evaluations used an independently created benchmark beyond LOGICAL-SATA. I would support a revised version that adds the two requested controls and clarifies the variance reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper, read it in an afternoon. It splits each compound answer option into atomic statements, scores contrastive positive/negative hypotheses for each atom, calibrates the scores, and lets an ILP enforce the operator semantics. On Llama-8B the method lifts Macro-F1 from 48.3 to 77.0 on LOGICAL-COMMONSENSEQA-HV and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR. The effect sizes are large and the operator-wise pattern matches the paper's composition-bottleneck story.\n\nWhat's genuinely new is the package: contrastive atomic scoring plus operator-constrained ILP is a coherent synthesis of known pieces, and relative calibration—adding within-instance rank and margin features—is a reasonable tweak that pays off most in MIXED settings. LOGICAL-SATA, derived from SATA-Bench, looks like a reusable resource. Code and data are public. The paper is also honest: the limitations section names the single model, exact-one constraint, and prompt sensitivity, and the error analysis is more substantive than usual.\n\nThe soft spot is exactly the one the stress test flags. The h+ and h- hypotheses are written by the same model that later scores them, and the pipeline never checks that they preserve the atomic statement's meaning. The prompt says to use the atom exactly as written and the demonstrations look faithful, but that is not a guarantee. A paraphrase that drops a modifier or broadens an open-ended question changes the proposition under evaluation; calibration only corrects a stationary bias, not per-instance drift. Since the ILP rearranges these scores, the headline numbers inherit this unverified step. A control using templated hypotheses (e.g., 'The atom is true/false given the context') would settle it. I don't think this is fatal: the NEITHER/NOR jumps are far too large to be pure paraphrase artifacts, and direct prompting has its own asymmetries. But it is the load-bearing assumption and the paper should address it.\n\nTwo smaller concerns. There is no ablation replacing the ILP with a natural-language composition step over the same atomic judgments, so part of the gain could be from decomposition alone rather than enforced constraints. And results are for a single model family, which the authors acknowledge. No significance tests, but the effect sizes make that minor.\n\nThis paper deserves a serious referee. The fix list is short: templated-hypothesis control, a NL-composition ablation, and ideally a second model. I'd send it out.","headline":"A clean decomposition-plus-ILP paper with large gains on NEITHER/NOR, but the unverified fidelity of model-authored contrastive hypotheses is the main load-bearing weakness.","tokens_in":23304,"tokens_out":3240,"would_cite":true,"duration_ms":28054,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can judge the atoms of a compound answer yet fail to combine them; scoring each atom separately and letting a constraint solver do the composition lifts Macro-F1 from 48 to 77.","keywords":["compound answer options","logical composition","integer linear programming","contrastive hypothesis scoring","relative calibration","compositionality gap","NEITHER/NOR reasoning","LLM logical reasoning"],"falsifier":"Run the constrained ILP against an unconstrained aggregation that scores each option by a fixed combination of its atoms' calibrated scores (minimum for AND, maximum for OR, negative-product for NEITHER/NOR) and takes the argmax: if the unconstrained version matches the reported 77.0 and 75.6 Macro-F1, the operator constraints are not what produce the gains. In parallel, sample the generated contrastive hypotheses and have annotators check whether each $h^+$ and $h^-$ is logically equivalent to 'the atom holds' and 'the atom does not hold' under the context, because systematic paraphrase drift would mean the scores reward propositions the benchmark never labeled.","tokens_in":22417,"feed_emoji":"🧩","tokens_out":12939,"duration_ms":114893,"temperature":0.7,"pith_summary":"Large language models often answer multiple-choice questions correctly when the options are simple, yet stumble when the same choice is phrased as two atomic answers joined by AND, OR, or NEITHER/NOR, even when they judge each atom correctly on its own. This paper tries to show that the failure is a composition problem, not a knowledge problem, and that it can be repaired by architecture rather than by more prompting. The proposed framework hides compound options from the model entirely: it splits each option into atoms, elicits contrasting positive and negative judgments about each atom, and hands the calibrated scores to an integer linear program whose constraints encode the operator truth tables and the requirement that exactly one option be valid. The balanced F1 score (Macro-F1) rises from 48.3 to 77.0 on the commonsense benchmark LOGICAL-COMMONSENSEQA and from 47.0 to 75.6 on LOGICAL-SATA, a new reading-comprehension benchmark the paper constructs from SATA-Bench, with NEITHER/NOR — the operator where direct prompting nearly collapses — recovering from about 14 to roughly 77. If the argument holds, part of LLM logical-reasoning failure is a fixable wiring problem in how local judgments are combined, and the same recipe could apply to implication, exclusive or, and longer compounds.","feed_headline":"Judge atoms first, combine later: LLM logic scores reach 77.","feed_subtitle":"Decomposing options into atoms and constraining the combination with an ILP lifted Macro-F1 from 48 to 77.","key_machinery":"The carrying mechanism is an operator-constrained integer linear program (ILP, a solver that maximizes a linear objective subject to linear inequalities over binary variables) acting on atomic status variables, fed by paired contrastive hypothesis scoring and an instance-relative calibration. Each unique atomic answer gets one binary variable $y_a$ and each compound option one variable $x_i$; linear inequalities ($x_i \\leq y_1$, $x_i \\leq y_2$, $x_i \\geq y_1 + y_2 - 1$ for AND; the mirror set for OR; the complemented set $x_i \\leq 1 - y_1$, $x_i \\leq 1 - y_2$, $x_i \\geq 1 - y_1 - y_2$ for NEITHER/NOR) make it structurally impossible to mark an option valid unless its atoms satisfy the operator, and the constraint $\\sum_i x_i = 1$ forces exactly one choice. The objective maximizes the total calibrated evidence $\\sum_{a \\in U_C} [s^+_C(a) y_a + s^-_C(a) (1 - y_a)]$, so the solver never generates or paraphrases a judgment — it only selects. The evidence itself comes from a single prompt offering the positive and negative hypotheses as options A and B and normalizing the log probabilities of the first answer tokens, and the new relative calibration adds within-instance features (standardized score, rank among the atoms, gap to the instance maximum) to a logistic mapping, which matters most in MIXED instances where options simultaneously demand accepting and rejecting atoms.","core_discovery":"The paper's central claim is that the compositionality gap in compound answer reasoning is real, separable, and repairable: a model that is handed the individual atomic answers and asked to judge each one in isolation retains far more usable evidence than its performance on the compound option suggests. The authors demonstrate this by decomposing every candidate option into two atomic answers and an explicit operator, collecting positive and negative hypotheses for each atom, normalizing the model's token-level preference between the two into a score in $[0,1]$, calibrating those scores (including a new relative calibration that also encodes each atom's rank and standing within the instance), and then solving an integer linear program whose linear inequalities enforce the exact semantics of AND, OR, and NEITHER/NOR while selecting exactly one option. On Llama-3.1-8B-Instruct the framework lifts Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR (14.0 to 76.8; 12.6 to 73.4), and with near-exact recovery once the gold atomic statuses are supplied to the solver. The conclusion a sympathetic reader should take is that LLM logical failures on these tasks are largely failures of combining local judgments, not of knowing the facts.","pith_inferences":["A direct extension the paper leaves implicit: the framework's exact-solver guarantee becomes an upper bound on what a model's atomic evidence can support, so the residual gap between atomic accuracy (about 0.83) and compound accuracy (about 0.76) measures how much the operators themselves demand of the local judgments; one could use it to compare models by how much of the composition their local e","Because all results come from one 8-billion-parameter instruction-tuned model, the claim that composition is a distinct burden would be tested by a cross-model replication: if a stronger or differently trained checkpoint shows no NEITHER/NOR collapse under direct prompting, the 'distinct burden' would be model-specific rather than a general property of LLM reasoning.","The operator-graded difficulty ordering (AND easiest, NEITHER/NOR hardest) matches the mental-model literature the paper cites; applying the framework to implication, exclusive disjunction, or nested expressions would indicate whether that ordering generalizes beyond the three operators studied."],"forward_implications":["If the framework's results hold, the same model that scored 14 to 48 under direct prompting demonstrably carries the atomic evidence needed to reach roughly 77, so its logical failure is in combination, not in knowing or judging the atoms.","Because the ILP encodes operator semantics exactly, the framework cannot be coaxed or generated into violating the logic; every remaining compound error traces to a wrong atomic judgment, turning logical evaluation into a diagnostic of atomic evidence.","The recovery pattern — small gains on AND, large on OR, largest on NEITHER/NOR and MIXED — indicates that explicit composition pays off exactly where the possibility structure humans find hardest also defeats the model.","The same decomposition-and-constrain recipe transfers across evidence types, commonsense plausibility and passage-grounded reading comprehension, suggesting the compositionality gap is not an artifact of one benchmark or one kind of knowledge.","Atoms shared across options are scored once, so the framework guarantees a single consistent judgment for a proposition wherever it appears, something unconstrained generation cannot promise."],"supporting_citations":[{"why":"Supplies the target commonsense benchmark LOGICAL-COMMONSENSEQA and the observed operator-graded difficulty that direct prompting must be compared against.","marker":"(Junias and Pacheco, 2026)"},{"why":"Provides the human-labeled select-all-that-apply SATA-Bench data from which the new LOGICAL-SATA benchmark is constructed.","marker":"(Xu et al., 2025)"},{"why":"Defines the compositionality gap that the paper identifies as the problem its framework is designed to close.","marker":"(Press et al., 2023)"},{"why":"Supplies the prompt-based structured prediction template that the confidence elicitation and calibration stages follow.","marker":"(Pauk and Pacheco, 2026)"},{"why":"Motivates combining prompted local predictions under global constraints, the inference paradigm the ILP layer instantiates.","marker":"(Mehta et al., 2024)"},{"why":"Establishes the precedent of ILP-based combination of per-option comprehension scores with relations between answer choices in question answering.","marker":"(Pujari and Goldwasser, 2019)"},{"why":"Supports the choice of paired contrastive judgments over isolated pointwise scores as the more reliable evidence signal.","marker":"(Liusie et al., 2024)"},{"why":"Identifies Llama-3.1-8B-Instruct, the single model on which all reported experiments and the central empirical claims rest.","marker":"(Dubey et al., 2024)"}],"fun_headline_variants":["Judge atoms, not options: LLM logic jumps to 77","Decompose, then combine: LLM logic reaches 77","Atom-wise scoring with ILP lifts LLM logic to 77","48 to 77: Atom-first logic beats compound reasoning","LLM logic: Break options into atoms, combine with ILP — 77 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework trusts that the contrastive hypotheses the model writes for each atomic answer faithfully restate that answer and its negation under the question, so if the model's paraphrases drift, the elicited scores become evidence about a different proposition than the one the benchmark labeled, and every downstream step inherits the drift.","fun_headline_variants_meta":{"raw":{"variants":["Judge atoms, not options: LLM logic jumps to 77","Decompose, then combine: LLM logic reaches 77","Atom-wise scoring with ILP lifts LLM logic to 77","48 to 77: Atom-first logic beats compound reasoning","LLM logic: Break options into atoms, combine with ILP — 77 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":3017,"prompt_tokens":994,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1929}},"tokens_in":610,"tokens_out":2023,"duration_ms":14283,"temperature":1.0,"reasoning_tokens":1929,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:17:18.354014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the constrained ILP against an unconstrained aggregation that scores each option by a fixed combination of its atoms' calibrated scores (minimum for AND, maximum for OR, negative-product for NEITHER/NOR) and takes the argmax: if the unconstrained version matches the reported 77.0 and 75.6 Macro-F1, the operator constraints are not what produce the gains. In parallel, sample the generated contrastive hypotheses and have annotators check whether each $h^+$ and $h^-$ is logically equivalent to 'the atom holds' and 'the atom does not hold' under the context, because systematic paraphrase drift would mean the scores reward propositions the benchmark never labeled.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the target commonsense benchmark LOGICAL-COMMONSENSEQA and the observed operator-graded difficulty that direct prompting must be compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-based structured prediction template that the confidence elicitation and calibration stages follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates combining prompted local predictions under global constraints, the inference paradigm the ILP layer instantiates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the precedent of ILP-based combination of per-option comprehension scores with relations between answer choices in question answering."},{"cited_title":"Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others","cited_arxiv_id":null,"evidence_quote":"Identifies Llama-3.1-8B-Instruct, the single model on which all reported experiments and the central empirical claims rest."}],"review_version":1}