{"id":"dc48e6e5-bca9-4947-a56b-7d76ea15a4ba","arxiv_id":"1908.05514","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MTMSN combines multi-type answer prediction, multi-span extraction, and arithmetic expression reranking to reach 79.9 F1 on DROP.","lead":"MTMSN is a neural reading comprehension model that predicts four answer types (span, count, arithmetic expression, negation) and can output several text spans at once. On the DROP benchmark, it raised the state of the art to 79.9 F1, about 13 points above a same-encoder baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negation type hard-codes a 100-base complement, so its +9 F1 contribution may not generalize to other-base negation questions; the paper provides no base-distribution analysis.","rationale":"The paper's central claim is the 79.88 F1 test score, which is backed by an official evaluation and released code; I found no reason to doubt it. The same-encoder NABERT baseline makes the comparison fair and the ablation design isolates components. The weakest point in the contribution is the negation answer type. Its annotation and inference rules hard-code the complement base to 100 (Section 3.5). The type delivers a large measured gain (roughly 9 F1 in Table 2) and nearly perfect accuracy on the 11.5% of dev examples where it is predicted (Table 5), but this only shows that the model exploits the 100-complement pattern. Whether DROP contains a material fraction of negation questions with other bases is unknown from the paper. If it does, the model cannot express those answers as negation and the component's generality is lower than the \"logical negation\" framing suggests. This concern is about scope and interpretation, not about the validity of the reported numbers, so the reader's CONDITIONAL verdict (rather than ACCEPT or REJECT) is appropriate.","tokens_in":12317,"tokens_out":24426,"duration_ms":217746,"concrete_test":"Analyze the DROP dev and train sets: for every question whose gold answer is a number, enumerate all numbers x in the passage and all constants C such that C - x equals the gold answer, focusing on C ≠ 100 (e.g., 50, 1000, explicitly stated totals). Tabulate the frequency of such questions, whether they have any alternative annotation (span/add-sub/count), and MTMSN's EM/F1 on them. If such questions are common and answered poorly, the 100-base negation assumption is a material coverage gap; if rare or handled by other types, the assumption is safe.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 fixes negation to the pattern \"100 minus a number\": the weak-supervision rule searches for numbers x with 100 - x equal to the gold answer, and inference outputs 100 minus the number with the highest negation probability. This is an inductive bias for percentage-complement questions (e.g., \"What percent are not X?\"), but it cannot express negations with other bases, such as a population total minus a subgroup count. Table 2 attributes about 9 F1 to the negation type, and Table 5 shows 11.5% of dev predictions are negation with 96.3 EM/F1. If a non-negligible fraction of DROP negation questions use a base other than 100, the model cannot produce their answers via this type; those questions must be solved by other components or are lost. The paper does not report the base distribution for negation questions, so the scope of the claimed \"logical negation\" contribution is unverified. This does not undermine the reported test score, but it qualifies the generality of the component and the interpretation of the ablation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MTMSN, a reading-comprehension model for the DROP benchmark that combines a multi-type answer predictor (span, arithmetic expression, count, and a new negation type), a multi-span extraction method based on a predicted span count and non-maximum suppression, and an arithmetic-expression reranking mechanism over beam-search candidates. The model is trained with weakly supervised annotation rules that search for spans, add/sub expressions, counts, and negations consistent with the gold answer. On the DROP development set, MTMSN-Large reaches 76.68 EM / 80.54 F1, and on the hidden test set 75.85 EM / 79.88 F1, which the paper reports as a new state of the art. Ablations attribute large gains to the add/sub and negation answer types, and smaller gains to multi-span extraction and expression reranking. The paper also provides a breakdown by answer type, an annotation-coverage analysis, and a 100-example error analysis.","tokens_in":12529,"tokens_out":5142,"duration_ms":51511,"significance":"If the results hold, this is a strong empirical contribution to discrete-reasoning reading comprehension: it demonstrates large gains over a same-encoder BERT baseline on the externally evaluated DROP benchmark, and the released source code and detailed ablations make the components reproducible and comparable. The multi-span extraction and expression-reranking ideas are useful and clearly evaluated. The main caveat is that the 'negation' answer type is implemented as a hard-coded 100-minus-number complement rule rather than as general logical negation, so the roughly 9 F1 ablation gain attributed to negation should be interpreted narrowly until its coverage is analyzed. The headline test score itself is an external benchmark result and is not circular.","major_comments":[{"comment":"The negation component is hard-coded to a base-100 complement: training annotations are generated only when 100 minus a mentioned number equals the gold answer (§3.5), and inference outputs 100 minus the number with the largest negation probability (§3.5). The ablation in Table 2 removes negation and loses roughly 9 F1 (69.5% relative for the large model), and Table 5 shows that 11.5% of dev predictions use the negation type with 96.3 EM/F1. However, the paper does not report the distribution of complement bases among DROP negation questions, so it is unclear whether this gain reflects a general logical-negation capability or a narrow percentage-complement heuristic. This is not merely a presentation issue: the error analysis in Table 7 includes a 'not born in the UK' question whose answer requires subtracting a subgroup count from a total, and that example is categorized as arithmetic computation, not negation, confirming the limitation. Please either generalize the negation rule to use an arbitrary base number from the passage, or explicitly reframe the contribution as a percentage-complement predictor and report how many negation-type questions in DROP are actually base-100 complements.","section":"§3.2, §3.5, Tables 2 and 5"},{"comment":"There is a discrepancy between the training annotation limit and the inference setting for arithmetic expressions. Section 3.5 states that the search for addition/subtraction annotations considers 'three numbers at most,' while Section 4.1 sets the maximum number of signed numbers M to 4, and Figure 4 varies M from 3 to 6 during decoding. Since the reranker is trained on labels produced by the model's own beam search, an expression with four signed numbers can never match a gold annotation under the three-number training limit, so all such candidates are labeled wrong. This inconsistency affects reproducibility and the interpretation of the reranking experiments; please clarify whether the three-number limit applies only to gold-annotation search and whether M is intended as an inference-only parameter.","section":"§3.5 vs. §4.1"}],"minor_comments":[{"comment":"The main results and several ablations are reported for a single run without variance or significance tests. For large-margin comparisons this is acceptable, but small differences such as the reranking gain of 1.8 F1 and the gated-span gain of 0.8 F1 in Tables 2 and 3 should be interpreted cautiously; please state whether the reported numbers are from a single seed or averaged.","section":"§4.2 and §4.3"},{"comment":"In Algorithm 1, the span count is computed as t = arg max pspan + 1, but the label range for pspan is not defined in the text. Please clarify whether pspan predicts the number of spans minus one, and state the maximum value used in training.","section":"Algorithm 1"},{"comment":"The counting class bound is set to 10 without discussion of how frequently DROP counts exceed 10; a sentence noting the coverage of this bound would help readers assess the count component's scope.","section":"§4.1"},{"comment":"There is a typo in the Related Work section: 'Morevoer' should be 'Moreover'.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with an externally validated headline result and reproducible code. The main revision need is to address the base-100 negation limitation: either extend the component or narrow the claim and support it with a base-distribution analysis. The M discrepancy between training and inference should also be resolved. The paper's novelty relative to prior DROP work is clear, and I see no circularity concern with the benchmark evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it last night. The headline holds up: MTMSN is a genuine step forward on DROP, and the 79.9 F1 test number is credible. The new pieces are the predicted multi-span count with NMS, the negation answer type, and the arithmetic-expression reranking; the same-encoder BERT baseline makes the gains attributable. The ablations are clean and the code is out. Nothing in the paper looks cooked.\n\nThe main soft spot is the negation component, which is narrower than the word \"negation\" suggests. Training and inference hard-code the complement base to 100: the weak supervision searches for x such that 100 - x = answer, and inference outputs 100 minus the argmax number. The paper reports 11.5% of dev predictions are negation at 96.3 F1, and ablation gives it roughly 9 F1. If a material fraction of DROP negation questions use another base (population totals, subgroup counts), this component cannot produce those answers; the paper gives no base-distribution analysis. That doesn't refute the reported score, but it does mean the generality claim \"logical negation\" is overbroad. I'd ask for a base breakdown or a softening of that claim.\n\nSome minor points: no error bars or multiple seeds, hyperparameters (max spans, beam size, M) are tuned on dev, and the reranking training uses the model's own beam candidates as labels—reasonable in practice but worth noting for selection effects. None of these threaten the main result.\n\nThis is for people working on DROP-style reading comprehension or QA systems that must emit counts, numbers, and multiple spans. It is not a conceptual advance, but it is a well-executed empirical recipe with a fair baseline and honest ablations.\n\nSend it to review. A serious referee should engage; the negation base issue is fixable and the central result holds.","headline":"MTMSN's 79.9 F1 on DROP is credible and the whole is a solid empirical package; the negation component, while effective on the benchmark, hard-codes a 100-base complement that may not generalize.","tokens_in":13075,"tokens_out":1440,"would_cite":true,"duration_ms":14155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a reading-comprehension model can handle discrete reasoning by combining four answer types with multi-span extraction, reaching 79.9 F1 on DROP's hidden test set.","keywords":["reading comprehension","discrete reasoning","multi-span extraction","logical negation","arithmetic expression reranking","DROP dataset","weak supervision","pre-trained transformers"],"falsifier":"Measure on a held-out sample of DROP how many gold answers are complements of a passage number with a base other than 100, or where the intended operation is a subtraction of the number from a total different from 100; if that fraction is substantial, the negation component's base-100 assumption is violated and its measured gain should shrink accordingly.","tokens_in":12109,"feed_emoji":"🧠","tokens_out":7730,"duration_ms":64664,"temperature":0.7,"pith_summary":"This paper tries to establish that a reading-comprehension system can handle questions requiring discrete reasoning by making answer structure a first-class modeling target rather than relying on one span prediction. Its Multi-Type Multi-Span Network (MTMSN) combines four answer types (span, arithmetic expression, count, and a newly added logical negation), dynamically extracts one or several non-overlapping spans, and reranks candidate arithmetic expressions. On the DROP benchmark, MTMSN with a large pre-trained transformer encoder reports 79.9 F1 on the hidden test set and 80.54 F1 on the development set, which the paper counts as new state-of-the-art results. If this is right, the implication is that explicit type and span-number prediction, plus expression reranking, are the components that buy most of the numerical-reasoning performance on this task.","feed_headline":"Reading model hits 79.9 F1 on DROP via negation and multi-span","feed_subtitle":"Adding logical negation, multi-span extraction, and expression reranking beats the prior best by 32.9 F1 points.","key_machinery":"The load-bearing object is the multi-type answer predictor sitting on top of a pre-trained transformer encoder. It computes four probability distributions: start/end positions for spans, plus/minus/zero signs for every number in the passage, a count class, and a negation flag per number; it also predicts the number of spans and uses non-maximum suppression to output that many non-overlapping spans. Arithmetic expression candidates from beam search are embedded by summing number and sign vectors and re-ranked with a feed-forward scorer, so the same network both generates and verifies discrete operations.","core_discovery":"The central claim is that a single neural reading-comprehension architecture can cover most DROP answers by predicting not just text spans but the kind of answer being produced. On top of BERT contextual representations, MTMSN learns a type classifier over span, addition/subtraction, count, and negation; for each type it applies a dedicated predictor, including sign assignment to every number in the passage for arithmetic expressions and a binary negation decision per number. A separate head predicts how many text spans should be returned, and non-maximum suppression removes overlapping candidates until that number is reached. Candidate arithmetic expressions are decoded with beam search and then re-scored by a reranker that reads the expression's context, and training uses a weakly supervised marginal-likelihood objective over all annotations consistent with the gold answer. The reported results are 75.85 EM and 79.88 F1 on the DROP test set, with ablations attributing the largest losses to removing addition/subtraction, negation, count, multi-span extraction, and expression reranking.","pith_inferences":["Extension the authors leave implicit: the hard-coded complement base of 100 could be replaced by a learned or selected base, which would preserve the negation gain on questions whose complement is not 100.","Extension the authors leave implicit: the span-amount head plus overlap pruning is a generic recipe for turning any single-span extractor into a set predictor, applicable to table and multi-hop QA.","Extension the authors leave implicit: because the error analysis finds 18% of errors are sorting, a sorting-aware head or a program-like decoder is the natural next type to add."],"forward_implications":["Removing the addition/subtraction predictor lowers dev F1 by more than 20 points for both base and large models, so explicit arithmetic sign prediction is the single most load-bearing component for this task.","Adding negation as an answer type contributes roughly 9 F1 points, showing that many DROP questions are answered by complementing a number rather than by extracting or adding.","Predicting the number of answer spans and pruning overlaps raises F1 more than EM, because partially matching a multi-span answer is easier than exactly matching the full set.","Reranking arithmetic expressions with context gives about 1.8 F1 over taking the single highest-probability expression, so verifying candidates helps filter wrong-but-confident arithmetic predictions.","Using the same pretrained encoder, MTMSN outperforms the augmented-BERT baseline by 12.07 EM and 13.19 F1 on the dev set, meaning the gains are due to the answer-structure components rather than the encoder."],"supporting_citations":[{"why":"Supplies the DROP benchmark, the prior NAQANet multi-type predictor baseline, and the weakly supervised annotation search over arithmetic expressions that MTMSN extends.","marker":"Dua et al., 2019"},{"why":"Provides the pre-trained BERT transformer encoder whose contextual representations all answer-type predictors are built on.","marker":"Devlin et al., 2019"},{"why":"Provides the standard start/end span decoding strategy that the multi-span extraction component generalizes.","marker":"Seo et al., 2017"},{"why":"Supplies the gating mechanism used in MTMSN's span prediction.","marker":"Wang et al., 2017"},{"why":"Defines the Transformer block architecture that makes up the encoder.","marker":"Vaswani et al., 2017"},{"why":"Provides the non-maximum suppression algorithm used to prune overlapping candidate spans.","marker":"Rosenfeld and Thurston, 1971"},{"why":"Supplies the marginal likelihood objective used to train over multiple weakly supervised answer annotations.","marker":"Clark and Gardner, 2018"}],"fun_headline_variants":["MTMSN: neural reader hits 79.9 F1 on DROP with multi-span answers","Multi-span, count, negation, arithmetic: MTMSN tops DROP leaderboard","New neural model handles discrete reasoning, achieves 79.9 F1 on DROP","MTMSN: one model for span, count, negation, and arithmetic questions","Weakly supervised multi-type reader: DROP SOTA 79.9 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negation answer type depends on the rule that every negation question's answer equals 100 minus one number mentioned in the passage.","fun_headline_variants_meta":{"raw":{"variants":["MTMSN: neural reader hits 79.9 F1 on DROP with multi-span answers","Multi-span, count, negation, arithmetic: MTMSN tops DROP leaderboard","New neural model handles discrete reasoning, achieves 79.9 F1 on DROP","MTMSN: one model for span, count, negation, and arithmetic questions","Weakly supervised multi-type reader: DROP SOTA 79.9 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1616,"prompt_tokens":941,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":557,"tokens_out":675,"duration_ms":5596,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:11:11.888483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure on a held-out sample of DROP how many gold answers are complements of a passage number with a base other than 100, or where the intended operation is a subtraction of the number from a total different from 100; if that fraction is substantial, the negation component's base-100 assumption is violated and its measured gain should shrink accordingly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DROP benchmark, the prior NAQANet multi-type predictor baseline, and the weakly supervised annotation search over arithmetic expressions that MTMSN extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard start/end span decoding strategy that the multi-span extraction component generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the gating mechanism used in MTMSN's span prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the non-maximum suppression algorithm used to prune overlapping candidate spans."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the marginal likelihood objective used to train over multiple weakly supervised answer annotations."}],"review_version":1}