Pith. sign in

REVIEW 2 major objections 4 minor 35 references

A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that a reading-comprehension model can handle discrete reasoning by combining four answer types with multi-span extraction, reaching 79.9 F1 on DROP's hidden test set.

desk verdict MTMSN's 79.9 F1 on DROP is credible and the whole is a solid empirical package; the negation component, while effective on the benchmark, hard-codes a 100-base complement that may not generalize. read the letter →

arxiv 1908.05514 v2 pith:MU4YLIXQ submitted 2019-08-15 cs.CL

Minghao Hu , Yuxing Peng , Zhen Huang , Dongsheng Li This is my paper · ORCID
classification cs.CL
keywords readingcomprehensiondiscretereasoningmulti-spanextractionlogicalnegationarithmeticexpressionrerankingDROPdatasetweaksupervisionpre-trainedtransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a reading-comprehension system can handle questions requiring discrete reasoning by making answer structure a first-class modeling target rather than relying on one span prediction. Its Multi-Type Multi-Span Network (MTMSN) combines four answer types (span, arithmetic expression, count, and a newly added logical negation), dynamically extracts one or several non-overlapping spans, and reranks candidate arithmetic expressions. On the DROP benchmark, MTMSN with a large pre-trained transformer encoder reports 79.9 F1 on the hidden test set and 80.54 F1 on the development set, which the paper counts as new state-of-the-art results. If this is right, the implication is that explicit type and span-number prediction, plus expression reranking, are the components that buy most of the numerical-reasoning performance on this task.

What carries the argument

The load-bearing object is the multi-type answer predictor sitting on top of a pre-trained transformer encoder. It computes four probability distributions: start/end positions for spans, plus/minus/zero signs for every number in the passage, a count class, and a negation flag per number; it also predicts the number of spans and uses non-maximum suppression to output that many non-overlapping spans. Arithmetic expression candidates from beam search are embedded by summing number and sign vectors and re-ranked with a feed-forward scorer, so the same network both generates and verifies discrete operations.

What would settle it

Measure on a held-out sample of DROP how many gold answers are complements of a passage number with a base other than 100, or where the intended operation is a subtraction of the number from a total different from 100; if that fraction is substantial, the negation component's base-100 assumption is violated and its measured gain should shrink accordingly.

Watch

Extended reading notes

Core claim

The central claim is that a single neural reading-comprehension architecture can cover most DROP answers by predicting not just text spans but the kind of answer being produced. On top of BERT contextual representations, MTMSN learns a type classifier over span, addition/subtraction, count, and negation; for each type it applies a dedicated predictor, including sign assignment to every number in the passage for arithmetic expressions and a binary negation decision per number. A separate head predicts how many text spans should be returned, and non-maximum suppression removes overlapping candidates until that number is reached. Candidate arithmetic expressions are decoded with beam search and then re-scored by a reranker that reads the expression's context, and training uses a weakly supervised marginal-likelihood objective over all annotations consistent with the gold answer. The reported results are 75.85 EM and 79.88 F1 on the DROP test set, with ablations attributing the largest losses to removing addition/subtraction, negation, count, multi-span extraction, and expression reranking.

Load-bearing premise

The negation answer type depends on the rule that every negation question's answer equals 100 minus one number mentioned in the passage.

Editorial extensions

If this is right

  • Removing the addition/subtraction predictor lowers dev F1 by more than 20 points for both base and large models, so explicit arithmetic sign prediction is the single most load-bearing component for this task.
  • Adding negation as an answer type contributes roughly 9 F1 points, showing that many DROP questions are answered by complementing a number rather than by extracting or adding.
  • Predicting the number of answer spans and pruning overlaps raises F1 more than EM, because partially matching a multi-span answer is easier than exactly matching the full set.
  • Reranking arithmetic expressions with context gives about 1.8 F1 over taking the single highest-probability expression, so verifying candidates helps filter wrong-but-confident arithmetic predictions.
  • Using the same pretrained encoder, MTMSN outperforms the augmented-BERT baseline by 12.07 EM and 13.19 F1 on the dev set, meaning the gains are due to the answer-structure components rather than the encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension the authors leave implicit: the hard-coded complement base of 100 could be replaced by a learned or selected base, which would preserve the negation gain on questions whose complement is not 100.
  • Extension the authors leave implicit: the span-amount head plus overlap pruning is a generic recipe for turning any single-span extractor into a set predictor, applicable to table and multi-hop QA.
  • Extension the authors leave implicit: because the error analysis finds 18% of errors are sorting, a sorting-aware head or a program-like decoder is the natural next type to add.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces MTMSN, a reading-comprehension model for the DROP benchmark that combines a multi-type answer predictor (span, arithmetic expression, count, and a new negation type), a multi-span extraction method based on a predicted span count and non-maximum suppression, and an arithmetic-expression reranking mechanism over beam-search candidates. The model is trained with weakly supervised annotation rules that search for spans, add/sub expressions, counts, and negations consistent with the gold answer. On the DROP development set, MTMSN-Large reaches 76.68 EM / 80.54 F1, and on the hidden test set 75.85 EM / 79.88 F1, which the paper reports as a new state of the art. Ablations attribute large gains to the add/sub and negation answer types, and smaller gains to multi-span extraction and expression reranking. The paper also provides a breakdown by answer type, an annotation-coverage analysis, and a 100-example error analysis.

Significance. If the results hold, this is a strong empirical contribution to discrete-reasoning reading comprehension: it demonstrates large gains over a same-encoder BERT baseline on the externally evaluated DROP benchmark, and the released source code and detailed ablations make the components reproducible and comparable. The multi-span extraction and expression-reranking ideas are useful and clearly evaluated. The main caveat is that the 'negation' answer type is implemented as a hard-coded 100-minus-number complement rule rather than as general logical negation, so the roughly 9 F1 ablation gain attributed to negation should be interpreted narrowly until its coverage is analyzed. The headline test score itself is an external benchmark result and is not circular.

major comments (2)
  1. [§3.2, §3.5, Tables 2 and 5] The negation component is hard-coded to a base-100 complement: training annotations are generated only when 100 minus a mentioned number equals the gold answer (§3.5), and inference outputs 100 minus the number with the largest negation probability (§3.5). The ablation in Table 2 removes negation and loses roughly 9 F1 (69.5% relative for the large model), and Table 5 shows that 11.5% of dev predictions use the negation type with 96.3 EM/F1. However, the paper does not report the distribution of complement bases among DROP negation questions, so it is unclear whether this gain reflects a general logical-negation capability or a narrow percentage-complement heuristic. This is not merely a presentation issue: the error analysis in Table 7 includes a 'not born in the UK' question whose answer requires subtracting a subgroup count from a total, and that example is categorized as arithmetic computation, not negation, confirming the limitation. Please either generalize the negation rule to use an arbitrary base number from the passage, or explicitly reframe the contribution as a percentage-complement predictor and report how many negation-type questions in DROP are actually base-100 complements.
  2. [§3.5 vs. §4.1] There is a discrepancy between the training annotation limit and the inference setting for arithmetic expressions. Section 3.5 states that the search for addition/subtraction annotations considers 'three numbers at most,' while Section 4.1 sets the maximum number of signed numbers M to 4, and Figure 4 varies M from 3 to 6 during decoding. Since the reranker is trained on labels produced by the model's own beam search, an expression with four signed numbers can never match a gold annotation under the three-number training limit, so all such candidates are labeled wrong. This inconsistency affects reproducibility and the interpretation of the reranking experiments; please clarify whether the three-number limit applies only to gold-annotation search and whether M is intended as an inference-only parameter.
minor comments (4)
  1. [§4.2 and §4.3] The main results and several ablations are reported for a single run without variance or significance tests. For large-margin comparisons this is acceptable, but small differences such as the reranking gain of 1.8 F1 and the gated-span gain of 0.8 F1 in Tables 2 and 3 should be interpreted cautiously; please state whether the reported numbers are from a single seed or averaged.
  2. [Algorithm 1] In Algorithm 1, the span count is computed as t = arg max pspan + 1, but the label range for pspan is not defined in the text. Please clarify whether pspan predicts the number of spans minus one, and state the maximum value used in training.
  3. [§4.1] The counting class bound is set to 10 without discussion of how frequently DROP counts exceed 10; a sentence noting the coverage of this bound would help readers assess the count component's scope.
  4. [Related Work] There is a typo in the Related Work section: 'Morevoer' should be 'Moreover'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline numbers are external-benchmark results, and the weak-supervision rules are training-signal constructions rather than predictions derived from themselves.

full rationale

MTMSN's headline claims are evaluated on the external DROP dev/test sets using the official scoring script, so the reported 79.9 F1 is not an artifact of the model's own definitions. The weak-supervision annotation rules in §3.5, including the 100-minus negation rule, are training-signal generators: they map gold answers to latent annotations, and inference applies the inverse operation. This makes the negation head a fixed-format function (100 − selected number), but its contribution is measured by ablation on held-out DROP examples, so the gain is an empirical finding about the benchmark, not a restatement of the annotation rule. The arithmetic-expression reranking in §3.4 uses beam-search candidates labeled correct or wrong against gold answers; although the candidates come from the model itself, the labels are gold-derived, making this a self-training/hard-EM loop rather than a prediction that reduces to its inputs. Self-citations (Hu et al. 2018, 2019) are contextual and not load-bearing. No uniqueness theorem is imported, and no fitted parameter is renamed as a prediction. The negation rule's fixed 100 base is a generality limitation and a potential correctness risk, but it is not circularity because the component's reported benefit is established against held-out data rather than by construction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central empirical claim is measured against an external benchmark, so the main free parameters are hyperparameters and one hard-coded constant (negation base 100). There are no invented theoretical entities.

free parameters (5)
  • Negation base 100 = 100
    Hard-coded in annotation ('if 100 minus a number is equal to the answer') and inference ('output the answer as 100 minus this number'). This restricts the negation type to percentage-complement cases.
  • Maximum number of spans = 8
    Chosen on the dev set; Figure 3 shows best at 8 with a recall-precision trade-off.
  • Beam size for expression reranking = 3
    Chosen on the dev set; Figure 4 shows beam size 3 is best.
  • Maximum signed numbers M in an expression = 4
    Chosen on the dev set; larger M hurts performance because most expressions contain two or three signed numbers.
  • Counting class bound = 10
    The number of counting classes is set to 10; count answers beyond this bound cannot be predicted.
assumptions (3)
  • domain assumption Pretrained BERT representations transfer to DROP and are sufficient for the downstream reasoning heads.
    The encoder is a fixed-architecture BERT; the model is trained on DROP only for the prediction heads, so the contribution of the pretrained weights is assumed positive and is evaluated relative to a BERT baseline.
  • domain assumption All DROP gold answers can be represented by one of span, count, add/sub of at most three numbers, or 100-minus negation.
    Training searches only these annotation types; Table 6 reports 97.9% coverage on the train set, so about 2.1% of examples are skipped, and test coverage is assumed to follow the same distribution.
  • domain assumption The official DROP evaluation script measures what the paper reports.
    Test-set numbers come from a hidden evaluation the authors submitted to; the paper cites the official script for metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning." pith.science (2026). https://pith.science/paper/MU4YLIXQ

@misc{pith2026190805514,
  author       = {Pith},
  title        = {Pith review of: A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MU4YLIXQ}},
  note         = {Machine review of arXiv:1908.05514}
}
read the original abstract

Rapid progress has been made in the field of reading comprehension and question answering, where several systems have achieved human parity in some simplified settings. However, the performance of these models degrades significantly when they are applied to more realistic scenarios, such as answers involve various types, multiple text strings are correct answers, or discrete reasoning abilities are required. In this paper, we introduce the Multi-Type Multi-Span Network (MTMSN), a neural reading comprehension model that combines a multi-type answer predictor designed to support various answer types (e.g., span, count, negation, and arithmetic expression) with a multi-span extraction method for dynamically producing one or multiple text spans. In addition, an arithmetic expression reranking mechanism is proposed to rank expression candidates for further confirming the prediction. Experiments show that our model achieves 79.9 F1 on the DROP hidden test set, creating new state-of-the-art results. Source code\footnote{\url{https://github.com/huminghao16/MTMSN}} is released to facilitate future work.

Figures

Figures reproduced from arXiv: 1908.05514 by the authors.

Figure 1
Figure 1. Question-answer pairs along with a passage [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of MTMSN architecture. The multi-type answer predictor supports four kinds of answer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. EM/F1 scores of MTMSNLARGE with differ￾ent maximum numbers of spans. 2 3 4 5 Beam Size 75 77 79 81 Score EM F1 3 4 5 6 M 75 77 79 81 EM F1 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: EM/F1 scores of MTMSNLARGE with differ￾ent beam sizes and amounts of signed numbers (M). proposed expression reranking mechanism helps validate candidate expressions. Second, a new in￾ductive bias that enables the model to perform log￾ical negation has been introduced.…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 25 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016 a . Learning to compose neural networks for question answering. In Proceedings of NAACL

    work page 2016

  4. [4]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016 b . Neural module networks. In Proceedings of CVPR

    work page 2016

  5. [5]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP

    work page 2013

  6. [6]

    Xavier Carreras and Llu \' s M \`a rquez. 2004. Introduction to the conll-2004 shared task: Semantic role labeling. In Proceedings of CONLL

    work page 2004

  7. [7]

    Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858

    arXiv 2016

  8. [8]

    Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of ACL

    work page 2018

Show all 35 references
  1. [9]

    Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2016. Attention-over-attention neural networks for reading comprehension. arXiv preprint arXiv:1607.04423

    2016 arXiv

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL

    2019

  3. [11]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of NAACL

    2019

  4. [12]

    Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401

    2014 arXiv

  5. [13]

    Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415

    2016 arXiv

  6. [14]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of NIPS

    2015

  7. [15]

    Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of ICLR

    2016

  8. [16]

    Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension. In Proceedings of ACL

    2019

  9. [17]

    Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In Proceedings of IJCAI

    2018

  10. [18]

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of ICCV

    2017

  11. [19]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL

    2017

  12. [20]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    2016 arXiv

  13. [21]

    Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of ACL

    2017

  14. [22]

    Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In Proceedings of ICLR

    2016

  15. [23]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP

    2016

  16. [24]

    Scott Reed and Nando De Freitas. 2016. Neural programmer-interpreters. In Proceedings of ICLR

    2016

  17. [25]

    Azriel Rosenfeld and Mark Thurston. 1971. Edge and curve detection for visual scene analysis. IEEE Transactions on computers, (5):562--569

    1971

  18. [26]

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In Proceedings of ICLR

    2019

  19. [27]

    Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of ICLR

    2017

  20. [28]

    Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Proceedings of NIPS

    2015

  21. [29]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS

    2017

  22. [30]

    Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018 a . Evidence aggregation for answer re-ranking in open-domain question answering. In Proceedings of ICLR

    2018

  23. [31]

    Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of ACL

    2017

  24. [32]

    Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018 b . Multi-passage machine reading comprehension with cross-passage answer verification. In Proceedings of ACL

    2018

  25. [33]

    Zhen Wang, Jiachen Liu, Xinyan Xiao, Yajuan Lyu, and Tian Wu. 2018 c . Joint training of candidate extraction and answer selection for reading comprehension. In Proceedings of ACL

    2018

  26. [34]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

    2016 arXiv

  27. [35]

    Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In Proceedings of ICLR

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.