REVIEW 4 major objections 6 minor 48 references
Semantic Role Labeling of NomBank Partitives
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read An ensemble of a BERT-based deep model and an AdaBoost feature model identifies the arguments of partitive nouns in the NomBank corpus, reaching an F1 of 91.74% with gold parses and 91.12% with an automatic parser.
desk verdict Useful new dataset and task framing for NomBank partitives; credible results, but the ensemble weighting needs to be specified before the top F1 is fully trustable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a weighted ensemble of two very different views of the same sentence. The deep view is a BERT-base transformer whose intermediate layers are pooled with learned weights, fine-tuned jointly with an auxiliary head that predicts part of speech, chunk, and tree-distance features; this auxiliary task improves the deep model over a plain BERT-plus-BiLSTM baseline. The feature view is an AdaBoost classifier over a hand-crafted feature set: the head word, its neighbors, POS and BIO tags, embedding-similarity scores, the partitive subclass label, and two path heuristics (one collapsing BIO sequences into phrase paths, one encoding four prototypical paths between predicate or support verb and the ARG1). A weighted vote, with weights learned on training data, merges the two models' outputs, so the final prediction requires both contextual semantics and surface grammar to agree.
What would settle it
Run the best ensemble on the original, unmodified NomBank test sentences for partitives, including those with multiple partitive predicates and with tokenization mismatches, and compare ARG1 F1 to the paper's 91.74%; a material drop would show that the cleaning rule inflated the result.
Extended reading notes
Core claim
The paper's central claim is that ARG1 identification for partitive nouns in NomBank is best modeled as a token classification task—deciding, for each word, whether it is the head of the ARG1 of the sentence's partitive predicate—and that the highest accuracy comes from fusing two very different models. One view is a BERT-based deep network that relies on contextual word representations and a weighted average of BERT layers; the other is an AdaBoost model built on linguistically motivated features such as the token distance between predicate and candidate, BIO chunk paths, head word identity, word embeddings and their contextual variants, and the partitive subclass label. The ensemble learns adaptive weights over the two models' outputs. Reported F1 on the gold test set for partitives is 91.74%, and 91.12% when the parser is the automatic neural parser; for the single frequent predicate percent the scores reach 95.19% and 94.78% respectively. The authors interpret the small gold-vs-parser gap as evidence that parse error is not the primary bottleneck and that partitives, while the largest shared noun frame, are among the simpler NomBank noun classes.
Load-bearing premise
The load-bearing premise is that the rare cases dropped during data cleaning—sentences with more than one partitive and tokenization mismatches like hyphenated group nouns—are not systematically harder than the kept cases; if they are, the reported F1 overstates how well the system would do on the full NomBank partitive distribution.
Editorial extensions
If this is right
- Partitive ARG1 detection can be treated as a single class-level task: one model covers hundreds of predicate nouns, so annotation effort does not need to be per-lexeme.
- Scores stay above 91% F1 when an automatic parser replaces gold parses, implying parsing error is not the dominant source of failure for this task.
- Head-based matching is sufficient for strong scores; the common simplification of representing an ARG1 phrase by its head noun does not prevent high accuracy.
- The reported numbers provide a high baseline for other NomBank noun classes, which the authors expect to be harder because they often have more arguments and looser semantic constraints.
- Because the corpus is Wall Street Journal from the 1990s, the system's performance should be checked on other genres before generalizing; the authors themselves flag this.
Reading between the lines
- A likely consequence the paper leaves implicit: the ensemble recipe—one contextual model plus one sparse feature model with learned weights—could transfer to the other noun frame classes in NomBank, with path features adjusted per class.
- Because the cleaning rule drops multi-partitive sentences, the published score is not directly comparable to a system evaluated on the full corpus; we would expect the all-sentence score to be somewhat lower, and the drop would measure the cost of the rule.
- The 'transparency' property of partitives—the ARG1 acts as the semantic head—suggests that span selection models might do even better than head token classification, since they could directly predict the whole argument phrase instead of its head.
- A testable prediction: the same ensemble evaluated on semantically odd partitives such as 'a pound of sincerity' should still assign an ARG1 because it has no world-knowledge features, revealing the limit of purely local cues.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper defines a semantic role labeling task for English partitive nouns (e.g., 'percent', 'portion', 'committee') in NomBank, using a simplified setup in which each sentence contains exactly one partitive predicate and exactly one ARG1 to be identified. The authors describe a feature-based AdaBoost baseline, several student systems, a BERT-based deep learning system, and an ensemble of the deep learning and feature-based systems. The best ensemble achieves F1 of 91.74% on the gold PTB-parse version and 91.12% with Berkeley Neural Parser output (Tables 2 and 3). The paper also discusses the creation of datasets, evaluation methodology, and limitations.
Significance. If the reported results hold, the paper provides a new benchmark for a focused SRL subtask: identifying the ARG1 of partitive nouns, with a strong result from a relatively simple ensemble. The main contributions are the task formulation, the dataset definitions, and the demonstration that a combination of classical feature-based and transformer models reaches high F1. The paper is transparent about some limitations (e.g., data cleaning, WSJ domain). However, the evaluation protocol has ambiguities—particularly the ensemble weight learning—that currently prevent full confidence in the headline number. The task itself is narrow, so the significance is moderate.
major comments (4)
- [Section 8] The paper states that ensemble weights are 'learned adaptively as we train the ensemble model on the data' but does not specify the training data, the weighting schema, or the selected weight values. Since the reported headline F1 (91.74% in Table 2, system 6) is the ensemble's score, an honest evaluation requires a clear statement that weights were learned only on training/development data, not selected based on test-set performance. Please specify the protocol, report the learned weights or a development-based selection, and include an ablation with equal weights or weights fixed on the development set. Without this, the gap over deep learning system 2 (89.26%) could be an artifact of test-set overfitting.
- [Sections 5.2 and 6.3] The dataset statistics are internally inconsistent. Section 5.2 reports training partitive ARG1s as '1K', while Section 6.3 reports 9,987; development partitive counts are 370 (Section 5.2) vs. 372 (Section 6.3/Table 1); and test counts appear as 550 partitive / 150 % (Section 5.2) vs. 555 partitive / 50 % (Section 6.3). These discrepancies make the experiments difficult to reproduce and call into question the completeness of the reporting. Please provide a single consistent table of instance counts for each split and task.
- [Section 5.3] The scoring rule that 'any of the name words (NNP) is correct' for proper noun ARG1s relaxes exact-match evaluation and can only increase reported F1. Please quantify how many test instances are affected by this rule and, if feasible, report results under strict head-word matching. This is needed to assess whether the reported 91.74% is partly an artifact of the scoring definition.
- [Section 5.1] The exclusion of sentences with multiple partitives and of tokenization mismatches (e.g., 'warehouse-club') is described as removing rare cases, but the paper does not report how many examples are removed or investigate whether these cases are systematically harder. Since the task is defined over the cleaned subset, the abstract's claims about performance on 'NomBank partitives' should be qualified, and the impact of cleaning on difficulty should be assessed. At minimum, report the number of removed examples per split.
minor comments (6)
- [Introduction] The word 'arguments' is misspelled as 'aguments' in the first paragraph.
- [Section 2] In the sentence about nomenclature, 'ARGO' should be 'ARG0'.
- [Section 5.3] In the proper-name example, 'Exon' should be 'Exxon'.
- [Section 6.1.2] The embedding resource 'SPACY' should be styled as 'spaCy', and 'en_core_web_md' should be written consistently.
- [Figure 4] The label 'SUP' is used for the support verb but is not defined in the caption or surrounding text; please define it.
- [General] The paper uses 'F-score', 'F1', and 'f-measure' interchangeably; please choose one convention and apply it consistently.
Circularity Check
No demonstrated circularity: the 91.74% F1 is an out-of-sample test-set measurement, with a transparency gap in the ensemble-weight learning procedure.
full rationale
The central result is an empirical F1 on PTB section 23, held out from training (sections 02-21) and development (24); Section 6 explicitly states that feature experimentation was measured on the development set and only the best system was run once on the test corpus. The partitive ARG1 gold labels come from NomBank, a fixed corpus with stated annotation conventions; citing Meyers et al. 2004a,b is standard resource attribution, and the labels are not outputs of the systems being evaluated. The baseline features (Sections 6.1.1-6.1.4) use POS, BIO, path, embedding, and NomBank class metadata; none of these features are the test ARG1 labels themselves, and the prediction target is not used to construct the feature definitions. Data cleaning in Section 5.1 (keep the first partitive per sentence, remove tokenization mismatches) and the head-matching convention in Section 5.3 change the effective evaluation distribution but do not make the reported score a renamed input. Section 8's statement that ensemble weights are 'learned adaptively as we train the ensemble model on the data' is underspecified: it does not state on which split the weights are learned, nor report the learned weights or an equal-weight ablation. This is a methodological transparency and potential-leakage concern, not a demonstrated reduction of the prediction to a fitted input; absent an exhibited equation or procedure that selects test labels, no circular step is established.
Assumptions & free parameters
free parameters (2)
- Class-imbalance loss scaling =
Not reported
- Ensemble voting weights =
Not reported
assumptions (4)
- domain assumption NomBank annotations are correct ground truth for partitive ARG1 labels.
- domain assumption Head-word matching is a valid proxy for full-span ARG1 accuracy.
- domain assumption The standard PTB train/dev/test split (02-21/24/23) is appropriate.
- domain assumption Berkeley Neural Parser output is close enough to gold that non-gold evaluation remains meaningful.
Cite this review
Pith. "Pith review of Semantic Role Labeling of NomBank Partitives." pith.science (2026). https://pith.science/paper/EGFPQZ4Q
@misc{pith2026241214328,
author = {Pith},
title = {Pith review of: Semantic Role Labeling of NomBank Partitives},
year = {2026},
howpublished = {\url{https://pith.science/paper/EGFPQZ4Q}},
note = {Machine review of arXiv:2412.14328}
}
read the original abstract
This article is about Semantic Role Labeling for English partitive nouns (5%/REL of the price/ARG1; The price/ARG1 rose 5 percent/REL) in the NomBank annotated corpus. Several systems are described using traditional and transformer-based machine learning, as well as ensembling. Our highest scoring system achieves an F1 of 91.74% using "gold" parses from the Penn Treebank and 91.12% when using the Berkeley Neural parser. This research includes both classroom and experimental settings for system development.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[4]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[5]
C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. The Berkeley FrameNet Project . In Coling-ACL98 , pages 86--90
work page 1998
-
[6]
Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg
work page 2006
-
[7]
Leo Breiman. 2001. Random forests. Machine learning, 45:5--32
work page 2001
-
[8]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...
arXiv 2020
Show all 48 references
-
[9]
Xavier Carreras and Llu \' s M \`a rquez. 2005. https://www.aclweb.org/anthology/W05-0620 Introduction to the C o NLL -2005 shared task: Semantic role labeling . In Proceedings of the Ninth Conference on Computational Natural Language Learning ( C o NLL -2005) , pages 152--164...
2005
-
[10]
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/W19-4828 What does BERT look at? an analysis of BERT ' s attention . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...
2019 doi
-
[11]
M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing . Ph.D. thesis, University of Pennsylvania
1999
-
[12]
Ronan Collobert, Jason Weston, L \'e on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493--2537
2011
-
[13]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...
2019 doi
-
[14]
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. http://www.lrec-conf.org/proceedings/lrec2004/pdf/5.pdf The automatic content extraction ( ACE ) program -- tasks, data, and evaluation . In Proceedings of the Fo...
2004
-
[15]
Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. 2020. A survey on ensemble learning. Frontiers of Computer Science, 14:241--258
2020
-
[16]
David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. 2014. https://arxiv.org/abs/1312.4314 Learning factored representations in a deep mixture of experts . Preprint, arXiv:1312.4314
2014 arXiv
-
[17]
C. J. Fillmore and B. T. S. Atkins. 1998. FrameNet and Lexicographic Relevance . In LREC: First International Conference on Language Resources and Evaluation , pages 417--423
1998
-
[18]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. https://arxiv.org/abs/1703.03400 Model-agnostic meta-learning for fast adaptation of deep networks . Preprint, arXiv:1703.03400
2017 arXiv
-
[19]
J. Firth. 1957. A synopsis of linguistic theory. In Studies in Linguistic Analysis. Blackwell
1957
-
[20]
Freund and R
Y. Freund and R. E. Schapire. 1995. A decision-theoretic generalization of on-line learning and an application to boosting. j-LECT-NOTES-COMP-SCI, 904
1995
-
[21]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org
2016
-
[22]
J. S. Gruber. 1965. Studies in Lexical Relations . Ph.D. thesis, MIT. Reproduced by Indiana University Linguistics Club, January 1970
1965
-
[23]
Haji c , M
J. Haji c , M. Ciaramita, R. Johansson, D. Kawahara, M. A. Mart\' , L. M\` a rquez, A. Meyers, J. Nivre, S. Pad\' o , J. S t e p\' a nek, P. Stra n \' a k, M. Surdeanu, N. Xue, and Y. Zhang. 2009. The CoNLL -2009 shared task: Syntactic and semantic dependencies in multiple lan...
2009
-
[24]
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473--483
2017
-
[25]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780
1997
-
[26]
Jacobs, Michael I
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. https://doi.org/10.1162/neco.1991.3.1.79 Adaptive mixtures of local experts . Neural Computation, 3(1):79--87
1991 doi
-
[27]
Z. P. Jiang and H. T. Ng. 2006. Semantic role labeling of nombank: A maximum entropy approach. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), Sydney, Australia
2006
-
[28]
Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the 3rd International Conference on Language Resources and Evaluation ( LREC -2002) , Las Palmas, Spain. European Language Resources Association (ELRA)
2002
-
[29]
Nikita Kitaev and Dan Klein. 2018. https://doi.org/10.18653/v1/P18-1249 Constituency parsing with a self-attentive encoder . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676--2686, Melbourne, Austral...
2018 doi
-
[30]
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven J Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrat...
2014
-
[31]
Meyers, R
A. Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004 a . Annotating Noun Argument Structure for NomBank . In Proceedings of LREC-2004 , Lisbon, Portugal
2004
-
[32]
Meyers, R
A. Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004 b . The NomBank Project: An Interim Report . In NAACL/HLT 2004 Workshop Frontiers in Corpus Annotation , Boston
2004
-
[33]
Miltsakaki, A
E. Miltsakaki, A. Joshi, R. Prasad, and B. Webber. 2004. Annotating discourse connectives and their arguments. In NAACL/HLT 2004 Workshop: Frontiers in Corpus Annotation, pages 9--16, Boston, Massachusetts, USA. Association for Computational Linguistics
2004
-
[34]
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. https://doi.org/10.18653/v1/D18-1191 A span selection model for semantic role labeling . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1630--1642, Brussels, Belgium. Ass...
2018 doi
-
[35]
Palmer, D
M. Palmer, D. Gildea, and P. Kingsbury. 2005. http://www.cs.rochester.edu/ gildea/palmer-propbank-cl.pdf The Proposition Bank : An annotated corpus of semantic roles . Computational Linguistics, 31(1):71--106
2005
-
[36]
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Assoc...
2018 doi
-
[37]
Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James Martin, and Daniel Jurafsky. 2005. https://doi.org/10.3115/1219840.1219912 Semantic role labeling using different syntactic views . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics ( AC...
2005
-
[38]
Michael Roth and Mirella Lapata. 2016. https://doi.org/10.18653/v1/P16-1113 Neural semantic role labeling with dependency path embeddings . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1192--1202, Ber...
2016 doi
-
[39]
Devendra Singh Sachan, Yuhao Zhang, Peng Qi, and William Hamilton. 2021. https://arxiv.org/abs/2008.09084 Do syntax trees help pre-trained transformers extract information? Preprint, arXiv:2008.09084
2021 arXiv
-
[40]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108
2019 arXiv
-
[41]
Shiliang Sun. 2013. A survey of multi-view machine learning. Neural computing and applications, 23:2031--2038
2013
-
[42]
Surdeanu, R
M. Surdeanu, R. Johansson, A. Meyers, L. M\' a rquez, and J. Nivre. 2008. The CoNLL-2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies . In Proceedings of the CoNLL-2008 Shared Task , Manchester, GB
2008
-
[43]
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950
2019 arXiv
-
[44]
Tjong Kim Sang and Sabine Buchholz
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. https://www.aclweb.org/anthology/W00-0726 Introduction to the C o NLL -2000 shared task chunking . In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop
2000
-
[45]
Minghao Wu, Fei Liu, and Trevor Cohn. 2018. https://doi.org/10.18653/v1/D18-1310 Evaluating the utility of hand-crafted features in sequence labelling . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2850--2856, Brussels, Belgi...
2018 doi
-
[46]
Nianwen Xue and Martha Palmer. 2003. https://doi.org/10.3115/1119250.1119257 Annotating the propositions in the P enn C hinese treebank . In Proceedings of the Second SIGHAN Workshop on C hinese Language Processing , pages 47--54, Sapporo, Japan. Association for Computational ...
2003
-
[47]
Nianwen Xue and Martha Palmer. 2004. https://www.aclweb.org/anthology/W04-3212 Calibrating features for semantic role labeling . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 88--94, Barcelona, Spain. Association for Computati...
2004
-
[48]
Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum...
2015
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.