{"id":"0901bd78-4def-46b8-8532-ab4b05aced7c","arxiv_id":"2412.14328","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An ensemble of BERT and feature-based models labels the ARG1 of partitive nouns with 91.74% F1 on gold NomBank parses and 91.12% with the Berkeley Neural parser.","lead":"This paper trains machine-learning models to identify the noun a partitive word like 'percent' or 'portion' refers to, such as the price in 'the price rose 5 percent'. Its best ensemble reaches about 92 percent F1 on the NomBank benchmark for this narrow task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation leaks test-set information through the ensemble weight-learning procedure.","rationale":"The reader's verdict is CONDITIONAL, focusing on data cleaning (Section 5.1) and reproducibility. My strongest concern is different: the ensemble's weight-learning procedure in Section 8 is the only component of the pipeline whose evaluation protocol is unspecified with respect to the test set. The paper claims weights are 'learned adaptively' but never states the training data for those weights, the form of the weighting, or the number of free parameters. If those weights were tuned on the test set, the 91.74% figure is a selection-maximized number, not a fair estimate, and it would be misleading to compare it against end-to-end systems whose parameters were trained only on training data. This is not an accusation of bad faith—the ambiguity is genuine and testable. The data-cleaning issue the reader flagged is real but less load-bearing: even if multi-partitive sentences or tokenization mismatches are harder, the cleaning is disclosed in Section 5.1, and the claim is explicitly about the cleaned subset. A reader can see exactly what was evaluated. By contrast, the ensemble protocol is not disclosed at all, so the claim's validity cannot be checked from the paper. The concrete_test I propose—re-running the ensemble with fixed/dev-based weights—would settle whether the central number is robust. Hence the verdict remains CONDITIONAL, and no change to the reader's overall verdict is needed, but the condition should prioritize specifying and validating the ensemble weighting procedure before artifact release.","tokens_in":13973,"tokens_out":1461,"duration_ms":12444,"concrete_test":"Re-run the ensemble on the gold test set with three fixed weighting schemes—equal weights, weights set on the development set only, and weights learned on the training set only—and also re-run the reported procedure while recording exactly which data the learned weights use. If a fixed/dev-only weighting yields partitive gold F1 within ~0.5 of 91.74%, the claim is honest; if the dev-only or equal-weight F1 drops materially (e.g., below 89.5%), the headline number is test-set-selected and should be reported as a selection result, not a predictive evaluation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (91.74% F1 on the gold partitive test set) depends on the ensemble's weighted voting scheme being trained honestly. Section 8 states weights are 'learned adaptively as we train the ensemble model on the data,' but the paper never specifies on which data the weights are learned or the weighting schema (e.g., softmax over dev F1, learned scalars, gating network). It also does not report the learned weights, their variance, or any ablation fixing equal weights. If the ensemble weights were optimized on the test set—selected because they maximize held-out test F1—then the reported 91.74% is a selected maximum over a weight space, not an unbiased estimate, and the gap over deep learning system 2 (89.26% partitive gold) could shrink or invert under honest evaluation. The ablation in Table 1 and the per-system results give no evidence about this step. This is the weakest link because every other reported component is either a fixed baseline or a student system with a stated training procedure, whereas the ensemble weighting is the one contributor whose protocol is unconstrained by the text.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a semantic role labeling task for English partitive nouns (e.g., 'percent', 'portion', 'committee') in NomBank, using a simplified setup in which each sentence contains exactly one partitive predicate and exactly one ARG1 to be identified. The authors describe a feature-based AdaBoost baseline, several student systems, a BERT-based deep learning system, and an ensemble of the deep learning and feature-based systems. The best ensemble achieves F1 of 91.74% on the gold PTB-parse version and 91.12% with Berkeley Neural Parser output (Tables 2 and 3). The paper also discusses the creation of datasets, evaluation methodology, and limitations.","tokens_in":14056,"tokens_out":8010,"duration_ms":63867,"significance":"If the reported results hold, the paper provides a new benchmark for a focused SRL subtask: identifying the ARG1 of partitive nouns, with a strong result from a relatively simple ensemble. The main contributions are the task formulation, the dataset definitions, and the demonstration that a combination of classical feature-based and transformer models reaches high F1. The paper is transparent about some limitations (e.g., data cleaning, WSJ domain). However, the evaluation protocol has ambiguities—particularly the ensemble weight learning—that currently prevent full confidence in the headline number. The task itself is narrow, so the significance is moderate.","major_comments":[{"comment":"The paper states that ensemble weights are 'learned adaptively as we train the ensemble model on the data' but does not specify the training data, the weighting schema, or the selected weight values. Since the reported headline F1 (91.74% in Table 2, system 6) is the ensemble's score, an honest evaluation requires a clear statement that weights were learned only on training/development data, not selected based on test-set performance. Please specify the protocol, report the learned weights or a development-based selection, and include an ablation with equal weights or weights fixed on the development set. Without this, the gap over deep learning system 2 (89.26%) could be an artifact of test-set overfitting.","section":"Section 8"},{"comment":"The dataset statistics are internally inconsistent. Section 5.2 reports training partitive ARG1s as '1K', while Section 6.3 reports 9,987; development partitive counts are 370 (Section 5.2) vs. 372 (Section 6.3/Table 1); and test counts appear as 550 partitive / 150 % (Section 5.2) vs. 555 partitive / 50 % (Section 6.3). These discrepancies make the experiments difficult to reproduce and call into question the completeness of the reporting. Please provide a single consistent table of instance counts for each split and task.","section":"Sections 5.2 and 6.3"},{"comment":"The scoring rule that 'any of the name words (NNP) is correct' for proper noun ARG1s relaxes exact-match evaluation and can only increase reported F1. Please quantify how many test instances are affected by this rule and, if feasible, report results under strict head-word matching. This is needed to assess whether the reported 91.74% is partly an artifact of the scoring definition.","section":"Section 5.3"},{"comment":"The exclusion of sentences with multiple partitives and of tokenization mismatches (e.g., 'warehouse-club') is described as removing rare cases, but the paper does not report how many examples are removed or investigate whether these cases are systematically harder. Since the task is defined over the cleaned subset, the abstract's claims about performance on 'NomBank partitives' should be qualified, and the impact of cleaning on difficulty should be assessed. At minimum, report the number of removed examples per split.","section":"Section 5.1"}],"minor_comments":[{"comment":"The word 'arguments' is misspelled as 'aguments' in the first paragraph.","section":"Introduction"},{"comment":"In the sentence about nomenclature, 'ARGO' should be 'ARG0'.","section":"Section 2"},{"comment":"In the proper-name example, 'Exon' should be 'Exxon'.","section":"Section 5.3"},{"comment":"The embedding resource 'SPACY' should be styled as 'spaCy', and 'en_core_web_md' should be written consistently.","section":"Section 6.1.2"},{"comment":"The label 'SUP' is used for the support verb but is not defined in the caption or surrounding text; please define it.","section":"Figure 4"},{"comment":"The paper uses 'F-score', 'F1', and 'f-measure' interchangeably; please choose one convention and apply it consistently.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest but useful contribution to a narrow task. The main risk is the underspecified ensemble weighting; if the authors can clarify and validate that, the central claim is likely sound. The inconsistent dataset statistics are concerning and must be fixed. The paper's informal style (classroom origins) is acceptable but needs professional polish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is worth reading for anyone working on NomBank or noun SRL. The new thing is treating all partitive nouns as one frame-like class, which is a sensible extension of FrameNet-style thinking to NomBank, and the paper actually implements it with a new dataset split and several baselines. The best system's 91.74 F1 on gold parses is plausible and honestly reported as a single held-out test run, and the limitations section is unusually candid: partitives are the simplest NomBank class, WSJ is one register, and the task is simplified.\n\nWhat it does well: the feature-based baseline is described in enough detail to be reproduced; the ablation in Table 1 shows which feature groups matter; and the comparison across gold and non-gold parses gives a realistic sense of parser error contribution. The classroom origin is a bit unusual but not a weakness; the student systems are clearly credited.\n\nSoft spots, in order of importance. Section 8 says ensemble weights are 'learned adaptively' without saying on which split or what the weighting schema is. If those weights were chosen on the test set, the 91.74 is a selected maximum, not an honest estimate. The paper doesn't give evidence that they weren't, so the top-line number is conditional on this being done on dev. This is fixable, but it needs a clear statement and ideally the weights themselves. Second, the evaluation cleans the data: only the first partitive per sentence, and tokenization mismatches are dropped. That's probably fine, but it means the F1 is on a slightly easier distribution than raw NomBank. Third, a correct ARG1 match for a proper name counts any NNP, which is lenient; with mostly numeric or common-noun ARG1s this likely doesn't move much. Fourth, there are small internal inconsistencies in the counts (370 vs 372 dev, 50 vs 150 percent test) and a cross-reference pointing to Figure 2 instead of Table 2. All minor.\n\nBottom line: the central claim holds up as far as the text shows, but the ensemble weighting is a genuine gap. I'd send this to reviewers—the task framing is worth publishing—and ask for the weighting protocol, an equal-weight ablation, and the data/code. With those, it's a solid resource paper.","headline":"Useful new dataset and task framing for NomBank partitives; credible results, but the ensemble weighting needs to be specified before the top F1 is fully trustable.","tokens_in":14688,"tokens_out":3073,"would_cite":true,"duration_ms":24910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of a BERT-based deep model and an AdaBoost feature model identifies the arguments of partitive nouns in the NomBank corpus, reaching an F1 of 91.74% with gold parses and 91.12% with an automatic parser.","keywords":["semantic role labeling","NomBank","partitive nouns","ensemble learning","BERT","AdaBoost","feature-based model","gold parses"],"falsifier":"Run the best ensemble on the original, unmodified NomBank test sentences for partitives, including those with multiple partitive predicates and with tokenization mismatches, and compare ARG1 F1 to the paper's 91.74%; a material drop would show that the cleaning rule inflated the result.","tokens_in":13684,"feed_emoji":"🧩","tokens_out":6674,"duration_ms":55907,"temperature":0.7,"pith_summary":"Partitive nouns—words like percent, portion, or group—take an argument (ARG1) that specifies what is being measured or collected, and this paper asks whether a computer can identify that argument reliably across the roughly 500 partitive predicates in the NomBank corpus. Treating all partitives as one shared class rather than as thousands of separate predicate-specific classes, the authors build an ensemble that combines a BERT-based deep model with an AdaBoost classifier over hand-engineered features. On the test section, the ensemble finds the correct ARG1 head 91.74% of the time when given gold Penn Treebank parses, and 91.12% when given an automatic neural parser's output. If the result holds, it means noun-based semantic roles are learnable with high accuracy from a modest-size corpus, and that merging complementary model views costs little.","feed_headline":"Ensemble finds partitive arguments at 91.74% F1","feed_subtitle":"Two views—a BERT model and hand-built features—join to spot the ARG1 of partitive nouns.","key_machinery":"The carrying mechanism is a weighted ensemble of two very different views of the same sentence. The deep view is a BERT-base transformer whose intermediate layers are pooled with learned weights, fine-tuned jointly with an auxiliary head that predicts part of speech, chunk, and tree-distance features; this auxiliary task improves the deep model over a plain BERT-plus-BiLSTM baseline. The feature view is an AdaBoost classifier over a hand-crafted feature set: the head word, its neighbors, POS and BIO tags, embedding-similarity scores, the partitive subclass label, and two path heuristics (one collapsing BIO sequences into phrase paths, one encoding four prototypical paths between predicate or support verb and the ARG1). A weighted vote, with weights learned on training data, merges the two models' outputs, so the final prediction requires both contextual semantics and surface grammar to agree.","core_discovery":"The paper's central claim is that ARG1 identification for partitive nouns in NomBank is best modeled as a token classification task—deciding, for each word, whether it is the head of the ARG1 of the sentence's partitive predicate—and that the highest accuracy comes from fusing two very different models. One view is a BERT-based deep network that relies on contextual word representations and a weighted average of BERT layers; the other is an AdaBoost model built on linguistically motivated features such as the token distance between predicate and candidate, BIO chunk paths, head word identity, word embeddings and their contextual variants, and the partitive subclass label. The ensemble learns adaptive weights over the two models' outputs. Reported F1 on the gold test set for partitives is 91.74%, and 91.12% when the parser is the automatic neural parser; for the single frequent predicate percent the scores reach 95.19% and 94.78% respectively. The authors interpret the small gold-vs-parser gap as evidence that parse error is not the primary bottleneck and that partitives, while the largest shared noun frame, are among the simpler NomBank noun classes.","pith_inferences":["A likely consequence the paper leaves implicit: the ensemble recipe—one contextual model plus one sparse feature model with learned weights—could transfer to the other noun frame classes in NomBank, with path features adjusted per class.","Because the cleaning rule drops multi-partitive sentences, the published score is not directly comparable to a system evaluated on the full corpus; we would expect the all-sentence score to be somewhat lower, and the drop would measure the cost of the rule.","The 'transparency' property of partitives—the ARG1 acts as the semantic head—suggests that span selection models might do even better than head token classification, since they could directly predict the whole argument phrase instead of its head.","A testable prediction: the same ensemble evaluated on semantically odd partitives such as 'a pound of sincerity' should still assign an ARG1 because it has no world-knowledge features, revealing the limit of purely local cues."],"forward_implications":["Partitive ARG1 detection can be treated as a single class-level task: one model covers hundreds of predicate nouns, so annotation effort does not need to be per-lexeme.","Scores stay above 91% F1 when an automatic parser replaces gold parses, implying parsing error is not the dominant source of failure for this task.","Head-based matching is sufficient for strong scores; the common simplification of representing an ARG1 phrase by its head noun does not prevent high accuracy.","The reported numbers provide a high baseline for other NomBank noun classes, which the authors expect to be harder because they often have more arguments and looser semantic constraints.","Because the corpus is Wall Street Journal from the 1990s, the system's performance should be checked on other genres before generalizing; the authors themselves flag this."],"supporting_citations":[{"why":"Provides the NomBank corpus and its annotation scheme; every system and evaluation in this paper is defined against these data.","marker":"Meyers et al., 2004a,b"},{"why":"Defines the PropBank argument-labeling conventions (ARG0, ARG1, REL) that the paper adopts for noun predicates.","marker":"Palmer et al., 2005"},{"why":"Supplies the BERT model that is the deep-view component of the ensemble and of the student-inspired deep learning systems.","marker":"Devlin et al., 2019"},{"why":"Supplies AdaBoost, the boosting algorithm used as the feature-based component of the baseline and the ensemble.","marker":"Freund and Schapire, 1995"},{"why":"Supplies the automatic neural parser whose output defines the non-gold evaluation setting.","marker":"Kitaev and Klein, 2018"},{"why":"Defines the CONLL chunking format and BIO tags that shape the data representation and one path heuristic.","marker":"Tjong Kim Sang and Buchholz, 2000"}],"fun_headline_variants":["Partitive arguments found at 91.74% F1 with ensemble","Ensemble beats 91% F1 on partitive SRL","BERT+AdaBoost hits 91.74% F1 for partitives","Token classification nails partitive ARG1 at 91.74%","Two-model fusion tops 91% F1 on partitives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rare cases dropped during data cleaning—sentences with more than one partitive and tokenization mismatches like hyphenated group nouns—are not systematically harder than the kept cases; if they are, the reported F1 overstates how well the system would do on the full NomBank partitive distribution.","fun_headline_variants_meta":{"raw":{"variants":["Partitive arguments found at 91.74% F1 with ensemble","Ensemble beats 91% F1 on partitive SRL","BERT+AdaBoost hits 91.74% F1 for partitives","Token classification nails partitive ARG1 at 91.74%","Two-model fusion tops 91% F1 on partitives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000622,"raw_usage":{"total_tokens":2842,"prompt_tokens":867,"completion_tokens":1975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":483,"tokens_out":1975,"duration_ms":13185,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:19:49.691517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the best ensemble on the original, unmodified NomBank test sentences for partitives, including those with multiple partitive predicates and with tokenization mismatches, and compare ARG1 F1 to the paper's 91.74%; a material drop would show that the cleaning rule inflated the result.","supporting_citations":[],"review_version":1}