REVIEW 4 major objections 5 minor 41 references
Semantic Role Labeling with Associated Memory Network
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A memory module that retrieves similar labeled sentences from training data and attends to their labels improves syntax-agnostic semantic role labeling, matching or beating the previous best results on CoNLL-2009.
desk verdict A genuinely new memory-network component for SRL with small gains, but the gains are not convincingly attributed to the label memory — the added encoder is uncontrolled and the numbers lack significance testing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Associated memory network (AMN): a memory module that turns labeled training sentences into an inter-sentence attention source. It has three stages: associated-sentence selection (edit distance on POS tags chosen by development results), inter-sentence attention (a shared BiLSTM encodes target and memory sentences, then softmax-normalized dot products align each target word to memory words), and label merging (four strategies compared; plain average of the m attention-weighted argument-label embeddings performs best). The resulting memory embedding is concatenated with the base word representation before the main LSTM encoder, so the model can condition its labeling decision on how similar words were labeled in remembered sentences.
What would settle it
Re-run the reported experiments with at least ten random seeds and construct confidence intervals for the AMN-minus-base F1 difference on the CoNLL-2009 English development set; if the interval includes zero, or if edit-distance retrieval does not beat random retrieval beyond noise, the central claim is not supported.
Extended reading notes
Core claim
The paper's central claim is that a semantic role labeler can be improved by attending to the gold labels of retrieved training sentences. The AMN stores labeled sentences from the training set; for each target sentence it selects the top m by edit distance over POS tag sequences, encodes target and associated sentences with a shared BiLSTM, computes an inter-sentence attention matrix by dot product followed by row-wise softmax, and uses the attention weights to take an average of the associated sentences' argument-label embeddings. That averaged memory embedding is concatenated onto each target word's input before the main BiLSTM classifier. In the reported results, the full model reaches 89.6 F1 on CoNLL-2009 English WSJ, equal to the prior syntax-agnostic state of the art, 79.7 on Brown, 0.7 higher, and 83.8 on Chinese, 2.0 higher than the similar base architecture; the confusion-matrix analysis attributes part of the gain to fewer core-argument confusions, and the memory-size trend suggests further gains with more associated sentences.
Load-bearing premise
The load-bearing premise is that the reported F1 differences (0.3 to 0.7 points) are real and not run-to-run noise: the paper averages only 2 to 4 runs and reports no variance or significance test.
Editorial extensions
If this is right
- The AMN gain persists when the base already includes ELMo: the memory module contributes 0.5 F1 over the base model and 0.3 F1 over the ELMo-enhanced base on the English development set, so memory and contextual embeddings are complementary.
- The largest reported advantage is in out-of-domain generalization: 79.7 F1 on Brown, 0.7 above the previous syntax-agnostic state of the art.
- Performance trends upward as memory size increases from 2 to 6 associated sentences, suggesting the approach has room to grow with more memory.
- The method is not tied to a particular decoder: the authors note it is compatible with biaffine attention decoding, which was the strongest prior SRL decoder, so the two lines can be combined.
- Memory over the training set reduces core-argument confusions (A0/A1/A2) relative to the base, which is where most of the improvement appears.
Reading between the lines
- Random sentence retrieval nearly matches edit-distance retrieval on the development set (88.1 vs 88.3 F1), so the paper's evidence does not show that the specific distance matters; the active ingredient may be the attention over any labeled sentence's role labels.
- If the mechanism is robust, the same memory-and-attention recipe should transfer to other token-level tasks with gold-label supervision, such as named entity recognition or chunking, where attending over labeled examples could substitute for extra features.
- The reported gains are small relative to typical seed variance in neural SRL; a direct test would be a multi-seed comparison with confidence intervals, which the paper does not provide.
- The memory is static and selected once per target sentence; a natural extension is to make retrieval differentiable or update memory during inference, which could turn the observed memory-size trend into larger gains.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an Associated Memory Network (AMN) for dependency-based semantic role labeling on the CoNLL-2009 benchmark. For each target sentence, the model selects a small set of training sentences (by edit distance, word-mover distance, SIF distance, or randomly), encodes the target and the selected sentences with an additional BiLSTM encoder, computes inter-sentence attention, and merges the associated label embeddings into the base model's word representation before the main BiLSTM encoder. The full model with ELMo achieves 89.6 F1 on English in-domain (tying Cai et al., 2018), 79.7 F1 on English Brown (0.7 points above that system), and 83.8 F1 on Chinese. Ablations on retrieval distance, label-merging strategy, and memory size show small improvements over the base model on the English development set.
Significance. If the central claim is supported, this would be a novel and useful contribution: it is the first application of an inter-sentence memory/retrieval mechanism to dependency SRL, it is evaluated on a standard benchmark against strong recent baselines, and the authors make the implementation publicly available. The consistent positive differences in Tables 5-7, the out-of-domain improvement on Brown, and the confusion-matrix analysis are encouraging. However, the reported gains are small (0.3-0.5 F1 on development), are averaged over only 2-4 runs without variance or significance testing, and the AMN architecture adds a substantial BiLSTM encoder and attention module that is not controlled for in the comparisons. The abstract's 'state-of-the-art' claim is also stronger than the in-domain result supports, since the model ties Cai et al. (2018) on WSJ and only exceeds it on Brown.
major comments (4)
- [§2.2, Table 7] The comparison between 'Base' and 'Base + AMN' in Table 7 does not control for the additional model capacity introduced by the AMN. The AMN adds a 3-layer BiLSTM encoder LSTM_a with hidden size 512 (Table 1) and an attention module that are entirely absent from the base model. The observed gains of 0.3-0.5 F1 could therefore come from the extra encoder and attention machinery rather than from the semantic content of the associated labels. The paper should include a capacity-matched control in which an equivalently sized BiLSTM and attention head are added to the base model but the associated label information is withheld or randomized, so that the attribution of the improvement to the 'associated memory' is justified.
- [§3, Tables 5-7] The central improvements are reported without variance or significance testing; the paper states only that experiments were re-run 2-4 times and averaged. The differences are small (e.g., 3.2: 87.8 vs 88.3; Table 7: 86.0 vs 86.3 and 87.8 vs 88.3), and these magnitudes are plausibly within run-to-run noise for this type of model. The authors should report per-run results, standard deviations, and significance tests (or a bootstrap over prediction units) for the base-vs-AMN comparisons. In addition, Table 5 shows that random retrieval (RD, 88.1 F1) is nearly as effective as edit-distance retrieval (ED, 88.3 F1) and both improve over the base model, so the specific retrieval criterion is not demonstrated to be the key ingredient; the paper should address this explicitly, for example by testing whether random selection under the same capacity-matched setup still yields the gain.
- [Tables 5 and 7] There is an apparent inconsistency in the reported base-model F1 on the English development set: Table 5 reports the base model at 87.8, while Table 7 reports 'Base' (without ELMo) at 86.0 and 'Base + ELMo' at 87.8. Since the ablations in Section 3.2 do not explicitly state whether they include ELMo, the reader cannot determine which configuration these numbers correspond to. This ambiguity directly affects the interpretation of the ablation results and should be corrected with a clear statement of the ELMo setting for each table.
- [Abstract and §3.1] The abstract's claim that the full model 'reaches state-of-the-art' is overstated for the in-domain English setting. In Table 2, the model achieves 89.6 F1, which is exactly the same as Cai et al. (2018), and the only clear improvement is the +0.7 F1 on Brown in Table 3. The text should be revised to say that the model matches the current state of the art on in-domain English and improves on out-of-domain English, rather than implying a general state-of-the-art result.
minor comments (5)
- [§2.2] There are typos in the description of the memory encoder: 'BiSLTM' should be 'BiLSTM', and the Figure 1 caption refers to 'LSTM 1' where it should refer to 'LSTM_a'.
- [§3.4] The parenthetical in Section 3.4 appears to reverse the direction of the improvements: from Table 7, the AMN gain without ELMo is 0.3 (86.0 to 86.3) and with ELMo is 0.5 (87.8 to 88.3), but the text says 'a 0.5% enhancement on baseline over the 0.3% enhancement on ELMo baseline'.
- [Figures 2-5] In the current manuscript, the text in Figures 2-5 appears as unreadable private-use Unicode sequences rather than legible labels and numbers; these figures should be regenerated so that the memory-size curve, confusion matrices, and distance plots can actually be inspected by readers.
- [Figure 1] The example sentence in Figure 1 contains a spelling error: 'skiptical' should be 'skeptical'.
- [References] The reference for Cai et al. (2018) lists 'Proceedings of the 27th International Conference on Computational Linguistics (CoNLL)', but this work appeared at COLING 2018, not at CoNLL; the venue should be corrected.
Circularity Check
No circularity: AMN predicts with training-set labels on held-out test data and its gains are empirical ablations, not derived from its inputs.
full rationale
The paper's derivation chain is self-contained and does not reduce to its inputs. The AMN component is defined using sentences and gold labels from the training set only: associated sentence selection uses edit distance, WMD, SIF, or random distance over training sentences; inter-sentence attention aligns the target sentence with those retrieved training sentences; label merging combines retrieved training labels. At no point is the target sentence's own label used to compute the prediction, so the argument labeling is not self-definitional. Hyperparameters such as the distance method, merging method, and memory size are selected on the development set, and the headline results in Tables 2-4 are reported on the official CoNLL-2009 test splits, which are held out from the training memory. The comparisons in Tables 5-7 are empirical ablations; even the observation that random retrieval helps is an experimental finding, not an identity or a fitted-parameter tautology. The self-citations to Cai et al. 2018, He et al. 2018b, and Li et al. 2018 are used only as reported baselines or for standard components such as predicate disambiguation, and none of these citations is used to prove the AMN improvement. Any concern that the added LSTM_a encoder rather than the memory content drives the gain is a question of ablation completeness, not circularity, because the paper never defines the added encoder in terms of the test labels and does not fit anything to the test set and then call it a prediction. Therefore no circular step is present.
Assumptions & free parameters
free parameters (4)
- Memory size m =
4 (English), 5 (Chinese)
- Associated sentence selection strategy =
Edit distance (ED)
- Label merging strategy =
Average
- Architecture hyperparameters (d_re, d_pe, d_pos, d_le, d_ce, d_pred, d_ae, LSTM layers, hidden sizes, dropout… =
100, 100, 32, 100, 128, 16, 128, 2, 3, 512, 512, 0.1, 0.001
assumptions (5)
- domain assumption CoNLL-2009 gold labels are correct and the standard train/dev/test split is used.
- domain assumption Pretrained GloVe and ELMo representations are valid semantic features for SRL.
- domain assumption Sentences with small edit distance on POS tag sequences tend to share SRL label patterns.
- domain assumption The improvement from AMN is not an artifact of run-to-run variance.
- domain assumption Using training data as memory at test time does not leak test information.
Cite this review
Pith. "Pith review of Semantic Role Labeling with Associated Memory Network." pith.science (2026). https://pith.science/paper/SBKF2IEN
@misc{pith2026190802367,
author = {Pith},
title = {Pith review of: Semantic Role Labeling with Associated Memory Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/SBKF2IEN}},
note = {Machine review of arXiv:1908.02367}
}
read the original abstract
Semantic role labeling (SRL) is a task to recognize all the predicate-argument pairs of a sentence, which has been in a performance improvement bottleneck after a series of latest works were presented. This paper proposes a novel syntax-agnostic SRL model enhanced by the proposed associated memory network (AMN), which makes use of inter-sentence attention of label-known associated sentences as a kind of memory to further enhance dependency-based SRL. In detail, we use sentences and their labels from train dataset as an associated memory cue to help label the target sentence. Furthermore, we compare several associated sentences selecting strategies and label merging methods in AMN to find and utilize the label of associated sentences while attending them. By leveraging the attentive memory from known training data, Our full model reaches state-of-the-art on CoNLL-2009 benchmark datasets for syntax-agnostic setting, showing a new effective research line of SRL enhancement other than exploiting external resources such as well pre-trained language models.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated template generation for question answering over knowledge graphs. In Proceedings of the 26th International Conference on World Wide Web (WWW), pages 1191--1200
work page 2017
-
[4]
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In Proceedings of the International Conference on Learning Representations (ICLR)
work page 2017
-
[5]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[6]
Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (COLING), volume 1
work page 1998
-
[7]
Claire Bonial, Jena Hwang, Julia Bonn, Kathryn Conger, Olga Babko-Malaya, and Martha Palmer. 2012. English propbank annotation guidelines. Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder
work page 2012
-
[8]
Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018. A full end-to-end semantic role labeler, syntax-agnostic over syntax-aware? In Proceedings of the 27th International Conference on Computational Linguistics (CoNLL), pages 2753--2765
work page 2018
Show all 41 references
-
[9]
Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better ud parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pa...
2018
-
[10]
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of 5th International Conference on Learning Representations (ICLR)
2017
-
[11]
Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics (NoDaLiDa), pages 271--276
2017
-
[12]
a ckstr \
Nicholas FitzGerald, Oscar T \"a ckstr \"o m, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 960--970
2015
-
[13]
Jan Haji c , Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Ant \`o nia Mart \' , Llu \' s M \`a rquez, Adam Meyers, Joakim Nivre, Sebastian Pad \'o , Jan S t e p \'a nek, et al. 2009. The conll-2009 shared task: Syntactic and semantic dependencies in multi...
2009
-
[14]
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018 a . Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 364--369
2018
-
[15]
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pages 473--483
2017
-
[16]
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018 b . Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pages 2061--2071
2018
-
[17]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014 arXiv
-
[18]
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 957--966
2015
-
[19]
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2401--2411
2018
-
[20]
Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence
2019
-
[21]
Jind r ich Libovick \`y and Jind r ich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 196--202
2017
-
[22]
Feifan Liu, Jinying Chen, Abhyuday Jagannatha, and Hong Yu. 2016. Learning for biomedical information extraction: Methodological review of recent advances. arXiv preprint arXiv:1606.07993
2016 arXiv
-
[23]
Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL), pages 411--420
2017
-
[24]
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1506--1515
2017
-
[25]
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1400--1409
2016
-
[26]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543
2014
-
[27]
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H...
2018
-
[28]
Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James H Martin, and Daniel Jurafsky. 2005. Semantic role labeling using different syntactic views. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL), pages 581--588
2005
-
[29]
Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pages 1192--1202
2016
-
[30]
Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 407--413
2014
-
[31]
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5027--5038
2018
-
[32]
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS), volume 2, pages 2440--2448
2015
-
[33]
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 4929--4936
2018
-
[34]
Kristina Toutanova, Aria Haghighi, and Christopher D Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34(2):161--191
2008
-
[35]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NIP...
2017
-
[36]
Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916
2014 arXiv
-
[37]
Hai Zhao, Wenliang Chen, Jun'ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009 a . Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proceedings of the Thirteenth Conference on Computational Natural Language...
2009
-
[38]
Hai Zhao, Wenliang Chen, and Chunyu Kit. 2009 b . Semantic dependency parsing of nombank and propbank: An efficient integrated approach via a large-scale feature selection. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volu...
2009
-
[39]
Hai Zhao, Wenliang Chen, Chunyu Kit, and Guodong Zhou. 2009 c . Multilingual dependency learning: A huge feature engineering method to semantic dependency parsing. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL): Shared Task, pages 55--60
2009
-
[40]
Hai Zhao, Xiaotian Zhang, and Chunyu Kit. 2013. Integrative semantic dependency parsing via efficient large-scale feature selection. Journal of Artificial Intelligence Research, 46:203--233
2013
-
[41]
Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-I...
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.