Pith. sign in

REVIEW 3 major objections 7 minor 36 references

Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution

T0 review · 3 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that frozen embeddings from simple source tasks, especially paraphrase detection, can be repurposed for coreference resolution, with attention-based combination giving the best transfer.

desk verdict A useful empirical map of which frozen task embeddings help coreference, but the headline layer and combination gains are likely inflated by selecting configurations on the validation set. read the letter →

arxiv 2501.19316 v2 pith:NKIOTR7G submitted 2025-01-31 cs.CL

Tatiana Anikina , Arne Binder , David Harbecke , Stalin Varanasi , Leonhard Hennig , Simon Ostermann , Sebastian Möller , Josef van Genabith This is my paper · ORCID
classification cs.CL
keywords reverseprobingcoreferenceresolutiontaskembeddingstransferlearningattentionaggregationBERTclassifiersOntoNotes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a complex target task can be solved largely with frozen representations from simpler source tasks, a 'reverse' of classical probing. Using coreference resolution as the target, the authors show that embeddings from a BERT model fine-tuned on paraphrase detection transfer best, while named-entity-recognition embeddings transfer worst. They also find that representations from intermediate layers of fine-tuned source models are often more useful than final-layer outputs, and that combining several frozen embedding streams with a learned attention aggregator consistently improves performance. On the OntoNotes validation split, the best frozen-embedding setup reaches 69.56 F1, close to the 73.75 F1 of a fully trained coreference model. This matters because it maps which task knowledge transfers where and suggests a cheaper path to building task-capable representations.

What carries the argument

The central mechanism is a reverse-probing pipeline over frozen, fine-tuned BERT encoders. For each source task, hidden-layer outputs $H_s^{(l)}$ are extracted, optionally truncated to a chosen intermediate layer or formed by concatenating the last 4, 6, or 12 layers, and optionally L2-normalized; the streams are then merged token-wise either by the mean $E_T = \frac{1}{k}\sum_i H_{s_i}$ or by attention $E_T = \sum_i \alpha_i H_{s_i}$ with $\alpha_i = \mathrm{softmax}(W \cdot H_{s_i})$. The aggregated embeddings feed a trainable coreference head trained on OntoNotes while all source weights stay frozen. The attention aggregator is the load-bearing component: it learns which source and layer to trust, which explains why combinations and low-scoring sources gain most from it.

What would settle it

Take the same frozen source embeddings and the same trainable coreference head, split the data into train/validation/test instead of using one validation set for both selection and reporting, and compare configurations chosen on validation against their test F1. The central claim collapses if the best validation configuration is not the best on the test set, or if MRPC no longer beats NER once selection bias is removed.

Watch

Extended reading notes

Core claim

Frozen embeddings from simple source tasks can be repurposed for coreference resolution if extracted and combined correctly. Across single-model probes, paraphrase detection (MRPC) and vanilla BERT give the highest F1 (64.32 and 64.01), relation extraction and question answering sit in the middle (52.43 and 47.51), and NER is lowest (36.03). Truncating each source model to a non-final layer, layer 9 for MRPC and RE, layer 8 for QA, layer 6 for NER, improves every model, with NER gaining up to 26.2 points. Attention-based token-wise aggregation outperforms simple averaging, especially for low-scoring sources such as NER (+19.7) and QA (+9), and combining source tasks with attention plus truncation gives the best overall validation score of 69.56 F1 from MRPC+NER+QA. The authors interpret this as evidence that semantic-similarity knowledge is most aligned with coreference, while entity-span knowledge encoded by NER is less transferable unless reweighted by attention.

Load-bearing premise

The load-bearing premise is that the OntoNotes validation set is a fair place to both choose the best layer, normalization, and aggregation settings and to report the final F1; if the best settings on validation do not transfer to unseen test data, the ranking of source tasks and the claimed gains from attention and truncation are overstated.

Editorial extensions

If this is right

  • If these results hold, task selection for embedding reuse is not neutral: semantic-similarity training (paraphrase detection) transfers to coreference far better than named-entity recognition.
  • Intermediate layers of fine-tuned transformers should be treated as reusable assets; final task-specialized layers can hurt a downstream target, so layer truncation is a cheap and effective adjustment.
  • Attention-based aggregation of frozen embeddings from several tasks is a viable way to approach a fully trained model, reaching 69.56 F1 against a 73.75 upper bound.
  • Combining several hidden layers (last 4 or 6) is helpful, but pooling all 12 layers is not, so more context does not always transfer better.
  • The reverse-probing setup gives a low-cost way to compare source tasks for a target before committing to full finetuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors did not run is to apply the same protocol to other complex targets such as semantic role labeling or discourse relations; if the MRPC-first ranking repeats, semantic similarity may be a general-purpose predictor of transferability.
  • The observation that QA transfers better than NER despite lower cosine similarity with the target suggests representation-distance measures are incomplete; a learned attention projection may be capturing a more useful notion of task relatedness.
  • The validation-set dependency flagged in the paper could be tested by re-running layer and aggregation selection on a training split and reporting test F1; that would separate genuine transfer from selection on the evaluation set.
  • Because the authors note the added overhead of combining layers and models, a practical follow-up is to prune the attention aggregator to a single per-source weight after training, reducing inference cost while keeping most of the gain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. Reverse Probing introduces a probing-style framework in which frozen BERT embeddings from models fine-tuned on simpler source tasks (MRPC, NER, RE, QA) are used as token representations for coreference resolution. The paper systematically varies the extraction layer (full vs. truncated), optionally applies L2 normalization, concatenates multiple hidden layers, and compares mean versus attention-based aggregation. All experiments are evaluated on the OntoNotes validation split, with the coreference head trained on top of the frozen embeddings. The central claims are that semantic-similarity source tasks (MRPC) transfer best, intermediate layers often outperform final layers, and combining multiple source models, especially with attention aggregation, improves coreference F1 up to 69.56 versus 73.75 for a fully trained model.

Significance. If the empirical claims are robust, this is a useful and clearly presented contribution: it extends probing methodology toward knowledge transfer from simpler to more complex tasks, provides a systematic comparison of source tasks and embedding manipulations, and the authors publicly release their code. The paper also reports seed-averaged scores with standard deviations and includes a limitations section. However, the headline quantitative claims rest on a selection-on-validation protocol, so the reported gains and rankings are not unbiased estimates of transfer performance and need verification on a held-out test set before they can be accepted at face value.

major comments (3)
  1. [§4.4, Tables 1–3] The central claims about intermediate layers and truncation gains are based on selecting the best truncation layer on the OntoNotes validation set and then reporting F1 on that same validation set. Section 4.4 states that after comparing layers 6 through 12, the best performing layer was selected for each source model, and Tables 1–3 report the resulting 'trunc' scores with the chosen layer as a subscript. Early stopping in §4.1 also uses the validation split. This is selection-on-validation: the reported numbers are effectively maxima over a grid of layer choices, normalization choices, and aggregation methods, not the expected performance of a fixed configuration. The +26.2% NER gain from truncation and the conclusion that intermediate layers outperform final layers could therefore be inflated. The authors should evaluate the selected configurations on the held-out CoNLL-2012 test set, or use nested validation, for at least the central single-model and combination comparisons.
  2. [§4.6, Tables 2–3] The claim that 'combinations of two models always outperform single models' and the reported combination gains (e.g., +13.23% for NER+MRPC) are computed from configurations that were themselves selected on the validation set. The seed-level standard deviations (roughly 0.2–3 F1) capture only variation across runs of a fixed configuration, not the variance introduced by choosing among many configurations on the same data. Consequently, the reported ranking of single vs. combined models, and the ordering among combinations, may not generalize to unseen data. A held-out test evaluation, or at least a nested-validation estimate, is needed to support the combination claims.
  3. [§4.3 and §4.4] The comparison between mean and attention aggregation confounds the aggregation mechanism with added trainable parameters: attention introduces a learned projection while mean aggregation does not. The paper acknowledges this in the text ('we get by simply adding additional projections'), but the conclusion that 'attention-based aggregation yields substantial gains' is still stated as an effect of the aggregation method. To make this claim load-bearing, the authors should include a control condition with matched parameters, such as a linear projection followed by mean pooling, or explicitly frame the comparison as 'attention with extra parameters vs. mean without extra parameters.'
minor comments (7)
  1. [Abstract and §3.2] There are typos: the abstract contains 'tasks tasks' and §3.2 contains 'quesion answering'.
  2. [§4.4] The text reads 'NE+MRPC' in the normalization discussion; this should be 'NER+MRPC'.
  3. [§4.2 and Table 1] The text reports BERT and MRPC scores of 64.01 and 64.32 F1 from Figure 2, while Table 1's 'full mean' column reports 64.95 and 61.16 for the same settings. The discrepancy should be clarified, for example by stating which configuration Figure 2 uses.
  4. [§2] The mean aggregation formula is missing the averaging factor in the displayed equation, and the notation H_s in the text omits the layer superscript used elsewhere.
  5. [Figures 5–8] The text refers to Figures 5, 6, 7, and 8, but the figure numbering in the submission is inconsistent and some captions are incomplete; the authors should ensure all figures are numbered and captioned consistently.
  6. [§4.4] The sentence 'RE+QA pair gains +8.17% F1 with mean aggregation and +2.84% with attention aggregation when both models are truncated' should state the baseline to which these gains are relative.
  7. [Limitations] The Limitations section lists model architecture, source-task coverage, and efficiency limitations, but it does not mention the validation-set selection issue; adding an explicit statement about this would help readers calibrate the reported results.

Circularity Check

1 steps flagged · score 6.0 of 10

Intermediate-layer superiority is partly built into the validation-based layer selection, so the reported truncation gains are not independent evidence.

  1. fitted input called prediction [Section 4.4, Tables 1-3]
    "after comparing the embeddings from layer 6 to 12 we found that the best performing layer on our probing task was typically not the final one. E.g., it was layer 9 for MRPC and RE, layer 8 for QA and 6 for NER (see Figure 5). Tables 1, 2 and 3 show the detailed comparisons between the original (full) model outputs as well as the normalized and truncated (to the “best” layer) versions ... Truncation seems to be a good strategy for embedding aggregation and consistently yields best results across different settings. Truncation improves NER by up to +26.2%."

    The 'best' truncation layer is chosen on the OntoNotes validation set as the highest-F1 layer among layers 6 through 12, and the 'full' setting is layer 12, which is one of the candidates. Therefore the reported validation F1 for 'trunc' is, by construction, at least as large as the 'full' F1 for the same model and seed, since it is the maximum of a set containing that value. The conclusion that truncation consistently helps and that intermediate layers outperform final layers is thus a restatement of the selection criterion, not an independent empirical result.

full rationale

This paper is an empirical study, not a formal derivation, so there is no equation-level self-reference or load-bearing self-citation chain. The main circularity concern is concentrated in the truncation/intermediate-layer claim (Section 4.4 and Tables 1-3). The authors select, for each source model, the layer from 6 to 12 with the highest validation F1 and then report that selected score as evidence that truncation improves performance over the final layer. Since the final layer (layer 12) is among the candidates, the reported improvement is non-negative by construction; the 'intermediate layers often outperform final layers' finding is therefore partly a selection artifact rather than an independent discovery. Other findings, such as the benefit of attention-based aggregation, the gains from combining multiple source models, and layer-concatenation effects, are genuine comparisons across fixed, pre-specified configurations and do not exhibit circularity. With a held-out test set or pre-registered layer choices, the truncation claim would be evaluable; as reported, the central truncation result reduces to the selection criterion, warranting a partial-circularity score of 6.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on a small number of fitted choices (truncation layer, normalization, number of layers) that are selected on the target validation set, plus the domain assumption that frozen task-tuned embeddings are useful. No new physical or conceptual entities are introduced. The main unacknowledged burden is the validation-set selection bias, which is logged as an axiom and a red flag.

free parameters (4)
  • Truncation layer per source model = MRPC:9, NER:6, RE:9, QA:8, BERT:10, Coref:12
    Selected by highest validation F1 in Section 4.4; not predicted a priori. This directly affects the 'intermediate layer' finding.
  • L2 normalization flag per experiment = on/off per model and aggregation method
    Applied only when it improved validation F1 (Tables 1-3), introducing selection on the evaluation set.
  • Number of concatenated layers = 4, 6, or 12
    Chosen in Section 4.5 to explore embedding context; best reported on validation, with 4 or 6 layers generally best.
  • Learning rate = 1e-4
    Tuned on frozen BERT only (Section 4.1) and reused for all models and combinations.
assumptions (3)
  • domain assumption Frozen BERT source models fine-tuned on simple tasks retain transferable knowledge for coreference resolution.
    Core premise of reverse probing, stated in the introduction and Section 2. The entire experiment depends on this being true.
  • domain assumption Token embeddings from different BERT-based source models share a common tokenizer and can be aggregated token-wise via mean or attention.
    Required for the aggregation formulas in Section 2. All models are bert-base-cased, so this holds approximately, but it is an unstated operational premise.
  • ad hoc to paper Validation-set selection of truncation layer and aggregation settings provides an unbiased estimate of transfer performance.
    The paper reports final F1 on the same validation set used to pick the best layer and normalization (Section 4.4, Tables 1-3). This assumption is load-bearing for the quantitative claims but is not validated with held-out data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution." pith.science (2026). https://pith.science/paper/NKIOTR7G

@misc{pith2026250119316,
  author       = {Pith},
  title        = {Pith review of: Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKIOTR7G}},
  note         = {Machine review of arXiv:2501.19316}
}
read the original abstract

In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer.

Figures

Figures reproduced from arXiv: 2501.19316 by the authors.

Figure 1
Figure 1. Probing workflow with Coreference Resolu [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Source task models: CorefTarget, BERT, MRPC, RE, QA, NER, SemTag, Chunking, NER-dslim, POS [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average cosine similarity between the embeddings of the source tasks and the target coreference task, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Mean vs attention aggregation (full setting) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Source task model performance truncated to the best layer (in parentheses) with mean aggregation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Single models: full vs normalized vs truncated [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pairs of models: full vs normalized vs truncated [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Single and combined (2x) models proved to be successful only for single models. Finally, it would be interesting to replicate our ex￾periments on larger (non-BERT) models and tasks (e.g., semantic role labeling, discourse relation clas￾sification etc.). We hope that ou…
Figure 9
Figure 9. Figure 9: MRPC+RE+NER with attention aggregation (full setting) [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 7 canonical work pages

  1. [1]

    Amit Bagga and Breck Baldwin. 1998. https://aclanthology.org/C98-1012 Entity-based cross-document coreferencing using the vector space model . In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics

    work page 1998

  2. [2]

    Robert Belanec, Simon Ostermann, Ivan Srba, and Maria Bielikova. 2024. Task prompt vectors: Effective initialization through multi-task soft-prompt transfer. arXiv preprint arXiv:2408.01119

    arXiv 2024

  3. [3]

    Yonatan Belinkov. 2022. https://doi.org/10.1162/coli_a_00422 Probing classifiers: Promises, shortcomings, and advances . Computational Linguistics, 48(1):207--219

    doi:10.1162/coli_a_00422 2022

  4. [4]

    Bernd Bohnet, Chris Alberti, and Michael Collins. 2023. https://doi.org/10.1162/tacl_a_00543 Coreference resolution through a seq2seq transition-based system . Transactions of the Association for Computational Linguistics, 11:212--226

    doi:10.1162/tacl_a_00543 2023

  5. [5]

    Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2023. Language and task arithmetic with parameter-efficient layers for zero-shot summarization. arXiv preprint arXiv:2311.09344

    arXiv 2023

  6. [6]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. https://doi.org/10.18653/v1/P18-1198 What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    doi:10.18653/v1/p18-1198 2018

  7. [7]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, ...

    doi:10.18653/v1/n19-1423 2019

  8. [8]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. 2005. https://aclanthology.org/I05-5002 Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005)

    2005

Show all 36 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    John Hewitt and Percy Liang. 2019. https://doi.org/10.18653/v1/D19-1275 Designing and interpreting probes with control tasks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua...

    2019 doi

  3. [11]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. https://openreview.net/forum?id=6t0Kwf8-jrj Editing models with task arithmetic . In The Eleventh International Conference on Learning Representations

    2023

  4. [12]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  5. [13]

    Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/D17-1018 End-to-end neural coreference resolution . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188--197, Copenhagen, Denmark. Associ...

    2017 doi

  6. [14]

    Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. https://doi.org/10.18653/v1/W19-4825 Open sesame: Getting inside BERT `s linguistic knowledge . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241--253, Florence,...

    2019 doi

  7. [15]

    Xiaoqiang Luo. 2005. https://aclanthology.org/H05-1004 On coreference resolution performance metrics . In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25--32, Vancouver, British Columbia, Canada. ...

    2005

  8. [16]

    Giuliano Martinelli, Edoardo Barba, and Roberto Navigli. 2024. https://doi.org/10.18653/v1/2024.acl-long.722 Maverick: Efficient and accurate coreference resolution defying recent trends . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  9. [17]

    Michael S Matena and Colin A Raffel. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/70c26937fbf3d4600b69a129031b66ec-Paper-Conference.pdf Merging models with fisher-weighted averaging . In Advances in Neural Information Processing Systems, volume 35, pages 17...

    2022

  10. [18]

    Simon Ostermann, Sheng Zhang, Michael Roth, and Peter Clark. 2019. https://doi.org/10.18653/v1/D19-6007 Commonsense inference in natural language processing ( COIN ) - shared task report . In Proceedings of the First Workshop on Commonsense Inference in Natural Language Proces...

    2019 doi

  11. [19]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. https://aclanthology.org/W12-4501 C o NLL -2012 shared task: Modeling multilingual unrestricted coreference in O nto N otes . In Joint Conference on EMNLP and C o NLL - Shared Task , page...

    2012

  12. [20]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  13. [21]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don`t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 78...

    2018 doi

  14. [22]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  15. [23]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  16. [24]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019 a . https://doi.org/10.18653/v1/P19-1452 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Italy. Association f...

    2019 doi

  17. [25]

    Thomas McCoy , Najoung Kim, Benjamin Van Durme , Samuel R

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy , Najoung Kim, Benjamin Van Durme , Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019 b . What do you learn from context? probing for sentence structure in contextualized word representations. ...

    2019

  18. [26]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. https://www.aclweb.org/anthology/W03-0419 Introduction to the C o NLL -2003 shared task: Language-independent named entity recognition . In Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003 ...

    2003

  19. [27]

    Lisa Torrey and Jude Shavlik. 2010. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pages 242--264. IGI global

    2010

  20. [28]

    Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. https://aclanthology.org/M95-1005 A model-theoretic coreference scoring scheme . In Sixth Message Understanding Conference ( MUC -6): Proceedings of a Conference Held in C olumbia, M aryland...

    1995

  21. [29]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. http://aclweb.org/anthology/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationa...

    2018

  22. [30]

    Liyan Xu and Jinho D. Choi. 2020. https://www.aclweb.org/anthology/2020.emnlp-main.686 Revealing the myth of higher-order inference in coreference resolution . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527--8533. ...

    2020

  23. [31]

    Jun Yu, Yutong Dai, Xiaokang Liu, Jin Huang, Yishan Shen, Ke Zhang, Rong Zhou, Eashan Adhikarla, Wenxuan Ye, Yixin Liu, et al. 2024. Unleashing the power of multi-task learning: A comprehensive survey spanning traditional, deep, and pretrained foundation model eras. arXiv prep...

    2024 arXiv

  24. [32]

    Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/D17-1004 Position-aware attention and supervised data improve slot filling . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Proces...

    2017 doi

  25. [33]

    Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pre-trained language models. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9733--9740

    2020

  26. [34]

    Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler

    Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. https://doi.org/10.1109/ICCV.2015.11 Aligning books and movies: Towards story-like visual explanations by watching movies and reading books . In 2015 IEEE I...

    2015 doi

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.