Effective Use of Transformer Networks for Entity Tracking

Aditya Gupta; Greg Durrett

REVIEW 4 major objections 4 minor 29 references

Effective Use of Transformer Networks for Entity Tracking

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The way a transformer is pointed at an entity decides whether it can track that entity in procedural text.

desk verdict Useful empirical templates and honest negative analysis, but the entity-conditioning mechanism is not fully isolated from per-entity compute. read the letter →

arxiv 1909.02635 v1 pith:W4ZRHNQZ submitted 2019-09-05 cs.CL

Aditya Gupta , Greg Durrett This is my paper · ORCID

classification cs.CL

keywords entitytrackingproceduraltexttransformernetworksself-attentioninputconditioningingredientdetectionstatechangepre-trainedlanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Procedural text—recipes, scientific process descriptions—requires tracking how entities change as actions apply to them. This paper asks whether pre-trained transformers (GPT, BERT) can do that entity tracking. The answer is a qualified yes: if you run the transformer over the whole paragraph and then try to read off an entity's state, the model does no better than simple baselines; but if you restructure the input so the target entity appears at the very start, before the process steps, the same transformer achieves state-of-the-art results on both ingredient detection in recipes and state-change question answering in scientific processes. The qualification matters: analysis shows the model largely exploits verb semantics and exact mentions rather than forming representations of intermediate compositions like 'the egg mixture.' The paper's contribution is thus both a practical input-formatting recipe and a caution that high benchmark scores do not equal deep entity-state tracking.

What carries the argument

The load-bearing mechanism is the entity-first input template: the input sequence starts with a [START] token, then the target entity's name, then a [SEP] token, followed by the process steps, with a [CLS] token after each sentence to anchor predictions. This reorders the standard transformer input so that the entity is always visible during self-attention, effectively letting the attention mechanism build an entity-centric representation of every token. For ProPara, the [CLS] representations produce tag potentials fed into a conditional random field, and Viterbi decoding enforces valid state-change sequences (create → move → destroy). The paper's ablation shows that removing the ingredient from the input drops F1 from 81.96 to 63.77, and removing previous context drops it to 71.67, demonstrating that the entity conditioning and the surrounding context both carry the performance.

What would settle it

Train an entity-conditioned model and a post-conditioning model with identical compute (same number of transformer passes) and identical domain fine-tuning, and compare on RECIPES and ProPara; if the post-conditioning model matches or exceeds the entity-first F1, the paper's restructuring claim fails. Alternatively, test on a held-out set of recipe steps where the target ingredient appears only as part of a named composition (e.g., 'the egg mixture') with uninformative verbs: if accuracy does not fall far below the reported 82.50 F1, the paper's shallow-cue conclusion would be wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that entity conditioning—placing the target entity at the beginning of the transformer input, with prediction anchors after each sentence—is what makes pre-trained transformers effective at entity tracking. Post-conditioning, where the transformer encodes the paragraph without knowing the entity and an entity embedding is combined afterward (via concatenation or bilinear attention), underperforms rule-based baselines such as predicting that an ingredient stays present after its first explicit mention. Entity-conditioned GPT achieves 82.50 F1 on RECIPES ingredient detection versus 74.60 for that baseline, and on ProPara raises exact-step state-change accuracy (Cat-2) to 52.21 versus 47.09 for the prior neural CRF model, with BERT slightly higher. The paper argues that a left-to-right transformer naturally builds a target-specific representation for every token when the entity leads the input, whereas post-hoc conditioning cannot extract what the unconditional self-attention did not encode. At the same time, the paper establishes a negative result: the models still attend mostly to shallow cues—verbs indicating actions and exact entity mentions—and fail on cases requiring tracking of intermediate compositions, achieving only 51.1% accuracy on 0-to-1 combined-ingredient transitions.

Load-bearing premise

The claim that input restructuring causes the performance gain rests on comparing entity-conditioned models (which run one transformer pass per entity and use extra domain-specific language-model fine-tuning) against post-conditioning models (which do neither); if those extra compute and data advantages were removed, the restructuring might account for less of the gain.

Editorial extensions

If this is right

Entity-first input conditioning is a cheap, architecture-agnostic way to make pre-trained transformers work on entity tracking: it needs no new modules, only a different input layout, and it lifts RECIPES F1 from 74.60 (First Occurrence baseline) to 82.50.
Post-conditioning (encode first, condition later) is not viable for these tasks; the paper's comparison predicts that other entity-intensive tasks will also need entity-aware input construction rather than late fusion.
Document-level entity-first variants outperform sentence-level ones, so letting the transformer see the whole process while keeping the entity at the front captures more of the long-range state dynamics.
State-of-the-art accuracy on ProPara Cat-2 (exact step of creation, movement, or destruction) shows the model can localize change points, but per-type scores reveal movement events remain hard, pointing to compositional tracking as the bottleneck.
Because verbs and exact mentions drive the predictions, gains from domain-specific language-model fine-tuning concentrate in combined recall; models trained from scratch without extra unlabeled data lose most of that advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A compute-matched comparison (same per-entity transformer passes and same LM fine-tuning for both input layouts) would be needed to fully separate the effect of restructuring from extra compute and domain adaptation; the paper's setup does not isolate the mechanism.
The input-formatting insight likely transfers to other tasks where a model must reason about one object in a scene or one row in a table: putting the target identifier at the start of the sequence may be a general prompt-engineering principle.
The shallow-cue finding predicts that adversarial or rephrased inputs that hide the verb or replace the ingredient name with a composition alias should sharply drop accuracy; such stress tests would quantify how much of the benchmark score is genuine state tracking versus lexical pattern matching.
The 51.1% accuracy on combined-ingredient transitions suggests an explicit mechanism—for example, predicting composition membership and carrying a per-entity mixture state—might be a more direct route than relying on self-attention to discover compositionality.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — here is my read. The paper's practical takeaway is sound: if you want a pretrained transformer to track entities in procedural text, structure the input around the entity and predict at a special token; post-conditioning after an entity-agnostic pass is weak. The four-template comparison, the ablations, and the honest analysis of verb reliance are genuinely useful, and the code is out. The ProPara gains over NCET are on the standard split and look real. I also think the negative result — post-conditioning underperforms a First Occurrence baseline — is a useful caution for the field.

Where I push back is the causal story. The entity-conditioned models rerun the transformer once per entity (and per timestep for sentence-level variants), which the paper acknowledges, and the best variants also get domain-specific LM fine-tuning. So part of the gap could be compute or adaptation, not the input template per se. The stress-test note sharpens this: the sentence-level entity-last variant scores 79.33 even though the entity appears only after Step t and cannot steer earlier process-token hidden states in a unidirectional transformer. That means per-entity inference alone, with the entity near the prediction token, captures most of the benefit. It does not invalidate the paper, but it does mean the proposed mechanism — guiding self-attention over process tokens — is not isolated. A compute-matched baseline and a variant that feeds the entity to the CLS without rerunning per entity would settle it.

Other soft spots are proportionate. No error bars anywhere, so the small differences between templates (e.g., 81.96 vs 81.44) may be noise. The RECIPES SOTA comparison to Bosselut et al. is explicitly non-comparable, and the paper is honest about that, so it should be framed as a strong result on their split rather than SOTA. The ablation w/o context doing 71.67 F1 shows how much the task can be solved by local cues, which the authors themselves emphasize. The citation pattern is fine; using their own NCET as baseline on ProPara is standard and not circular.

Overall, this is a solid empirical paper worth a serious referee. I would accept it with requested revisions, mainly a matched-compute analysis and uncertainty estimates. I would cite the template finding and the post-conditioning negative result in my own work. It belongs on the reading-group list.

Referee Report

4 major / 4 minor

Summary. The paper studies how well pre-trained transformer models can track entities in procedural text, using RECIPES ingredient detection and ProPara state-change prediction as testbeds. It first shows that standard 'post-conditioning' use of GPT (reading off entity states from an entity-agnostic encoding via [CLS] or bilinear attention) underperforms a simple First-Occurrence baseline. It then proposes 'entity-conditioned' input templates that place the target entity before or after the process text and make predictions at [CLS] tokens, reporting large F1 gains over the post-conditioning models (from about 68 to 80-82 on RECIPES) and strong results on ProPara. The paper includes ablations showing the target ingredient matters and that the model relies primarily on verb semantics, and it candidly concludes that the models do not form deep representations of intermediate entity states.

Significance. If the central causal claim were cleanly established, the paper would make a useful contribution: it demonstrates a simple input-restructuring recipe that substantially improves transformer-based entity tracking and provides an honest analysis of what the models actually learn. Strengths include released code, multiple ablations, an ingredient-identity control, and a candid statement of the models' reliance on shallow cues. However, the significance is currently limited by three issues: the RECIPES 'state-of-the-art' claim is made without a same-test-set comparison to prior neural work; the entity-conditioning gains are not isolated from a large increase in per-entity compute and a different training objective; and the sentence-level entity-last variant's strong performance is in tension with the proposed 'guiding self-attention' mechanism. These are addressable with additional controlled experiments and revised claims.

major comments (4)

[Abstract and Section 4.2.1] The paper claims 'state-of-the-art results' on RECIPES, but the only prior neural model on this dataset (Bosselut et al., 2018) is not evaluated on the same test set; the manuscript explicitly states 'though these are not the exact same test set.' Without a like-for-like comparison, the SOTA claim over prior work is not substantiated. Please either provide a comparable evaluation on the same split or rephrase the claim to describe strong results on the authors' own split.
[Section 4 and Table 3] The main comparison confounds input structuring with per-entity compute and training objective. The entity-conditioned models rerun the transformer once per entity, as acknowledged in Section 4, and their training uses the additional LM loss λL_lm, whereas the post-conditioning models use a single forward pass and do not appear to use this objective. The F1 jump from 67.60/68.04 to 79.33-82.50 could therefore be partly due to additional forward passes or auxiliary loss rather than to the proposed entity-conditioned attention. A compute-matched or parameter-matched post-conditioning baseline (e.g., running the transformer once per entity with the same templates but no entity token, or otherwise controlling for the number of forward passes) is needed to attribute the gain to the mechanism.
[Section 4, Table 1, and Table 3 (ET_GPT_S_L)] The sentence-level entity-last variant places the target entity after all process tokens and immediately before [CLS]. In the unidirectional GPT architecture, the process-token hidden states cannot be conditioned on the target entity at all, because the entity appears later in the sequence; only the [CLS] token can attend to it. Yet ET_GPT_S_L reaches 79.33 F1, far above the post-conditioning models. This is not explained by the paper's stated mechanism that the entity guides the self-attention of the process tokens, and it suggests a simpler explanation: putting the entity near the classification token is what matters. Please test this directly, for example by probing the process-token representations in the entity-last variant or by adding a control where the entity is appended to the post-conditioning input at the [CLS] position.
[Tables 3 and 5] All results are reported from single runs, with no error bars, multiple seeds, or significance tests. Given that some headline comparisons are small (e.g., ET_BERT vs ET_GPT_D_F on ProPara: 63.07 vs 62.87 Ma-Avg) and the test sets are small, the reader cannot assess whether the reported ordering of variants is robust. Please report means and standard deviations over multiple seeds, or at least provide significance tests for the main comparisons.

minor comments (4)

[Section 4.2.2] Typo: 'increaesd' should be 'increased'.
[Section 2] Typo: 'entites' should be 'entities'.
[Table 3] The entity-centric model labels contain odd symbols (e.g., 'ET_GPT S© L©'); these should be plain subscripts such as ET_GPT_S_L for readability.
[Section 4.1] The source and size of the unlabeled recipe corpus used for domain-specific LM fine-tuning are not described; please cite the corpus and specify the training split used.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the only self-citations are baseline comparisons and task setup, not load-bearing assumptions.

full rationale

The paper's derivation chain is empirical and self-contained: it evaluates pre-trained transformer architectures on RECIPES and ProPara against rule-based baselines, post-conditioning baselines, and published prior systems. The only self-citations are to Gupta and Durrett (2019), used to define the ProPara structured-prediction task and as the NCET baseline; e.g., Section 4.3.1 states 'Our prior work (Gupta and Durrett, 2019) proposed a structured model for the task that achieved state-of-the-art performance,' but that prior model is an input comparison, not an assumed conclusion. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no input template is defined in terms of the target state label. The paper's own analysis (e.g., verb-ablation results in Table 9) concedes that performance is driven by shallow lexical clues, which undercuts the explanatory claim but is an empirical limitation, not circularity. The compute/parameter mismatch between entity-conditioned and post-conditioned models is a confound for attribution, not a circular derivation. Hence no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claims are empirical; no new entities are introduced. The free parameters are standard training and architecture choices, several unreported. The axioms are dataset validity, transfer from pretrained LMs, and faithfulness of interpretability analyses, all domain assumptions rather than mathematical axioms.

free parameters (4)

lambda (language model loss weight)
In L_total = L_task + lambda * L_lm, the weight lambda is chosen by hand and not reported; it controls the balance between task and LM fine-tuning and is not varied systematically.
unlabeled recipe corpus size for LM fine-tuning = 20k and 50k sentences
Table 7 shows performance depends on the amount of external recipe text used for domain-specific LM fine-tuning; the choice of 20k/50k is arbitrary.
from-scratch transformer architecture = 8 heads, 8 layers, 512 embedding size
Used for the no-pre-training ablation in Table 7; chosen by hand, not justified.
unspecified fine-tuning hyperparameters (learning rate, batch size, epochs)
The paper omits these values, yet the central results depend on them; a competent re-implementation would need to tune them.

assumptions (5)

domain assumption Gradient-based input attribution (following Sundararajan et al. 2017 and Jain and Wallace 2019) reflects the features the model relies on.
The paper uses gradient analysis to conclude the model attends to verbs and entity mentions; this assumes attribution methods are faithful, which is contested in the cited literature itself.
domain assumption The annotated labels in RECIPES and ProPara correctly represent ground-truth entity states.
All performance claims are measured against these datasets; if labels are noisy or ambiguous, F1 comparisons are affected.
domain assumption Pre-trained GPT/BERT weights transfer to procedural text and provide useful verb semantics.
The main models initialize from GPT/BERT; Table 7 shows external pretraining data matters, so the method relies on this transfer.
standard math Viterbi decoding with the CRF enforces the correct state-transition structure for ProPara.
The structured prediction uses a CRF over state tags; this assumes the existence-cycle constraints are the right model for scientific processes.
domain assumption The train/dev/test split (600/100/175 recipes) is representative and not leaked.
The RECIPES results, including the SOTA claim, depend on this split; the paper does not describe how the split was constructed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Effective Use of Transformer Networks for Entity Tracking." pith.science (2026). https://pith.science/paper/W4ZRHNQZ

@misc{pith2026190902635,
  author       = {Pith},
  title        = {Pith review of: Effective Use of Transformer Networks for Entity Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4ZRHNQZ}},
  note         = {Machine review of arXiv:1909.02635}
}

read the original abstract

Tracking entities in procedural language requires understanding the transformations arising from actions on entities as well as those entities' interactions. While self-attention-based pre-trained language encoders like GPT and BERT have been successfully applied across a range of natural language understanding tasks, their ability to handle the nuances of procedural texts is still untested. In this paper, we explore the use of pre-trained transformer networks for entity tracking tasks in procedural text. First, we test standard lightweight approaches for prediction with pre-trained transformers, and find that these approaches underperform even simple baselines. We show that much stronger results can be attained by restructuring the input to guide the transformer model to focus on a particular entity. Second, we assess the degree to which transformer networks capture the process dynamics, investigating such factors as merged entities and oblique entity references. On two different tasks, ingredient detection in recipes and QA over scientific processes, we achieve state-of-the-art results, but our models still largely attend to shallow context clues and do not form complex representations of intermediate entity or process state.

Figures

Figures reproduced from arXiv: 1909.02635 by the authors.

**Figure 1.** Process Examples from (a) RECIPES as a binary classification task of ingredient detection, and (b) PROPARA as a structured prediction task of identifying state change sequences. Both require cross-sentence reasoning, such as knowing what components are in a mixture and understanding verb semantics like combine. ing tasks we study, we perform additional analysis and find that these tasks still do not encourage tran… view at source ↗

**Figure 2.** Post-conditioning entity tracking models. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Entity conditioning model for guiding self [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Gradient of the classification loss of the gold class with respect to inputs when predicting the status [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Antoine Bosselut, Corin Ennis, Omer Levy, Ari Holtzman, Dieter Fox, and Yejin Choi. 2018. Simulating Action Dynamics with Neural Process Networks . In Proceedings of the International Conference on Learning Representations (ICLR)

work page 2018
[4]

Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. Neural Text Generation in Stories Using Entity Representations as Context . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (ACL): Human Language Technologies

work page 2018
[5]

Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

work page 2018
[6]

Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2019. Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension . In Proceedings of the International Conference on Learning Representations (ICLR)

work page 2019
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2019
[8]

Aditya Gupta and Greg Durrett. 2019. Tracking Discrete and Continuous Entity State for Process Understanding . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Workshop on Structure Predictions for NLP

work page 2019

Show all 29 references

[9]

Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the World State with Recurrent Entity Networks . In Proceedings of the International Conference on Learning Representations (ICLR)

2017
[10]

Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

2019
[11]

Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. 2017. Dynamic Entity Representations in Neural Language Models . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

2017
[12]

Weld, Luke Zettlemoyer, and Omer Levy

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. http://arxiv.org/abs/1907.10529 SpanBERT: Improving Pre-training by Representing and Predicting Spans

2019 arXiv
[13]

Ben Kantor and Amir Globerson. 2019. Coreference Resolution with Entity Equalization . In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

2019
[14]

Chlo \'e Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally Coherent Text Generation with Neural Checklist Models . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

2016
[15]

Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end Neural Coreference Resolution . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

2017
[16]

Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-Order Coreference Resolution with Coarse-to-Fine Inference . In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL)

2018
[17]

Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, and Tetsuro Sasada. 2014. Flow Graph Corpus from Recipe Texts . In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC)

2014
[18]

Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models . In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

2017
[19]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations . In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL)

2018
[20]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training . URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf

2018
[21]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf

2019
[22]

Altaf Rahman and Vincent Ng. 2009. Supervised Models for Coreference Resolution . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

2009
[23]

Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Query-Reduction Networks for Question Answering . In Proceedings of the International Conference on Learning Representations (ICLR)

2017
[24]

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks . In Proceedings of the International Conference on Machine Learning (ICML), pages 3319--3328

2017
[25]

Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about Actions and State Changes by Injecting Commonsense Knowledge . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

2018
[26]

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019 a . BERT Rediscovers the Classical NLP Pipeline . In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

2019
[27]

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019 b . https://openreview.net/forum?id=SJzSgnRcKX What do you learn from context? Probing for sentence structure in con...

2019
[28]

Rush, and Stuart M

Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning Global Features for Coreference Resolution . In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL)

2016
[29]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. http://arxiv.org/abs/1906.08237 XLNet: Generalized Autoregressive Pretraining for Language Understanding

2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Antoine Bosselut, Corin Ennis, Omer Levy, Ari Holtzman, Dieter Fox, and Yejin Choi. 2018. Simulating Action Dynamics with Neural Process Networks . In Proceedings of the International Conference on Learning Representations (ICLR)

work page 2018

[4] [4]

Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. Neural Text Generation in Stories Using Entity Representations as Context . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (ACL): Human Language Technologies

work page 2018

[5] [5]

Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

work page 2018

[6] [6]

Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2019. Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension . In Proceedings of the International Conference on Learning Representations (ICLR)

work page 2019

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2019

[8] [8]

Aditya Gupta and Greg Durrett. 2019. Tracking Discrete and Continuous Entity State for Process Understanding . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Workshop on Structure Predictions for NLP

work page 2019