Pith. sign in

REVIEW 4 major objections 5 minor 27 references

A sentence splitter that locates factual tails beats random masking as a self-supervised signal, improving commonsense QA and knowledge-graph completion.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:31 UTC pith:GRMCDEZJ

load-bearing objection Plausible recipe, but the evidence is under-specified; the ATOMIC overlap needs to be ruled out before the gains are credible. the 4 major comments →

arxiv 2607.19845 v1 pith:GRMCDEZJ submitted 2026-07-22 cs.CL cs.AI

Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning

Ahmad Pouramini , Mahsa Afsharizadeh This is my paper
classification cs.CL cs.AI
keywords sentence splittingself-supervised learningfactual structureknowledge graph completioncommonsense reasoningT5bootstrappingprefix-tail pairs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a sentence can be seen as a descriptive prefix plus a single factual tail, and that a model trained to find that boundary gives better self-supervised pretraining data than randomly masking spans. It builds a T5-based Sentence Splitter that, given a sentence, generates the tail; if the generated text exactly matches a contiguous span, that span is removed and the rest becomes the prefix. Training supervision comes from verbalized knowledge-graph triples, so no manual annotation is needed. The splitter then extracts prefix–tail pairs from raw text, and a second model proposes new tails in a one-step bootstrap. On CommonsenseQA and ATOMIC2020, the structure-aware objective improves accuracy and ROUGE over a matched random-boundary MLM baseline.

Core claim

The central discovery claim is that recovering the latent factual structure of a sentence—locating the boundary between a descriptive head and its factual completion—provides a stronger self-supervised signal than randomly masking contiguous spans. The authors formulate this as a discrete segmentation problem with N candidate split points in a sentence of length N, but solve it implicitly by training a T5 encoder–decoder to generate the tail from the full sentence and then accepting only predictions that exactly match a contiguous span. The splitter reaches 96.0% exact-match accuracy on templated ATOMIC verbalizations and 84.0% on manually annotated natural sentences from OMCS. When used as

What carries the argument

The Sentence Splitter: a T5-based encoder–decoder that maps an input sentence to its factual tail via probabilistic decoding instead of enumerating N split candidates; predictions are accepted only when they match a contiguous span of the input, which yields aligned prefix–tail pairs. The bootstrapping loop augments these pairs with a generator PLM and retrains the splitter, making symbolic knowledge and raw text interchangeable sources of structure-aware supervision.

Load-bearing premise

The paper's core empirical claim rests on the 3,000-sentence pretraining pool being free of ATOMIC2020 tails used for evaluation; if that pool leaks evaluation triples, the ROUGE gains could be memorization rather than structure-aware learning.

What would settle it

Inspect the 3,000-sentence pretraining corpus for overlap with the ATOMIC2020 evaluation triples; any overlap would make the ROUGE gains suspect. A cleaner test: run a sentence with the fact mid-sentence (e.g., 'Paris, the capital of France, is a city') — the splitter, trained for suffix tails, should fail to extract 'the capital of France' as the factual tail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If correct, structure-aware extraction gives a parser-free way to build factual training data from raw text in any domain with latent factual completions.
  • A single bootstrap iteration yields measurable gains; more iterations may improve further, though drift and error accumulation remain unstudied.
  • The splitter's 84% accuracy on natural text suggests that symbolic-verbalization training transfers beyond templates.
  • The objective reduces variance across seeds compared to random masking, implying more stable training behavior.
  • The framework connects symbolic knowledge-graph completion with self-supervised language-model pretraining, potentially benefiting tasks that need factual reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach likely transfers to encyclopedic, biomedical, or legal text, where factual completions often appear sentence-finally; a testable extension is evaluating the splitter on out-of-domain corpora with manual annotations.
  • The contiguous-suffix assumption is a real ceiling: mid-sentence facts (e.g., 'Paris, the capital of France, is a city') are out of reach; a span-selection variant could lift this limitation.
  • The few-shot downstream setting (30 examples per task) is small; the reported gains might shrink or grow under full fine-tuning, which would clarify whether the benefit is structural or data-efficiency-driven.
  • The validation module is intentionally unspecified; replacing fixed rules with an adaptive acceptance mechanism, perhaps learned from downstream task performance, could improve bootstrap quality and reduce error accumulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Sentence Splitter, a T5-based encoder-decoder model that recovers a factual completion (tail) from a natural-language sentence by predicting the single contiguous span that expresses the tail of an underlying knowledge-graph fact. The splitter is first trained on verbalized symbolic head–tail pairs, then applied to raw text to extract aligned prefix–tail pairs; a generator model produces additional plausible tails, and a bootstrap loop retrains the splitter on the augmented data. Downstream few-shot experiments on CommonsenseQA and ATOMIC2020 report that splitter-based pretraining improves over a random-boundary masked-language-model baseline, with further gains from one bootstrap iteration. The paper also reports 96% exact-match accuracy on ATOMIC verbalizations and 84% on 100 manually annotated OMCS sentences.

Significance. If the reported gains are genuine, the paper offers a lightweight and scalable way to translate symbolic knowledge-graph structure into a self-supervised pretraining signal, with a clean algorithmic formulation and explicit acknowledgment of the contiguous-suffix limitation. The central contribution is the sentence-splitting objective itself, which is simple and potentially useful. However, the current experimental evidence does not yet establish the claim: the decisive issue is that the 3,000-sentence pretraining pool is never identified, so the ATOMIC2020 improvements are compatible with a memorization confound. In addition, the only baseline is a random-boundary MLM, not a conventional masked-language-model or span-corruption baseline, and no comparison is made to existing structure-aware pretraining or knowledge-graph completion models. The evaluation of generalization to natural text rests on a very small single-annotator sample. These are load-bearing gaps rather than presentation issues.

major comments (4)
  1. [§4.4, Table 2] The pretraining pool of 3,000 sentences is not identified or characterized. Algorithm 2 Stage 1 trains the splitter on verbalized ATOMIC triples, and Section 4.3 reports 96% exact-match on ATOMIC verbalizations. If the 3,000-sentence pool contains ATOMIC-derived sentences or paraphrases of ATOMIC tails, then the splitter condition can memorize the exact tails used in the ATOMIC2020 downstream task, while the random-boundary MLM baseline receives no such structured tail-memorization signal. The paper must state whether the pretraining pool is disjoint from ATOMIC2020, provide the provenance of the pool, and ideally run a leakage test (e.g., removing all ATOMIC-related sentences from the pool, or evaluating on held-out relation types). Without this, the reported ROUGE gains of 25.14 and 27.51 versus 20.65 cannot be attributed to structure-aware pretraining.
  2. [§4.4, §5] The baseline labeled 'standard MLM' is described as randomly selecting a split boundary and replacing the contiguous suffix with a mask token. This is a random-boundary suffix-corruption baseline, not standard masked language modeling (e.g., BERT-style token masking or T5-style span corruption). It is a valid controlled comparison for the choice of boundary, but it does not support the Section 5 conclusion that the method improves 'beyond conventional masked language modeling.' The paper should compare against a standard span-corruption objective, and ideally against existing structure-aware pretraining (e.g., constituent masking) and knowledge-graph completion models (e.g., COMET, KG-BERT) to position the contribution.
  3. [§4.1, Table 1] The claim that the splitter 'generalizes beyond synthetic templates' rests on 100 manually annotated OMCS sentences with no inter-annotator agreement reported. The annotation instructions, annotator background, and number of annotators are not given, and exact-match accuracy on 100 sentences has a large confidence interval (e.g., 84/100 has a 95% CI of roughly 75–90%). A small single-annotator set is insufficient to support a strong generalization claim. The authors should report IAA, a larger evaluation set, or at least a per-boundary accuracy metric.
  4. [Algorithm 2, §3.4] The validation stage that filters generated sentences is said to be 'optional' and 'implementation dependent,' but the experiments use one bootstrap iteration and must have used some acceptance criterion. The paper does not state what criterion was actually applied (minimum tail length, perplexity threshold, etc.), whether any generated examples were rejected, or how the generator model was configured. This makes the bootstrapping results non-reproducible and leaves open the possibility that the gains come from the validation filter rather than the sentence-splitting structure. Please specify the exact experimental protocol.
minor comments (5)
  1. [§4.1] The dataset heading is typeset as 'A TOMIC2020.' Please fix the spacing.
  2. [References] Reference [15] is malformed ('Robert and Havasi'); the correct author list is Speer, R. and Havasi, C. Reference [20] also appears garbled in the author list. Please proofread the bibliography.
  3. [Figure 1] The text refers to Figure 1 but the figure is not present in the manuscript text. Ensure that the figure is included and legible.
  4. [§3.2, Algorithm 1] The exact-match acceptance rule means that all non-contiguous or non-matching predictions are discarded. The paper should report the fraction of corpus sentences discarded at this stage, since that determines the effective size and coverage of D_split and affects the interpretation of Table 2.
  5. [Table 2] Differences between configurations are reported as means with standard deviations but no significance tests. Given the small number of seeds (3), a paired test or confidence intervals on the differences would strengthen the claim of 'consistent improvements.'

Circularity Check

0 steps flagged

No significant circularity: the pretraining signal is not defined in terms of the evaluation targets, and no load-bearing self-citation or fitted-input-as-prediction step is present.

full rationale

The claimed derivation chain is: verbalize symbolic KG triples to train a splitter; apply the splitter to OMCS raw sentences to extract prefix–tail pairs; train a generator on those extracted pairs; pretrain T5 with splitter-based supervision versus random-boundary MLM; then fine-tune on CommonsenseQA and ATOMIC2020. Each stage uses an independent external resource: ATOMIC2020 verbalizations for splitter training, OMCS for raw-text extraction, and CommonsenseQA as a separate downstream benchmark. The ATOMIC2020 downstream task does share the head→tail mapping with the pretraining objective, but the paper never states that ATOMIC test tails are placed inside the 3,000-sentence pretraining pool; the pool is described only as 'the same pool of 3,000 sentences' (Sec. 4.4), while the raw corpus is OMCS (Sec. 4.1). Without a quoted statement placing ATOMIC triples in the pretraining pool, the contamination concern is an empirical validity risk, not a circularity reduction under the required standard. The splitter's 96% exact-match on ATOMIC verbalizations is explicitly presented as expected in-distribution performance, not as a novel prediction. There are no author self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The Conclusion's stated limitation (the contiguous-suffix assumption) is a scope restriction rather than a circular step. Therefore no circular step is exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No new physical or conceptual entities are introduced; the 'head–tail' structure is a label for text spans, not a new ontology. The main ledger items are the hand-chosen data sizes and the contiguous-suffix modeling assumption.

free parameters (4)
  • OMCS quality cutoff = top 8,000 sentences
    Section 4.1 selects the top 8,000 highest-scoring OMCS sentences by hand; no ablation shows how downstream results vary with this cutoff.
  • Downstream few-shot fine-tuning size = 30 examples per task
    Section 4.4 uses 30 training examples for each downstream task; the choice is arbitrary and no sensitivity analysis is provided.
  • Bootstrap iterations = 1
    Algorithm 2 allows T iterations, but only one is evaluated; the paper acknowledges this choice but the claim of 'bootstrapped structure-aware learning' is therefore tested only for a single step.
  • Pretraining pool size = 3,000 sentences
    Section 4.4 pretrains all models on 3,000 sentences, but the source and selection criteria are not specified, making it impossible to rule out overlap with downstream ATOMIC evaluation.
axioms (3)
  • domain assumption Every input sentence has a single dominant factual completion that is a contiguous suffix of the sentence.
    Stated in §3.2 and the Conclusion; necessary for the split-and-remove procedure in Algorithm 1 to produce aligned (p, t) pairs.
  • domain assumption Verbalized KG triples provide supervision that transfers to natural text.
    This underlies the claim of 84% exact-match accuracy on OMCS; it is tested on only 100 manually annotated sentences, with no inter-annotator agreement.
  • ad hoc to paper A prediction that exactly matches a contiguous span of the input is a correct factual tail, while non-matching predictions are noise to discard.
    Algorithm 1 uses this as the acceptance criterion, but the paper never reports the false-positive rate, false-negative rate, or the proportion of discarded predictions.

pith-pipeline@v1.3.0-alltime-deepseek · 8552 in / 12119 out tokens · 127319 ms · 2026-08-01T11:31:50.859870+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning." pith.science (2026). https://pith.science/paper/GRMCDEZJ

@misc{pith2026260719845,
  author       = {Pith},
  title        = {Pith review of: Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRMCDEZJ}},
  note         = {Machine review of arXiv:2607.19845}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper introduces Sentence Splitter, a self-supervised framework built upon a T5-based encoder--decoder architecture for uncovering the latent factual structure of natural language sentences. The proposed method identifies the semantic boundary between a descriptive prefix (head) and its factual completion (tail) by formulating sentence splitting as a discrete segmentation problem, where a sentence of length $N$ admits $N$ possible split points but only one recovers the intended head--tail structure. Rather than explicitly searching over all candidate boundaries, the model learns to recover the factual completion through probabilistic sequence generation. To eliminate the need for manual annotation, symbolic head--tail pairs are first verbalized into natural-language templates that provide supervision for training the Sentence Splitter. The trained splitter is then applied to raw text to extract aligned prefix--tail pairs, which are subsequently used to train a generative model that proposes additional plausible completions through a lightweight bootstrapping process. This unified pipeline provides a scalable and structure-aware approach to constructing self-supervised training data while bridging symbolic knowledge and natural language. Experiments on both structured and naturally occurring text demonstrate that the proposed splitter generalizes beyond synthetic templates and that the resulting structure-aware supervision consistently improves downstream performance on knowledge graph completion and commonsense question answering, highlighting the effectiveness of recovering latent factual structure for knowledge-centric NLP.

Figures

Figures reproduced from arXiv: 2607.19845 by Ahmad Pouramini, Mahsa Afsharizadeh.

Figure 1
Figure 1. Figure 1: The full pipeline for training a sentence-splitter and expanding its supervised data using a [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 9 linked inside Pith

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Sco...

    Pith/arXiv arXiv 2020

  2. [2]

    Analyzing commonsense emergence in few-shot knowledge models

    Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. Analyzing commonsense emergence in few-shot knowledge models. InConference on Automated Knowledge Base Construc- tion, 2021

    2021

  3. [3]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 Conference of 9 the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. Association for Computational Li...

    2019

  4. [4]

    Efficient sentence segmentation using syntactic features

    Benoit Favre, Dilek Hakkani-T¨ ur, Slav Petrov, and Dan Klein. Efficient sentence segmentation using syntactic features. InProceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 1577–1580. IEEE, 2008

    2008

  5. [5]

    Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosse- lut, and Yejin Choi

    Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosse- lut, and Yejin Choi. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. InarXiv preprint arXiv:2010.05953, 2020. COMET family — commonsense knowledge graph gen- eration / text-to-triple work

    Pith/arXiv arXiv 2020

  6. [6]

    Unifiedqa: Crossing format boundaries with a single qa system

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907. Association for Computational Linguistics, 2020

    2020

  7. [7]

    Lingpeng Kong, Chris Dyer, and Noah A. Smith. Segmental recurrent neural networks. InProceedings of the International Conference on Learning Representations (ICLR), 2016

    2016

  8. [8]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InAnnual Meeting of the Association for Computational Linguistics, 2004

    2004

  9. [9]

    Conceptnet: A practical commonsense reasoning toolkit

    Hugo Liu and Pushpak Singh. Conceptnet: A practical commonsense reasoning toolkit. InBT technology journal, volume 22, pages 211–226. Springer, 2004

    2004

  10. [10]

    Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark

    Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13480–13488, 2021

    2021

  11. [11]

    Orca: Progressive learning from complex explanation traces of gpt-4.arXiv preprint arXiv:2306.02707, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Awadallah, et al. Orca: Progressive learning from complex explanation traces of gpt-4.arXiv preprint arXiv:2306.02707, 2023. learning from explanation traces of stronger models

    Pith/arXiv arXiv 2023

  12. [12]

    Split and rephrase

    Shashi Narayan and Claire Gardent. Split and rephrase. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 606–616. ACL, 2017

    2017

  13. [13]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.arXiv preprint arXiv:1910.10683, 2019

    Pith/arXiv arXiv 1910

  14. [14]

    Smith, and Yejin Choi

    Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. ATOMIC: An atlas of machine com- monsense for if-then reasoning. In33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9t...

    2019

  15. [15]

    Conceptnet.Theory and Applications of Natural Language Processing., pages 121–160, 2013

    Catherine Speer, Robert and Havasi. Conceptnet.Theory and Applications of Natural Language Processing., pages 121–160, 2013

    2013

  16. [16]

    Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler

    Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Ul2: Unifying language learning paradigms.arXiv preprint arXiv:2205.05131, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Dibert: Dependency-injected bidirectional encoder representations from transformers.TechRxiv / preprint (dependency-injected BERT variants), 2023

    Abdul Wahab, Rafet Sifa, et al. Dibert: Dependency-injected bidirectional encoder representations from transformers.TechRxiv / preprint (dependency-injected BERT variants), 2023. dependency- aware pretraining (dependency-injected / dependency objectives)

    2023

  18. [18]

    Kepler: A unified model for knowledge embedding and pre-trained language representation

    Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194, 2021. 10

    2021

  19. [19]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    Rush, and Yoon Kim Zhou

    Yuntian Wang, Dengyong Deng, Alexander M. Rush, and Yoon Kim Zhou. Sequence modeling via segmentations. InProceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pages 3674–3683. PMLR, 2017

    2017

  21. [21]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. InProceedings of NeurIPS (or arXiv preprint arXiv:2109.01652), 2021

    Pith/arXiv arXiv 2021

  22. [22]

    Kg-bert: Bert for knowledge graph completion.arXiv preprint arXiv:1909.03193, 2019

    Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion.arXiv preprint arXiv:1909.03193, 2019

    Pith/arXiv arXiv 1909

  23. [23]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning.arXiv preprint arXiv:2203.14465, 2022. NeurIPS / STaR (Self-Taught Reasoner) — iterative rationale / bootstrapping

    Pith/arXiv arXiv 2022

  24. [24]

    Auto evol-instruct: Automatic instruction evolving for large language models

    Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Auto evol-instruct: Automatic instruction evolving for large language models. InProceedings of EMNLP 2024, page —,

    2024

  25. [25]

    Limit-bert: Linguistically informed multi-task bert (syntactic / semantic phrase masking)

    Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. Limit-bert: Linguistically informed multi-task bert (syntactic / semantic phrase masking). InFindings of the Association for Compu- tational Linguistics: EMNLP, pages 4450–4461. Association for Computational Linguistics, 2020. introduces syntactic/semantic-phrase masking (constituent-guided masking)

    2020

  26. [26]

    Syntactic and semantic phrase masking / semantic-role guided masking for pretraining

    Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. Syntactic and semantic phrase masking / semantic-role guided masking for pretraining. InFindings of the Association for Computa- tional Linguistics: EMNLP, pages 4450–4461, 2020. this entry is the semantic-role / semantic-phrase masking example referenced in the paragraph. 11

    2020

  27. [2024]

    Evol-Instruct / instruction evolving line of work (Auto Evol-Instruct paper)