Exploring sentence informativeness
Pith reviewed 2026-05-24 19:28 UTC · model grok-4.3
The pith
Sentence informativeness classifiers affect word embedding quality when used in training from scarce data.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Using the predictions of informativeness classifiers to train word embeddings has an impact on embedding quality, offering a way to build better representations from scarce data.
What carries the argument
Sentence-level classifiers to predict informativeness of sentences about contained words.
Load-bearing premise
The informativeness captured by the classifiers is useful for improving word embedding training in data-scarce conditions.
What would settle it
An experiment showing no difference in embedding quality metrics when using classifier predictions versus not using them.
read the original abstract
This study is a preliminary exploration of the concept of informativeness -how much information a sentence gives about a word it contains- and its potential benefits to building quality word representations from scarce data. We propose several sentence-level classifiers to predict informativeness, and we perform a manual annotation on a set of sentences. We conclude that these two measures correspond to different notions of informativeness. However, our experiments show that using the classifiers' predictions to train word embeddings has an impact on embedding quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper explores the concept of sentence informativeness—how much information a sentence provides about a contained word—and its potential to improve word embeddings trained on scarce data. Several sentence-level classifiers are proposed to predict informativeness. A manual annotation study is conducted, leading to the conclusion that classifier predictions and human judgments capture different notions of informativeness. Experiments are reported showing that using the classifiers' predictions to train word embeddings has an impact on embedding quality.
Significance. This work tackles the challenge of learning quality word representations from limited data, which is a significant issue in NLP. The separation of classifier-based and human-based informativeness is a notable observation. However, without detailed experimental results, metrics, or methodology for the embedding experiments, the practical significance for improving embeddings remains unclear and unverified.
major comments (2)
- [Abstract] Abstract: The statement 'our experiments show that using the classifiers' predictions to train word embeddings has an impact on embedding quality' provides no quantitative evidence, no description of the training integration method, no evaluation metrics, and no indication of whether quality improved or declined. This is the central claim and requires substantial elaboration.
- [Experiments/results] Experiments/results: There is no information on the base embedding algorithm (e.g., word2vec, GloVe), the scarce data setup (corpus size, vocabulary), the downstream or intrinsic evaluation tasks, or any comparison showing the nature of the 'impact'. This undermines the ability to assess the contribution.
minor comments (1)
- [Abstract] Abstract: Clarify the specific way in which informativeness is used in embedding training and the observed direction of the effect.
Simulated Author's Rebuttal
We thank the referee for the constructive comments. We agree that the abstract and the embedding experiments section lack sufficient detail to support the central claim, and we will revise the manuscript to address these issues.
read point-by-point responses
-
Referee: [Abstract] Abstract: The statement 'our experiments show that using the classifiers' predictions to train word embeddings has an impact on embedding quality' provides no quantitative evidence, no description of the training integration method, no evaluation metrics, and no indication of whether quality improved or declined. This is the central claim and requires substantial elaboration.
Authors: We agree that the abstract statement is too high-level and does not convey the necessary evidence or details. In the revised version we will expand the abstract to report quantitative results, briefly describe the integration method, name the evaluation metrics, and indicate the direction of the observed impact. revision: yes
-
Referee: [Experiments/results] Experiments/results: There is no information on the base embedding algorithm (e.g., word2vec, GloVe), the scarce data setup (corpus size, vocabulary), the downstream or intrinsic evaluation tasks, or any comparison showing the nature of the 'impact'. This undermines the ability to assess the contribution.
Authors: The current manuscript presents the embedding experiments at a high level because the work is framed as a preliminary exploration. We accept that this is insufficient. The revised experiments section will specify the base algorithm, corpus and vocabulary sizes for the scarce-data regime, the intrinsic or downstream tasks, and direct comparisons that illustrate the nature of the impact. revision: yes
Circularity Check
No circularity; experimental pipeline is independent of its inputs.
full rationale
The paper reports three sequential but independent stages: (1) manual sentence annotation, (2) training of sentence-level classifiers on those annotations, and (3) a separate embedding-training experiment that consumes the classifier outputs as an auxiliary signal. None of these stages is defined in terms of the others, no parameter is fitted on a subset and then re-used as a 'prediction' of a closely related quantity, and no uniqueness theorem or ansatz is imported via self-citation. The central empirical claim ('has an impact') is therefore not forced by construction and remains falsifiable by the reported experiments.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.