REVIEW 3 major objections 5 minor 36 references
CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Disserta\c{c}\~oes e Trabalhos de Gradua\c{c}\~ao em SI -- XXI Simp\'osio Brasileiro de Sistemas de Informa\c{c}\~ao
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A bi-objective instance-selection framework prunes redundant and noisy training documents for transformer text classifiers, cutting training sets by 41% on average (up to 60%) while preserving effectiveness across all 22 datasets and…
desk verdict Useful condensed summary of an already-published, genuinely interesting line of work; the 'same effectiveness' claim is not supported by the statistical protocol described here. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the confidence- and entropy-weighted removal probability. A calibrated weak classifier (logistic regression) assigns each instance a confidence; confident correct predictions are treated as redundant, and for mispredicted instances the inverse entropy of the posterior distribution estimates noise likelihood—low-entropy wrong answers are confidently wrong and thus more likely noise. These two signals define an α-weighting, and an iterative statistical comparison of weak-model effectiveness with and without reduction sets the β reduction rate. The final training set is a random sample of size (1−β) weighted by α, so removal is probabilistic and tuned to preserve the deep model's expected behavior.
What would settle it
Compare transformer fine-tuning on (a) biO-IS's selected subset, (b) a random subset of the same size, and (c) the discarded instances, across several datasets. If the random subset achieves the same MacroF1 as the selected subset within the statistical margin, the confidence/entropy signal is not actually carrying the redundancy/noise information. A second check: if the weak classifier's reduction-effectiveness curve on the validation set predicts no loss at a reduction rate where the transformer's accuracy drops significantly, the surrogate-assumption fails.
Extended reading notes
Core claim
biO-IS is a bi-objective instance-selection framework that removes both redundant and noisy training instances prior to transformer fine-tuning. It uses logistic regression as a calibrated weak classifier: high-confidence correct predictions mark easy, redundant documents, while low-entropy incorrect predictions mark likely noise; an iterative validation procedure estimates a near-optimal reduction rate using the weak model's effectiveness as a surrogate for the transformer's. In experiments across 22 topic and sentiment datasets, biO-IS reduced training sets by 40.1% on average (29–60%), removed 66.6% of manually inserted noise, preserved MacroF1 on every dataset, and delivered mean speedups of 1.67x (up to 2.46x), outperforming its predecessor E2SC and all tested baselines on the reduction-efficiency-effectiveness trade-off.
Load-bearing premise
The framework assumes that a cheap classifier's confidence and entropy faithfully identify which documents are redundant or noisy for a much larger transformer model, and that the weak model's behavior under reduction predicts the transformer's behavior; if these proxies misalign with what transformer fine-tuning needs, the selected subsets could hurt accuracy despite the reported results.
Editorial extensions
If this is right
- Training sets for transformer fine-tuning can be cut by roughly 40% without lowering MacroF1 on the tested benchmarks.
- Model construction time drops by 1.67x on average, with larger gains on larger datasets, making frequent re-training more feasible.
- Traditional IS methods rarely improve effectiveness, but biO-IS removes a large fraction of injected noise, a capability no tested baseline matched.
- The results support the dissertation's hypothesis that smaller, well-chosen training data can replace large datasets for many text classification tasks.
Reading between the lines
- If the confidence/entropy proxy generalizes, the same pruning could be applied to fine-tune larger language models, lowering the compute barrier for custom classifiers in low-resource settings.
- The framework could be coupled with active learning: the low-confidence, high-entropy instances it retains are precisely the ones a human annotator would most usefully label.
- A testable extension is moving from whole-document selection to passage- or aspect-level selection, since noisy or redundant content may live inside documents rather than in entire documents.
- The reported noise-removal result comes from simulated injected noise; real-world label noise may follow different distributions, so the 66.6% figure should be tested on naturally noisy labels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is a condensed PhD dissertation summary (CTDGSI) on instance selection (IS) for automatic text classification (ATC). It reports a systematic literature review and comparative evaluation of 13 traditional IS methods combined with transformer-based classifiers on 22 datasets, and proposes two frameworks: E2SC, a redundancy-oriented confidence-based instance selection method, and biO-IS, an extension that additionally removes noisy instances using an entropy-based criterion. The central claims are that biO-IS reduces training sets by about 41% on average (up to 60%), removes 66.6% of manually inserted noise, maintains the same effectiveness in all 22 datasets, and achieves average speedups of 1.67x (up to 2.46x).
Significance. If the reported results are reliable, this work is practically significant: it demonstrates that instance selection can substantially reduce the cost of fine-tuning transformer text classifiers without degrading MacroF1, and it extends the IS literature from small tabular data to large, high-dimensional text collections. The manuscript is grounded in several peer-reviewed publications, provides a broad benchmark (22 datasets) and a proposed taxonomy, and includes pointers to code and data for reproducibility. Its main weakness is that the statistical evidence in this text does not support the positive claim of 'maintaining the same effectiveness' in all datasets, and the per-dataset results needed for verification are not included here.
major comments (3)
- [Section 5 ('Metrics and Experimental Protocol') and Section 7 ('Experimental Results')] The claim that biO-IS 'maintained the same levels of effectiveness in all of the considered datasets' is supported only by failing to reject the null hypothesis in paired t-tests with Bonferroni correction. With k=5 folds for large datasets and k=10 for small ones, the tests have 4 or 9 degrees of freedom, and Bonferroni over 22 datasets lowers the per-test threshold to roughly 0.0023; real MacroF1 drops of 1-2 points are unlikely to be detected. To support a positive equivalence claim, the authors should report per-dataset effect sizes, confidence intervals, or an explicit equivalence margin (e.g., a TOST or confidence-interval-based test), rather than only non-significance.
- [Section 7 ('Experimental Results') and Section 6 ('Experimental Results')] The manuscript reports only aggregate point estimates for biO-IS (average reduction 40.1%, speedup 1.67x, 66.6% noise removal) and does not provide a per-dataset table of MacroF1, reduction rates, or speedups. Since the reduction rate beta is selected per dataset by a weak classifier on a validation set, the 41% average reduction is an optimized outcome whose stability is unknown. The authors should provide the per-dataset breakdown and the distribution/variance of the selected beta values so that readers can assess whether the average reduction is a robust property of the method or an artifact of per-dataset tuning.
- [Section 6 (footnote 5) and Section 7 (weak classifier description)] The core mechanisms of biO-IS rely on the assumption that a weak classifier's confidence and entropy are reliable proxies for redundancy and noise for transformer fine-tuning. The text states that this premise is 'tested and confirmed' in the dissertation, but the details are not provided in this manuscript. Since this assumption is load-bearing for the central effectiveness claim, the authors should include a concrete validation experiment in this text (for example, comparing the weak model's predicted safe reduction rate with the transformer's actual MacroF1 across a range of beta values) or explicitly mark this as an untested limitation with reference to the dissertation's evidence.
minor comments (5)
- [Abstract, Section 7, Section 8] The reduction and speedup numbers are inconsistent across the text: the abstract and conclusion say 41% reduction, Section 7 says 40.1%; Section 6 says E2SC achieved 27% average reduction, while Section 8 says 30%; Section 8 reports 'speedups of up to 70%' while Section 7 reports 1.67x average and 2.46x maximum. These should be reconciled.
- [Section 5 ('Metrics and Experimental Protocol')] The formula for reduction R is typeset incorrectly: 'R = P k i=0 |Ti|-|Si| / k' is malformed and should be written with a summation index, e.g., R = (1/k) * sum_i (|T_i|-|S_i|)/|T_i|.
- [Section 5 ('Text Classification Methods')] The text lists six transformer models (RoBERTa, BERT, DistilBERT, BART, AlBERT, XLNet) but then refers to 'the best of seven deep learning text classification methods'; the count should be made consistent.
- [Throughout] Several Portuguese labels remain in an English manuscript, such as 'Figura 1' and 'Tabela 1', and the references heading is 'Referências'; these should be translated or unified.
- [Throughout] Minor language and typographical issues include 'inspection-ed' (Section 4), 'effectivly' (Section 8), and inconsistent use of italics for method names; a careful proofread is needed.
Circularity Check
No significant circularity: biO-IS's effectiveness is independently measured, and the adaptive reduction rate is an algorithmic output, not a prediction fitted to the target.
full rationale
The paper's central claim, that biO-IS maintains effectiveness while reducing training sets, is not circular. The deep transformer model's effectiveness is measured independently on held-out folds after training on the selected subset. The per-dataset reduction rate (beta) is estimated by an iterative procedure that preserves the weak classifier's validation effectiveness; this is a proxy assumption, not an equation that forces the deep model's outcome. The noise-removal rate (66.6%) is evaluated in a simulated scenario against manually inserted noise, which is a direct measurement of the method's stated objective rather than a prediction derived from its own assumptions. The paper does rely on the authors' prior publications for method details, hyperparameter methodology, and the full dissertation for confirming the KNN proxy, but these self-citations are not load-bearing in a circular sense: the key effectiveness results are reported in this text through independent experiments. The statistical concern that failing to reject a difference under low-power paired t-tests does not establish equivalence is a validity or reporting issue, not a circularity issue. No step in the derivation reduces to its own inputs by construction, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- beta (reduction rate) =
29-60% per dataset (41% average for biO-IS)
- per-instance alpha removal probability =
derived from weak classifier confidence
- IS method hyperparameters =
grid-searched per baseline method
- transformer hyperparameters (max len, batch size) =
max len 150/256, batch size 16/32
assumptions (5)
- domain assumption Calibrated weak classifier confidence correlates with redundancy
- domain assumption Weak classifier effectiveness on a validation set predicts deep model effectiveness under data reduction
- domain assumption Entropy of incorrect weak-classifier predictions identifies noise
- domain assumption TF-IDF representation suffices as input to instance selection methods
- standard math Statistical tests justify the 'same effectiveness' claims
Cite this review
Pith. "Pith review of CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Disserta\c{c}\~oes e Trabalhos de Gradua\c{c}\~ao em SI -- XXI Simp\'osio Brasileiro de Sistemas de Informa\c{c}\~ao." pith.science (2026). https://pith.science/paper/R6EGZN5O
@misc{pith2026250607169,
author = {Pith},
title = {Pith review of: CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Disserta\cc\~oes e Trabalhos de Gradua\cc\~ao em SI -- XXI Simp\'osio Brasileiro de Sistemas de Informa\cc\~ao},
year = {2026},
howpublished = {\url{https://pith.science/paper/R6EGZN5O}},
note = {Machine review of arXiv:2506.07169}
}
read the original abstract
Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. This \textbf{Ph.D. dissertation} focuses on an under-investi\-gated NLP data engineering technique, whose potential is enormous in the current scenario known as Instance Selection (IS). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining the effectiveness of the trained models and reducing the training process cost. We provide a comprehensive and scientifically sound comparison of IS methods applied to an essential NLP task -- Automatic Text Classification (ATC), considering several classification solutions and many datasets. Our findings reveal a significant untapped potential for IS solutions. We also propose two novel IS solutions that are noise-oriented and redundancy-aware, specifically designed for large datasets and transformer architectures. Our final solution achieved an average reduction of 41\% in training sets, while maintaining the same levels of effectiveness in all datasets. Importantly, our solutions demonstrated speedup improvements of 1.67x (up to 2.46x), making them scalable for datasets with hundreds of thousands of documents.
Reference graph
Works this paper leans on
-
[1]
Introduction The exponential growth in data availability has made it challenging to organize and retrieve content effectively. In this context, Automatic Text Classification ( ATC) offers a solution by mapping textual documents (e.g., web pages, emails, reviews, tweets, social media messages) into predefined categories of interest. Indeed, ATC models are ...
work page Pith review arXiv 2025
-
[2]
Hypothesis and Research Questions The main hypothesis (H1) of this Ph.D. dissertation is: H1: It is possible to simultaneously reduce data, maintain model quality, and improve time for fine-tuning ATC models through IS methods. In order to confirm this hypothesis, we propose three research questions for our Ph.D. dissertation. In sum, RQ1 aims at evaluati...
-
[3]
Publications Our work in the Instance Selection field has been validated and published in the main Information Systems conferences and journals in the last four years only, including two published papers in the Information Processing and Management (IP&M) (h5-index: 96, Impact Factor: 7.4, A1) , a worldwide leading journal in Information Retrie- val [Cunh...
work page 2021
-
[4]
A Noise-Oriented and Redundancy-Aware Instance Selection Framework
Cunha, Washington, et al. “A Noise-Oriented and Redundancy-Aware Instance Selection Framework.” ACM Transactions on Information Systems(ACM TOIS) (2024)– h5-index: 48.0
work page 2024
-
[5]
Cunha, Washington, et al. “An effective, efficient, and scalable confidence-based instance selection framework for transformer-based text classification.” ACM SIGIR 2023 – h5-index: 103.0
work page 2023
-
[6]
Cunha, Washington, et al. “A Comparative Survey of Instance Selection Methods applied to Non- Neural and Transformer-Based Text Classification.”ACM Computing Surveys(2023) – h5-idx: 157.0
work page 2023
-
[7]
Cunha, Washington, et al. “On the cost-effectiveness of neural and non-neural approaches and repre- sentations for text classification:A comprehensive comparative study.”IP&M (2021) - h5-index: 114.0
work page 2021
-
[8]
Cunha, Washington, et al. “Extended pre-processing pipeline for text classification: On the role of meta-feature representations, sparsification and selective sampling.” IP&M (2020) – h5-index: 114.0 Our dissertation contributed directly to other journals during the doctorate period:
work page 2020
Show all 36 references
-
[9]
Mais com menos - processamento de linguagem natural inte- ligente e sustent´avel baseado em engenharia de dados e inteligˆencia artificial avanc ¸ada
Gonc ¸alves, M.,Cunha, W et al. (2024) “Mais com menos - processamento de linguagem natural inte- ligente e sustent´avel baseado em engenharia de dados e inteligˆencia artificial avanc ¸ada.”IV Semin´ario de Grandes Desafios da Computac ¸˜ao no Brasil 2025-2035 - Sociedade Bra...
2024
-
[10]
Franc ¸a, C.,Cunha, W et al. (2024). On representation learning-based methods for effective, efficient, and scalable code retrieval. Neurocomputing. – h5-index: 136.0
2024
-
[11]
the classifier does not matter when the (text) representation is so good!
Andrade, C., Cunha, W (2023) On the class separability of contextual embeddings representations – or “the classifier does not matter when the (text) representation is so good!”. IP&M. – h5-index: 114.0
2023
-
[12]
S., Cunha, W et al
Zanotto, B. S., Cunha, W et al. (2021). Pcv50 automatic classification of electronic health records for a value-based program through machine learning. Value in Health. – h5-index: 57.0
2021
-
[13]
S., Cunha, W et al
Zanotto, B. S., Cunha, W et al. (2021). Stroke outcome measurements from electronic medical records: Cross-sectional study on the effectiveness of neural and nonneural classifiers.JMIR Med – h5-idx: 52.0
2021
-
[14]
et al., Cunha, W (2024)
Viegas, F. et al., Cunha, W (2024). Exploiting contextual embeddings in hierarchical topic modeling and investigating the limits of the current evaluation metrics.Computational Linguistics. – h5-idx:38
2024
-
[15]
et al., Cunha, W (2024)
Felix, L. et al., Cunha, W (2024). Why are you traveling? Inferring trip profiles from online reviews and domain-knowledge. Online Social Networks and Media. – h5-index: 28.0
2024
-
[16]
Viegas, F., Cunha, W et al. (2024). Pipelining semantic expansion and noise filtering for sentiment analysis of short documents – clusent method. Journal on Interactive Systems. – h5-index: 9.0
2024
-
[17]
(2021) Extended Pre-Processing Pipeline For Text Classification: On the Role of Meta-Features, Sparsification and Selective Sampling
Cunha, W et al. (2021) Extended Pre-Processing Pipeline For Text Classification: On the Role of Meta-Features, Sparsification and Selective Sampling. CTDBD - SBBD’21 – h5-index: 7.0
2021
-
[18]
Gomes, C., Cunha, W et al. (2019). CluWords: Explorando Clusters Sem ˆanticos entre Palavras para Aprimorar Modelagem de T´opicos. CTIC - Iniciac ¸˜ao Cient´ıfica em Computac ¸˜ao. – h5-index: 1.0 Our dissertation contributed directly to other conference papers during the Ph.D...
2019
-
[19]
Viegas, F., Cunha, W et al. (2020). Cluhtm - semantic hierarchical topic modeling based on cluwords. Meeting of the Association for Computational Linguistics (ACL) ACL’20. – h5-index: 215.0
2020
-
[20]
Keep it Simple, Lazy
Mendes, L., Cunha, W et al. (2020). “Keep it Simple, Lazy”— Metalazy: A new metastrategy for lazy text classification. CIKM’20. – h5-index: 91.0
2020
-
[21]
Viegas, F., Cunha, W et al. (2019). Cluwords: Exploiting semantic word clustering for enhanced topic modeling. WSDM’19. – h5-index: 77.0
2019
-
[22]
Conference on Computational Natural Language Learning (CoNLL’24)
Andrade et al., Cunha, W (2024) Explaining the hardest errors of contextual embedding-based classi- fiers. Conference on Computational Natural Language Learning (CoNLL’24). – h5-index: 39.0
2024
-
[23]
Pasin, A., Cunha, W et al. (2024). A quantum annealing instance selection approach for efficient and effective transformer fine-tuning. ACM SIGIR ICTIR. – h5-index: 24.0
2024
-
[24]
Fonseca, G., Cunha, W et al. (2024). Estrat ´egias de Undersampling para Reduc ¸ ˜ao de Vi ´es em Classificac ¸˜ao de Texto Baseada em Transformers. WebMedia’24. – h5-index: 13.0
2024
-
[25]
Vasconcelos, N., Cunha, W et al. (2024). Integrando Avaliac ¸ ˜oes Textuais de Usu ´arios em Recomendac ¸˜ao baseada em Aprendizado por Reforc ¸o.WebMedia’24. – h5-index: 13.0
2024
-
[26]
Viegas, F., Cunha, W et al. (2023). Clusent–combining semantic expansion and de-noising for dataset- oriented sentiment analysis of short texts. WebMedia’23. – h5-index: 13.0
2023
-
[27]
Cecilio, P., Cunha, W et al. (2023). Um framework para extrac ¸ ˜ao autom ´atica de informac ¸˜oes em patentes farmacˆeuticas. WebMedia’23. – h5-index: 13.0
2023
-
[28]
(2022) Evaluating topic modeling pre-processing pipelines for portuguese texts
J ´unior, A., Cunha, W et al. (2022) Evaluating topic modeling pre-processing pipelines for portuguese texts. WebMedia’22 – h5-index: 13.0
2022
-
[29]
Pasin, A., Cunha, W et al. (2024). A quantum annealing-based instance selection approach for trans- former fine-tuning. Italian Information Retrieval Workshop. – h5-index: 7.0
2024
-
[30]
Santos, W., Cunha, W et al. (2023). Uma metodologia para tratamento do vi´es da maioria em modelos de stacking via identificac ¸˜ao de documentos dif´ıceis. SBBD’23. – h5-index: 7.0
2023
-
[31]
Macul, V ., Cunha, W et al. (2024). Intelig ˆencia Artificial Generativa para Personalizac ¸˜ao do Cuidado de Sa´ude Integral: Processo de caracterizac ¸˜ao dos dados. II Simp´osio CI-IA Sa´ude da UFMG
2024
-
[32]
This review aims to comprehensively assess the most relevant works related to IS strategies applied in different scenarios
Systematic Literature Review of Instance Selection Methods In this section, we present a critical analysis ( a.k.a., rapid (systematic-based) literature review of the most traditional and/or recent (state-of-the-art) proposals in the Instance Selection (IS) area. This review a...
2012
-
[33]
A Comparative Survey of Instance Selection Methods applied to NonNeural and Transformer-Based Text Classification In this section, we propose to assess the trade-off among reduction, efficiency, and ef- fectiveness of these 13 most representative traditional IS methods (Sectio...
2013
-
[34]
An Effective, Efficient, and Scalable Confidence-Based Instance Selection Framework for Transformer-Based Text Classification In the previous section, we found that no one method was able to meet all restrictions in all cases. Additionally, in some scenarios, using these metho...
2022
-
[35]
Figura 5
An Extended Noise-Oriented and Redundancy-Aware Instance Selection Framework for Transformer-Based Automatic Text Classification The main contribution of this section is the proposal of an extended bi-objective instance selection (biO-IS) framework built upon our first one aim...
2024
-
[36]
the classifier does not matter when the (text) representation is so good!
Conclusion and Future Work This dissertation surveyed classical and recent IS approaches, revealing significant advances but limited application scope. Most traditional methods target small tabular datasets, with rare applications in NLP despite its potential benefits. To addr...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.