How to Achieve Higher Accuracy with Less Training Points?

Anupam Pani; Jinghan Yang; Yunchao Zhang

REVIEW 4 major objections 5 minor 24 references

How to Achieve Higher Accuracy with Less Training Points?

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selecting training points by their influence on validation predictions lets a logistic regression model match full-data test accuracy with 10% of the extra data and exceed it with 60%.

desk verdict The paper's '10% of the data' claim is actually 64% of the full training set, but the underlying influence-based selection idea is coherent and worth a quick look. read the letter →

arxiv 2504.13586 v1 pith:GBM3GG3B submitted 2025-04-18 cs.LG stat.AP

Jinghan Yang , Anupam Pani , Yunchao Zhang This is my paper · ORCID

classification cs.LGstat.AP

keywords influencefunctionstrainingdataselectionefficiencylogisticregressionsentimentanalysissubsetmodelperformancegreenAI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a model can be trained on far fewer, better-chosen examples instead of an entire dataset. It proposes scoring candidate training points by their influence-function effect on validation predictions, then adding only those whose predicted effect pushes validation predictions toward the correct labels. In binary sentiment classification with logistic regression, the method matches full-data test accuracy using 10% of the additional training pool and exceeds it using 60%. If the pattern holds, targeted data addition could cut training cost while improving accuracy.

What carries the argument

The mechanism is the influence function applied to adding, rather than removing, training points. Using the Hessian of the empirical risk at the fitted parameters, Equation (4) approximates the parameter change from adding a subset, and Equation (5) translates that change into predicted output changes on validation points. Each candidate point is scored by summing the predicted changes aligned with moving each validation prediction toward its correct label, with variants that weight by label discrepancy, threshold, top-K prioritization, flip counts, or misalignment penalties. Positively scored points form the subset that is added and the model is retrained.

What would settle it

Train the model on the exact 60% subset selected by Method 1 and evaluate on the held-out test set; if the test accuracy does not exceed the 0.627 of the Add-Full baseline, the reported advantage does not reproduce. Separately, compute the per-candidate influence scores from Equation (4)-(5), retrain the model adding each single candidate alone, and compare the rank correlation between predicted and actual validation accuracy changes; a weak or negative correlation for high-scoring candidates would show the linearized ranking is not what carries the result.

Watch

Extended reading notes

Core claim

The central claim is that the influence of adding a training point on validation predictions, computed through a linearized influence-function approximation (IP-adding), provides a usable ranking for selecting a small subset of additional training data. On a binarized movie-review sentiment dataset with a logistic regression model, selecting the top few percent of additional examples by cumulative positive influence yields 0.630 test accuracy at 10% of the pool, above the 0.627 of training on the full pool; selecting 60% by the simplest variant (Method 1) yields 0.644, the highest reported. The authors interpret this as evidence that comparable or superior accuracy is achievable with a fraction of the training data.

Load-bearing premise

The influence ranking is computed with a linear approximation around the original model's parameters, and the paper assumes that ranking remains correct even when the added subset is large enough (60% of the pool, on the order of a thousand points) to move the optimum far from the linearization point and create interactions among the added points.

Editorial extensions

If this is right

Subset selection by influence can replace full-data training: 10% of the additional pool gives comparable test accuracy to using 100% (0.630 vs 0.627).
Using a larger selected fraction, 60%, can improve accuracy relative to full-data training (0.644 vs 0.627).
Random selection underperforms both full-data training and influence-based selection, showing that which points are added matters, not just how many.
The scoring rule is evaluated on a validation set and requires no retraining for each candidate, making the selection step cheap relative to exhaustive search.
The heuristic variants trade off fraction selected against accuracy, and the simplest variant (Method 1) gives the best test accuracy at 60%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The linearized influence approximation is only tested implicitly; directly comparing the predicted ranking against per-point retraining effects would reveal how much ranking quality degrades as the selected subset grows, especially at 60% of the pool.
Because selection is done once at the original parameters, the approach resembles a single-step active-learning procedure; iterating the selection with retraining between rounds could further reduce the required fraction, but that is an extension the paper does not test.
The validation set stands in for the test distribution, so the method's practical value depends on how well that proxy holds; under distribution shift, the selected points may be the wrong ones.
The same addition-based influence score could apply to other convex models or to mini-batch selection in neural networks, though Hessian-based approximations become more delicate there.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the thing to know: the headline '10% of the data' is not what the experiment does. The 10% is 10% of an additional pool of 2,768 points, on top of an original training set of 4,152. Method 3 actually trains on 4,429 points, which is 64% of the full 6,920-point dataset. Same issue for the 60% claim: Method 1 uses 5,813 points, 84% of the full data. The stress-test note is right.

What's genuinely new: applying influence functions to choose which points to add to a training set, with six heuristics. That's a natural extension of Koh & Liang's removal influence, and the specific IP-adding score is new. The paper is clear, the notation is fine, and it includes a random baseline column and an 'Add Full' comparison, which is more than many course projects do. Method 1 at 60% of the extra pool reaches 0.644 test accuracy versus 0.627 for Add Full; that is a plausible result.

The soft spots: single dataset (SST binary), single logistic regression, single run, no code, no error bars. The influence function is linearized at the original optimum, yet you add 1,661 points in Method 1; nobody checks whether that approximation still ranks points correctly at that scale. Six heuristics are tried; the best one is selected on the validation set, so there's selection over methods. And the abstract and conclusion repeat the 10%/60% language without the qualifier 'of the additional pool,' which is a genuine overstatement.

One correction to the review notes: the random baseline numbers are present in Table 1 (the 'Random Val Acc' column), so they are not missing. The deeper issue is that random selection is only reported for the same percentages as the methods, and the comparison is fine as far as it goes.

My take: the underlying idea is coherent and the empirical claim is testable. The paper is honestly labeled as a course project. With a corrected framing, multiple runs, error bars, and a check on the linearization, it would be a reasonable workshop paper. As is, the overstatement is enough to make me wary, but I'd still send it to a serious referee rather than desk reject, because the method is simple enough to verify quickly.

Who this is for: someone working on influence functions or data subset selection who wants a quick, if thin, empirical datapoint. I wouldn't cite it in my own work in the next year, but I'd bring it up in a reading group as a cautionary example of how to (not) frame percentages.

Referee Report

4 major / 5 minor

Summary. The paper proposes influence-function-based heuristics for selecting a subset of additional training points to add to an existing logistic-regression model for binary classification. The authors derive an IP-adding approximation for the change in test predictions when a set S of samples is added, then propose six heuristic selection methods. In experiments on a binarized Stanford Sentiment Treebank, they report that Method 3, which selects 10% of the additional pool, achieves test accuracy comparable to training on all data, and that Method 1, which selects 60% of the additional pool, achieves higher accuracy than full-data training. The abstract and conclusion restate these results as achieving comparable accuracy with "only 10% of the data" and better accuracy with "60% of the data."

Significance. If the empirical claims were as stated, the work would provide a practical data-selection strategy that substantially reduces training cost while preserving or even improving accuracy. The influence-function formalism is standard and the evaluation uses a held-out test set, not the validation set used for selection, so the core comparison is not circular. However, the headline claims are not supported by the experimental design as written: the 10% and 60% figures refer to the additional pool, not the total training set. Moreover, the validity of the influence approximation for the large added subsets is untested, and the random baseline is not reported on the test set. The paper's contribution is currently a collection of heuristics with preliminary evidence, not a demonstrated reduction in total training data.

major comments (4)

[Abstract and Section 5.1, Table 1] The central quantitative claim is misstated. In Section 5.1 the original training set Ztr has 4,152 points and the additional pool Ztr' has 2,768 points, so full-data training uses 6,920 points. Method 3 selects 10% of Ztr' (about 277 points), giving a total training set of 4,429 points, which is 64% of the full dataset, not 10%. Method 1 selects 60% of Ztr' (about 1,661 points), giving 5,813 points, which is 84% of the full dataset, not 60%. The 'Added Percentage' column in Table 1 explicitly refers to the additional pool, and the abstract and conclusion inherit that number without the qualifier. Please rephrase the claims to state the fraction of the additional pool and the corresponding fraction of the total data, and add baselines trained on the same total number of points sampled from scratch to support any claim about total-data efficiency.
[Section 3.1, Eqs. (4)-(5)] The influence approximation in Eqs. (4)-(5) is a first-order expansion around the original parameters \w^c and ignores interactions among the added points. For Method 1, the added subset is 60% of Ztr' (1,661 points), and for Method 3 it is 277 points. With such large subsets, retraining moves the optimum far from the linearization point, so the ranking produced by the approximation may not reflect the true effect of adding the points. The paper does not report any validation of the approximation, such as a correlation between the IP-adding scores and the actual accuracy changes measured by retraining on small subsets, or a comparison of one-shot selection versus iterative/alternating selection. Please add such an experiment or clearly restrict the claims to the regime where the linearization is justified.
[Section 5.2 and Table 1] The random baseline is incompletely reported and the comparison to it is unsupported. Table 1 has a 'Random Val Acc' column but no corresponding 'Random Test Acc' column, yet the text states that 'on average, the random selection strategy results in a 3% performance drop compared to our proposed selection mechanisms.' Without a random-selection test accuracy and without variance estimates across multiple random draws and multiple training runs, the claim that the method outperforms random selection is not demonstrated. Please report test accuracy for random selection (with standard errors over seeds) and for the proposed methods across multiple runs.
[Section 5.2 and Table 1] The paper compares six heuristic variants and highlights the best-performing ones (Method 1 at 60% and Method 3 at 10%). Because the variants were selected after seeing validation accuracy, the reported test numbers may reflect selection among six heuristics on the validation set rather than a single prespecified method. This is a multiple-comparisons concern. The authors should either prespecify one method or report results for all variants with a correction for the number of comparisons, at minimum by disclosing that the best result was chosen from six variants.

minor comments (5)

[Section 1] The phrase 'statical perspective' appears to be a typo for 'statistical perspective'.
[Section 3.1, Eq. (4)] Equation (4) has an unmatched parenthesis in the term \(\frac{1}{N}\sum_{z_i \in S}\nabla_w L(z_i,w))\). Please correct the notation.
[Section 4, Algorithm 1] The algorithm description uses 'test point' and 'validation point' interchangeably; the pseudocode computes \Delta f over Zval, while the text sometimes refers to a test point xt. Please make the roles of the validation set and test set consistent throughout.
[Section 5.1] No hyperparameter values (e.g., the regularization strength \lambda, the threshold \tau, or the feature dimensionality after bag-of-words) are reported, which prevents reproduction. Please include these details.
[Section 5.2, Table 1] The table caption is missing a period at the end of the caption text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the empirical claim is test-set evaluated, though the headline '10%/60%' percentages are relative to the additional pool only.

full rationale

I traced the claimed derivation chain: Eq. (1)-(3) define the training objective and the retrained model; Eq. (4)-(5) are the standard first-order influence-function approximation using the Hessian and gradient at the original optimum. The selection rules in Section 4 (Methods 1-6 and Algorithm 1) compute scores from the original model, the additional pool Ztr', and the validation set Zval. The test set is held out and is never used to fit any score, threshold, or hyperparameter. The reported accuracies in Table 1 come from retraining logistic regression on the selected subsets and evaluating on that held-out test set, so the central empirical claim is not circular: the influence scores are textbook quantities, and the selection does not encode the test outcome by construction. The only notable concern is a reporting mismatch, not circularity: the abstract and Section 6 state 'using only 10% of the data' and '60% of the dataset,' whereas Table 1's 'Added Percentage' is the percentage of the additional pool Ztr' (2,768 points). With the original 4,152 points, Method 3 actually trains on about 64% of the full 6,920-point data, and Method 1 on about 84%. This overstates the data reduction, but it is an accuracy/interpretation issue rather than a self-referential derivation. The self-citations [15,16] are background literature on prediction flipping and are not load-bearing for the selection method or the empirical result. I therefore find no circular step and assign score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no fitted constants except hand-picked heuristics. The central claim relies on the linearized influence approximation and on the validation/test distribution similarity, both of which are assumed rather than verified.

free parameters (3)

Method 3 top-K fraction = 0.10
Chosen by hand to produce the reported 10% data usage; no sensitivity analysis or theoretical justification is given.
Regularization strength lambda = not reported
Appears in Eq. (1) and (2) and affects the Hessian and influence values, but the experimental section does not report its value.
Weights in Method 2 and Method 5 = unspecified
The text describes weighting and penalizing misaligned samples but gives no explicit formula, so these are unstated modeling choices that affect selection.

assumptions (3)

standard math R(w) is twice differentiable and strongly convex, as assumed in Eq. (1)-(2).
This justifies the Hessian and the influence-function expansion for logistic regression with L2 regularization.
domain assumption The validation set distribution is similar to the test set distribution.
Stated in Section 4 before Method 1; if false, selecting points by influence on validation may not transfer to test performance.
domain assumption The infinitesimal influence linearization in Eq. (4)-(5) remains accurate for finite, large additions of up to 60% of the extra pool.
The paper applies a first-order approximation to additions of hundreds of points and does not check the approximation error, making this a load-bearing assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How to Achieve Higher Accuracy with Less Training Points?." pith.science (2026). https://pith.science/paper/GBM3GG3B

@misc{pith2026250413586,
  author       = {Pith},
  title        = {Pith review of: How to Achieve Higher Accuracy with Less Training Points?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GBM3GG3B}},
  note         = {Machine review of arXiv:2504.13586}
}

read the original abstract

In the era of large-scale model training, the extensive use of available datasets has resulted in significant computational inefficiencies. To tackle this issue, we explore methods for identifying informative subsets of training data that can achieve comparable or even superior model performance. We propose a technique based on influence functions to determine which training samples should be included in the training set. We conducted empirical evaluations of our method on binary classification tasks utilizing logistic regression models. Our approach demonstrates performance comparable to that of training on the entire dataset while using only 10% of the data. Furthermore, we found that our method achieved even higher accuracy when trained with just 60% of the data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

[1]

Ahmad, R

T. Ahmad, R. Madonski, D. Zhang, C. Huang, and A. Mujeeb, “Data-driven probabilistic machine learn- ing in sustainable smart energy/smart energy systems: Key developments, challenges, and future research op- portunities in the context of smart grid paradigm,” Re- newable and Sustainable Energy Reviews, vol. 160, p. 112128, 2022. 1

work page 2022
[2]

Green ai,

R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” Communications of the ACM , vol. 63, no. 12, pp. 54–63, 2020. 1

work page 2020
[3]

A systematic review of green ai,

R. V erdecchia, J. Sallou, and L. Cruz, “A systematic review of green ai,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 4, p. e1507, 2023

work page 2023
[4]

Toward green ai: A methodological survey of the scientiﬁc literature,

E. Barbierato and A. Gatti, “Toward green ai: A methodological survey of the scientiﬁc literature,” IEEE Access, vol. 12, pp. 23 989–24 013, 2024. 1

work page 2024
[5]

An empirical comparison of instance attribution methods for nlp,

P . Pezeshkpour, S. Jain, B. C. Wallace, and S. Singh, “An empirical comparison of instance attribution methods for nlp,” arXiv preprint arXiv:2104.04128 , 2021. 1

arXiv 2021
[6]

Iaeval: A comprehensive evaluation of in- stance attribution on natural language understanding,

P . Gu, Y . Shen, L. Wang, Q. Wang, H. Wu, and Z. Mao, “Iaeval: A comprehensive evaluation of in- stance attribution on natural language understanding,” in Findings of the Association for Computational Lin- guistics: EMNLP 2023 , 2023, pp. 11 966–11 977. 1

work page 2023
[7]

Machine learning explainability in ﬁnance: an application to default risk analysis,

P . Bracke, A. Datta, C. Jung, and S. Sen, “Machine learning explainability in ﬁnance: an application to default risk analysis,” 2019. 1

work page 2019
[8]

An au- tomatic ﬁnite-sample robustness metric: When can dropping a little data make a big difference?

T. Broderick, R. Giordano, and R. Meager, “An au- tomatic ﬁnite-sample robustness metric: When can dropping a little data make a big difference?” arXiv preprint arXiv:2011.14999, 2020. 2

arXiv 2011

Show all 24 references

[9]

Explainable machine learning for pub- lic policy: Use cases, gaps, and research directions,

K. Amarasinghe, K. T. Rodolfa, H. Lamba, and R. Ghani, “Explainable machine learning for pub- lic policy: Use cases, gaps, and research directions,” Data & Policy, vol. 5, p. e5, 2023. 1

2023
[10]

Understanding black-box predictions via inﬂuence functions,

P . W . Koh and P . Liang, “Understanding black-box predictions via inﬂuence functions,” in International conference on machine learning . PMLR, 2017, pp. 1885–1894. 1, 2

2017
[11]

De- bugging tests for model explanations,

J. Adebayo, M. Muelly, I. Liccardi, and B. Kim, “De- bugging tests for model explanations,” arXiv preprint arXiv:2011.05429, 2020. 2

2011 arXiv
[12]

Explain- ing black box predictions and unveiling data ar- tifacts through inﬂuence functions,

X. Han, B. C. Wallace, and Y . Tsvetkov, “Explain- ing black box predictions and unveiling data ar- tifacts through inﬂuence functions,” arXiv preprint arXiv:2005.06676, 2020

2005 arXiv
[13]

Interactive label cleaning with example- based explanations,

S. Teso, A. Bontempelli, F. Giunchiglia, and A. Passerini, “Interactive label cleaning with example- based explanations,” Advances in Neural Information Processing Systems, vol. 34, pp. 12 966–12 977, 2021. 1, 2

2021
[14]

R. D. Cook and S. Weisberg, Residuals and inﬂuence in regression. New Y ork: Chapman and Hall, 1982. 1, 2

1982
[15]

How many and which training points would need to be removed to ﬂip this prediction?

J. Y ang, S. Jain, and B. C. Wallace, “How many and which training points would need to be removed to ﬂip this prediction?” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , A. Vlachos and I. Augenstein, Eds. Dubro...

2023
[16]

Relabeling minimal training subset to ﬂip a prediction,

J. Y ang, L. Xu, and L. Y u, “Relabeling minimal training subset to ﬂip a prediction,” in Findings of the Association for Computational Linguistics: EACL 2024, Y . Graham and M. Purver, Eds. St. Julian’s, Malta: Association for Computational Linguistics, Mar. 2024, pp. 1085–10...

2024
[17]

The inﬂuence curve and its role in ro- bust estimation,

F. R. Hampel, “The inﬂuence curve and its role in ro- bust estimation,” Journal of the american statistical association, vol. 69, no. 346, pp. 383–393, 1974. 2

1974
[18]

Characterizations of an empirical inﬂuence function for detecting inﬂuential cases in regression,

R. D. Cook and S. Weisberg, “Characterizations of an empirical inﬂuence function for detecting inﬂuential cases in regression,” T echnometrics, vol. 22, no. 4, pp. 495–508, 1980. 2

1980
[19]

Machine unlearning of features and labels,

A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck, “Machine unlearning of features and labels,” arXiv preprint arXiv:2108.11577, 2021. 2

2021 arXiv
[20]

Resolving training biases via inﬂuence-based data relabeling,

S. Kong, Y . Shen, and L. Huang, “Resolving training biases via inﬂuence-based data relabeling,” in Interna- tional Conference on Learning Representations, 2021. 2

2021
[21]

Datamodels: Understanding predictions with data and data with predictions,

A. Ilyas, S. M. Park, L. Engstrom, G. Leclerc, and A. Madry, “Datamodels: Understanding predictions with data and data with predictions,” in Proceedings of the 39th International Conference on Machine Learn- ing, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. ...

2022
[22]

Minimal explanations for neural network predictions,

O. E. Harzli, B. C. Grau, and I. Horrocks, “Minimal explanations for neural network predictions,” arXiv preprint arXiv:2205.09901, 2022. 2

2022 arXiv
[23]

Learn- ing the difference that makes a difference with counterfactually-augmented data,

D. Kaushik, E. Hovy, and Z. C. Lipton, “Learn- ing the difference that makes a difference with counterfactually-augmented data,” arXiv preprint arXiv:1909.12434, 2019. 2

1909 arXiv
[24]

Recursive deep models for semantic compositionality over a sentiment tree- bank,

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Man- ning, A. Y . Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment tree- bank,” in Proceedings of the 2013 conference on em- pirical methods in natural language processing, 2013, pp. 1631–1642. 4

2013

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Ahmad, R

T. Ahmad, R. Madonski, D. Zhang, C. Huang, and A. Mujeeb, “Data-driven probabilistic machine learn- ing in sustainable smart energy/smart energy systems: Key developments, challenges, and future research op- portunities in the context of smart grid paradigm,” Re- newable and Sustainable Energy Reviews, vol. 160, p. 112128, 2022. 1

work page 2022

[2] [2]

Green ai,

R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” Communications of the ACM , vol. 63, no. 12, pp. 54–63, 2020. 1

work page 2020

[3] [3]

A systematic review of green ai,

R. V erdecchia, J. Sallou, and L. Cruz, “A systematic review of green ai,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 4, p. e1507, 2023

work page 2023

[4] [4]

Toward green ai: A methodological survey of the scientiﬁc literature,

E. Barbierato and A. Gatti, “Toward green ai: A methodological survey of the scientiﬁc literature,” IEEE Access, vol. 12, pp. 23 989–24 013, 2024. 1

work page 2024

[5] [5]

An empirical comparison of instance attribution methods for nlp,

P . Pezeshkpour, S. Jain, B. C. Wallace, and S. Singh, “An empirical comparison of instance attribution methods for nlp,” arXiv preprint arXiv:2104.04128 , 2021. 1

arXiv 2021

[6] [6]

Iaeval: A comprehensive evaluation of in- stance attribution on natural language understanding,

P . Gu, Y . Shen, L. Wang, Q. Wang, H. Wu, and Z. Mao, “Iaeval: A comprehensive evaluation of in- stance attribution on natural language understanding,” in Findings of the Association for Computational Lin- guistics: EMNLP 2023 , 2023, pp. 11 966–11 977. 1

work page 2023

[7] [7]

Machine learning explainability in ﬁnance: an application to default risk analysis,

P . Bracke, A. Datta, C. Jung, and S. Sen, “Machine learning explainability in ﬁnance: an application to default risk analysis,” 2019. 1

work page 2019

[8] [8]

An au- tomatic ﬁnite-sample robustness metric: When can dropping a little data make a big difference?

T. Broderick, R. Giordano, and R. Meager, “An au- tomatic ﬁnite-sample robustness metric: When can dropping a little data make a big difference?” arXiv preprint arXiv:2011.14999, 2020. 2

arXiv 2011