{"id":"45f5341e-5856-4a63-bc29-3c985a21436b","arxiv_id":"2504.13586","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selecting additional training examples with an influence-function score matches or beats full-data logistic regression on a sentiment task while using only 10% to 60% of the available extra data.","lead":"The authors score each extra training example by how much it would nudge a model's predictions, then keep only the examples with positive scores. On one sentiment task with logistic regression, they report matching or beating full-data training while using only 10% or 60% of the extra data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed '10% of the data' and '60% of the dataset' reductions are computed against the additional pool only, not the total training set; Method 3 actually uses 64% of the full data.","rationale":"The reader's weakest assumption concerns linearization error in the influence approximation (Eq. 4-5), which is a legitimate mechanistic concern. However, the more direct threat to the central claim is that the reported data-reduction percentages are miscalculated relative to the total training set. The paper's own Table 1 shows that the 'Added Percentage' column counts only the additional pool Ztr', not the original training set Ztr. Because the original 4,152 points are always retained, Method 3's 10% addition corresponds to 64% of all available training data, and Method 1's 60% addition corresponds to 84%. The abstract says '10% of the data' and the conclusion repeats '10% of the dataset,' which is inaccurate unless one treats the original training set as free. This is not a matter of experimental noise; it is a direct mismatch between the stated claim and the experimental setup. The reader's conditional verdict remains appropriate, but the conditions should include a corrected statement of data usage and a from-scratch baseline at the same total-data percentage before the claim can be accepted.","tokens_in":6817,"tokens_out":8923,"duration_ms":87828,"concrete_test":"Run a logistic-regression baseline that trains on a randomly drawn (or influence-selected) 10% of the entire 6,920-point training set, i.e., about 692 points with no pre-trained original model, and compare test accuracy to Method 3 and Add Full. If this from-scratch 10% baseline is well below Add Full, the paper's '10% of the data' claim is unsupported and the headline must be restated as '10% of the additional pool, 64% of total data.' Also run the analogous 60%-of-total baseline (about 4,152 points) to mirror Method 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and Section 6, is that the method achieves comparable accuracy 'using only 10% of the training samples' and performs better with '60% of the dataset.' In the experiment (Section 5.1), the original training set Ztr has 4,152 points and the additional pool Ztr' has 2,768 points, so Add Full uses 6,920 points. Method 3 selects 'top 10 percent' of Ztr' (about 277 points), so the total training set is 4,429 points, which is 64% of the full 6,920-point dataset, not 10%. Method 1 selects 60% of Ztr' (about 1,661 points), yielding 5,813 points, which is 84% of the full dataset, not 60%. The 'Added Percentage' column in Table 1 explicitly refers to the additional pool, and the abstract and conclusion inherit that number without the qualifier. Thus the central quantitative claim overstates the data reduction by factors of roughly 6.4 and 1.4. The method may still be a useful selection heuristic, but the headline as written is not supported by the experimental design: a fair '10% of total data' baseline would train on about 692 points from scratch, which is not what Method 3 does.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes influence-function-based heuristics for selecting a subset of additional training points to add to an existing logistic-regression model for binary classification. The authors derive an IP-adding approximation for the change in test predictions when a set S of samples is added, then propose six heuristic selection methods. In experiments on a binarized Stanford Sentiment Treebank, they report that Method 3, which selects 10% of the additional pool, achieves test accuracy comparable to training on all data, and that Method 1, which selects 60% of the additional pool, achieves higher accuracy than full-data training. The abstract and conclusion restate these results as achieving comparable accuracy with \"only 10% of the data\" and better accuracy with \"60% of the data.\"","tokens_in":7076,"tokens_out":2875,"duration_ms":27678,"significance":"If the empirical claims were as stated, the work would provide a practical data-selection strategy that substantially reduces training cost while preserving or even improving accuracy. The influence-function formalism is standard and the evaluation uses a held-out test set, not the validation set used for selection, so the core comparison is not circular. However, the headline claims are not supported by the experimental design as written: the 10% and 60% figures refer to the additional pool, not the total training set. Moreover, the validity of the influence approximation for the large added subsets is untested, and the random baseline is not reported on the test set. The paper's contribution is currently a collection of heuristics with preliminary evidence, not a demonstrated reduction in total training data.","major_comments":[{"comment":"The central quantitative claim is misstated. In Section 5.1 the original training set Ztr has 4,152 points and the additional pool Ztr' has 2,768 points, so full-data training uses 6,920 points. Method 3 selects 10% of Ztr' (about 277 points), giving a total training set of 4,429 points, which is 64% of the full dataset, not 10%. Method 1 selects 60% of Ztr' (about 1,661 points), giving 5,813 points, which is 84% of the full dataset, not 60%. The 'Added Percentage' column in Table 1 explicitly refers to the additional pool, and the abstract and conclusion inherit that number without the qualifier. Please rephrase the claims to state the fraction of the additional pool and the corresponding fraction of the total data, and add baselines trained on the same total number of points sampled from scratch to support any claim about total-data efficiency.","section":"Abstract and Section 5.1, Table 1"},{"comment":"The influence approximation in Eqs. (4)-(5) is a first-order expansion around the original parameters \\w^c and ignores interactions among the added points. For Method 1, the added subset is 60% of Ztr' (1,661 points), and for Method 3 it is 277 points. With such large subsets, retraining moves the optimum far from the linearization point, so the ranking produced by the approximation may not reflect the true effect of adding the points. The paper does not report any validation of the approximation, such as a correlation between the IP-adding scores and the actual accuracy changes measured by retraining on small subsets, or a comparison of one-shot selection versus iterative/alternating selection. Please add such an experiment or clearly restrict the claims to the regime where the linearization is justified.","section":"Section 3.1, Eqs. (4)-(5)"},{"comment":"The random baseline is incompletely reported and the comparison to it is unsupported. Table 1 has a 'Random Val Acc' column but no corresponding 'Random Test Acc' column, yet the text states that 'on average, the random selection strategy results in a 3% performance drop compared to our proposed selection mechanisms.' Without a random-selection test accuracy and without variance estimates across multiple random draws and multiple training runs, the claim that the method outperforms random selection is not demonstrated. Please report test accuracy for random selection (with standard errors over seeds) and for the proposed methods across multiple runs.","section":"Section 5.2 and Table 1"},{"comment":"The paper compares six heuristic variants and highlights the best-performing ones (Method 1 at 60% and Method 3 at 10%). Because the variants were selected after seeing validation accuracy, the reported test numbers may reflect selection among six heuristics on the validation set rather than a single prespecified method. This is a multiple-comparisons concern. The authors should either prespecify one method or report results for all variants with a correction for the number of comparisons, at minimum by disclosing that the best result was chosen from six variants.","section":"Section 5.2 and Table 1"}],"minor_comments":[{"comment":"The phrase 'statical perspective' appears to be a typo for 'statistical perspective'.","section":"Section 1"},{"comment":"Equation (4) has an unmatched parenthesis in the term \\(\\frac{1}{N}\\sum_{z_i \\in S}\\nabla_w L(z_i,w))\\). Please correct the notation.","section":"Section 3.1, Eq. (4)"},{"comment":"The algorithm description uses 'test point' and 'validation point' interchangeably; the pseudocode computes \\Delta f over Zval, while the text sometimes refers to a test point xt. Please make the roles of the validation set and test set consistent throughout.","section":"Section 4, Algorithm 1"},{"comment":"No hyperparameter values (e.g., the regularization strength \\lambda, the threshold \\tau, or the feature dimensionality after bag-of-words) are reported, which prevents reproduction. Please include these details.","section":"Section 5.1"},{"comment":"The table caption is missing a period at the end of the caption text.","section":"Section 5.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is explicitly labeled as an unreviewed course project. The central claims as stated in the abstract are not supported by the reported experiment because the percentages refer to the additional pool rather than the total training set. The core idea of using influence functions for data subset selection is not new, and the empirical evidence is limited to one dataset and one model. With a corrected framing and additional validation experiments, the paper could become a solid workshop contribution or a short conference paper, but in its current form it falls short of the standards of a journal publication in a serious venue. I recommend major revision rather than rejection because the claims can plausibly be reframed and the missing experiments are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing to know: the headline '10% of the data' is not what the experiment does. The 10% is 10% of an additional pool of 2,768 points, on top of an original training set of 4,152. Method 3 actually trains on 4,429 points, which is 64% of the full 6,920-point dataset. Same issue for the 60% claim: Method 1 uses 5,813 points, 84% of the full data. The stress-test note is right.\n\nWhat's genuinely new: applying influence functions to choose which points to add to a training set, with six heuristics. That's a natural extension of Koh & Liang's removal influence, and the specific IP-adding score is new. The paper is clear, the notation is fine, and it includes a random baseline column and an 'Add Full' comparison, which is more than many course projects do. Method 1 at 60% of the extra pool reaches 0.644 test accuracy versus 0.627 for Add Full; that is a plausible result.\n\nThe soft spots: single dataset (SST binary), single logistic regression, single run, no code, no error bars. The influence function is linearized at the original optimum, yet you add 1,661 points in Method 1; nobody checks whether that approximation still ranks points correctly at that scale. Six heuristics are tried; the best one is selected on the validation set, so there's selection over methods. And the abstract and conclusion repeat the 10%/60% language without the qualifier 'of the additional pool,' which is a genuine overstatement.\n\nOne correction to the review notes: the random baseline numbers are present in Table 1 (the 'Random Val Acc' column), so they are not missing. The deeper issue is that random selection is only reported for the same percentages as the methods, and the comparison is fine as far as it goes.\n\nMy take: the underlying idea is coherent and the empirical claim is testable. The paper is honestly labeled as a course project. With a corrected framing, multiple runs, error bars, and a check on the linearization, it would be a reasonable workshop paper. As is, the overstatement is enough to make me wary, but I'd still send it to a serious referee rather than desk reject, because the method is simple enough to verify quickly.\n\nWho this is for: someone working on influence functions or data subset selection who wants a quick, if thin, empirical datapoint. I wouldn't cite it in my own work in the next year, but I'd bring it up in a reading group as a cautionary example of how to (not) frame percentages.","headline":"The paper's '10% of the data' claim is actually 64% of the full training set, but the underlying influence-based selection idea is coherent and worth a quick look.","tokens_in":7571,"tokens_out":3182,"would_cite":false,"duration_ms":27114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting training points by their influence on validation predictions lets a logistic regression model match full-data test accuracy with 10% of the extra data and exceed it with 60%.","keywords":["influence functions","training data selection","data efficiency","logistic regression","sentiment analysis","subset selection","model performance","green AI"],"falsifier":"Train the model on the exact 60% subset selected by Method 1 and evaluate on the held-out test set; if the test accuracy does not exceed the 0.627 of the Add-Full baseline, the reported advantage does not reproduce. Separately, compute the per-candidate influence scores from Equation (4)-(5), retrain the model adding each single candidate alone, and compare the rank correlation between predicted and actual validation accuracy changes; a weak or negative correlation for high-scoring candidates would show the linearized ranking is not what carries the result.","tokens_in":6611,"feed_emoji":"🎯","tokens_out":7326,"duration_ms":57113,"temperature":0.7,"pith_summary":"The paper asks whether a model can be trained on far fewer, better-chosen examples instead of an entire dataset. It proposes scoring candidate training points by their influence-function effect on validation predictions, then adding only those whose predicted effect pushes validation predictions toward the correct labels. In binary sentiment classification with logistic regression, the method matches full-data test accuracy using 10% of the additional training pool and exceeds it using 60%. If the pattern holds, targeted data addition could cut training cost while improving accuracy.","feed_headline":"Influence-selected data: 10% matches full training, 60% beats it","feed_subtitle":"Training on influence-ranked picks matches or beats training on every available example, at a fraction of the cost.","key_machinery":"The mechanism is the influence function applied to adding, rather than removing, training points. Using the Hessian of the empirical risk at the fitted parameters, Equation (4) approximates the parameter change from adding a subset, and Equation (5) translates that change into predicted output changes on validation points. Each candidate point is scored by summing the predicted changes aligned with moving each validation prediction toward its correct label, with variants that weight by label discrepancy, threshold, top-K prioritization, flip counts, or misalignment penalties. Positively scored points form the subset that is added and the model is retrained.","core_discovery":"The central claim is that the influence of adding a training point on validation predictions, computed through a linearized influence-function approximation (IP-adding), provides a usable ranking for selecting a small subset of additional training data. On a binarized movie-review sentiment dataset with a logistic regression model, selecting the top few percent of additional examples by cumulative positive influence yields 0.630 test accuracy at 10% of the pool, above the 0.627 of training on the full pool; selecting 60% by the simplest variant (Method 1) yields 0.644, the highest reported. The authors interpret this as evidence that comparable or superior accuracy is achievable with a fraction of the training data.","pith_inferences":["The linearized influence approximation is only tested implicitly; directly comparing the predicted ranking against per-point retraining effects would reveal how much ranking quality degrades as the selected subset grows, especially at 60% of the pool.","Because selection is done once at the original parameters, the approach resembles a single-step active-learning procedure; iterating the selection with retraining between rounds could further reduce the required fraction, but that is an extension the paper does not test.","The validation set stands in for the test distribution, so the method's practical value depends on how well that proxy holds; under distribution shift, the selected points may be the wrong ones.","The same addition-based influence score could apply to other convex models or to mini-batch selection in neural networks, though Hessian-based approximations become more delicate there."],"forward_implications":["Subset selection by influence can replace full-data training: 10% of the additional pool gives comparable test accuracy to using 100% (0.630 vs 0.627).","Using a larger selected fraction, 60%, can improve accuracy relative to full-data training (0.644 vs 0.627).","Random selection underperforms both full-data training and influence-based selection, showing that which points are added matters, not just how many.","The scoring rule is evaluated on a validation set and requires no retraining for each candidate, making the selection step cheap relative to exhaustive search.","The heuristic variants trade off fraction selected against accuracy, and the simplest variant (Method 1) gives the best test accuracy at 60%."],"supporting_citations":[{"why":"Supplies the influence-function approximation for how training points affect predictions, on which the proposed IP-adding score is built.","marker":"[10]"},{"why":"Provides the classical influence-function theory that justifies the linearized parameter-change approximation.","marker":"[14]"},{"why":"Extends influence functions to flipping predictions by removing training points, the template the paper adapts to adding points.","marker":"[15]"},{"why":"Introduces relabeling-based prediction flipping, which informs the flip-based selection variant.","marker":"[16]"},{"why":"Supplies the binarized movie-review sentiment dataset used in the experiments.","marker":"[24]"}],"fun_headline_variants":["Influence picks 60% training, beats full set accuracy","Less training data, better accuracy: influence functions","10% matches, 60% beats: influence-selected data","Train on 60% of data, get higher accuracy","Influence-based data selection outperforms full training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The influence ranking is computed with a linear approximation around the original model's parameters, and the paper assumes that ranking remains correct even when the added subset is large enough (60% of the pool, on the order of a thousand points) to move the optimum far from the linearization point and create interactions among the added points.","fun_headline_variants_meta":{"raw":{"variants":["Influence picks 60% training, beats full set accuracy","Less training data, better accuracy: influence functions","10% matches, 60% beats: influence-selected data","Train on 60% of data, get higher accuracy","Influence-based data selection outperforms full training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2787,"prompt_tokens":785,"completion_tokens":2002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":401,"completion_tokens_details":{"reasoning_tokens":1922}},"tokens_in":401,"tokens_out":2002,"duration_ms":13031,"temperature":1.0,"reasoning_tokens":1922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:04:07.530654+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the model on the exact 60% subset selected by Method 1 and evaluate on the held-out test set; if the test accuracy does not exceed the 0.627 of the Add-Full baseline, the reported advantage does not reproduce. Separately, compute the per-candidate influence scores from Equation (4)-(5), retrain the model adding each single candidate alone, and compare the rank correlation between predicted and actual validation accuracy changes; a weak or negative correlation for high-scoring candidates would show the linearized ranking is not what carries the result.","supporting_citations":[{"cited_title":"Understanding black-box predictions via influence functions,","cited_arxiv_id":null,"evidence_quote":"Supplies the influence-function approximation for how training points affect predictions, on which the proposed IP-adding score is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the classical influence-function theory that justifies the linearized parameter-change approximation."},{"cited_title":"How many and which training points would need to be removed to flip this prediction?","cited_arxiv_id":null,"evidence_quote":"Extends influence functions to flipping predictions by removing training points, the template the paper adapts to adding points."},{"cited_title":"Relabeling minimal training subset to flip a prediction,","cited_arxiv_id":null,"evidence_quote":"Introduces relabeling-based prediction flipping, which informs the flip-based selection variant."},{"cited_title":"Recursive deep models for semantic compositionality over a sentiment tree- bank,","cited_arxiv_id":null,"evidence_quote":"Supplies the binarized movie-review sentiment dataset used in the experiments."}],"review_version":1}