{"id":"c2c89cd6-3b81-4cfd-a438-ba4364c2d4c8","arxiv_id":"2412.01011","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A simulation finds that e-fold cross-validation, which stops folding when the confidence interval of the mean stabilizes, uses 41.5% of the energy of 10-fold cross-validation with an average 1.81% difference in results.","lead":"This paper proposes e-fold cross-validation, a way to stop training fold-by-fold once the results look stable, to save energy in recommender-system evaluation. It reports that on five algorithms and six datasets it uses 41.5% of the energy of the standard 10-fold approach while changing results by only 1.81%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The e-CV stopping rule lacks a demonstrated link between CI-width stabilization and convergence of the cumulative mean to the 10-fold value, and the free parameter α is never reported; without a random-stopping baseline, the 1.81%/41.5% trade-off is not yet evidence of reliability.","rationale":"The reader's weakest assumption is the same one I would flag: the stopping rule is an ad hoc heuristic with no formal guarantee connecting CI-width stabilization to accuracy of the cumulative mean. I agree that this is load-bearing. I add two concrete complications the reader did not emphasize: the value of α is never reported, so the 1.81%/41.5% result cannot be reproduced or checked for sensitivity, and the paper does not compare against a random-stopping baseline, so even the reported average difference does not establish that the stopping rule is better than stopping at a fixed small fold count. The paper does have real strengths: it uses 5000 random fold permutations, covers six datasets and five algorithms, and shows rank consistency in Figure 4. Those strengths make the idea worth conditional acceptance, but they do not resolve the missing validation of the stopping rule. My recommendation is unchanged from the reader: conditional, with the required additions being the α value, a sensitivity analysis, and a baseline comparison that isolates the benefit of the stopping criterion.","tokens_in":6828,"tokens_out":3986,"duration_ms":38874,"concrete_test":"Run a controlled comparison on the existing per-fold NDCG@10 scores: for each of the 30 dataset-algorithm pairs, compare e-CV's stopping rule with (i) random stopping at the same mean stopping fold and (ii) fixed stopping at fold 4, over the same 5000 permutations. Report the full distribution and the 95th percentile of |e-CV minus 10-CV|, not just the mean, and repeat e-CV for α ∈ {0.001, 0.01, 0.1, 1.0}, documenting the α actually used. If the random baseline matches e-CV's 1.81% average error, or if the result is highly sensitive to α, the claimed reliability of the stopping rule is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 defines the stopping rule: stop folding if |c_{n−1} − c_n| ≤ α / c_n, where c_n is the confidence-interval width of the cumulative mean. The central reliability claim rests entirely on this rule, but the rule only tests that the CI width is changing slowly; it does not test that the cumulative mean is close to the eventual 10-fold mean. No derivation, error bound, or calibration study is provided, so the link between a stabilized CI width and an accurate early-stopped estimate is an unverified assumption. Moreover, α is described as user-selectable, but its value is never reported, making the headline numbers (4.15 folds, 1.81% difference) non-reproducible and potentially the result of a favorable parameter choice. The paper's own Figure 2 shows that e-CV performs markedly worse for MultiVAE/Pop and for LastFM; averaging over all dataset-algorithm pairs and over 5000 permutations can hide a heavy tail of large deviations. A random stopping rule that halts at the same average fold count might achieve comparable average error, which would indicate that the CI-based criterion adds no reliability beyond deciding to stop early.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes e-fold cross-validation (e-CV), an early-stopping alternative to k-fold cross-validation for recommender-system evaluation. The method computes the cumulative mean of per-fold NDCG@10 scores and a confidence interval for that mean, then stops folding once the change in confidence-interval width satisfies |c_{n-1} - c_n| <= alpha / c_n. The authors simulate e-CV by reordering the already-computed 10-fold CV scores of 5 algorithms on 6 datasets over 5000 random permutations per dataset-algorithm pair, compare the early-stopped mean to the full 10-fold mean, and report an average percentage difference of 1.81% at an average stopping point of 4.15 folds, which they translate into 41.5% of the energy of 10-CV. They also report that the average ranking of algorithms is largely preserved. The paper concludes that e-CV is a promising energy-efficient and reliable alternative to k-fold cross-validation.","tokens_in":7099,"tokens_out":6066,"duration_ms":57342,"significance":"If substantiated, the claim that a simple stopping rule can reduce cross-validation cost by more than half while changing results by under 2% would be practically valuable for energy-conscious recommender-system research, and the topic is timely given growing attention to green machine learning. The paper uses a reasonable spread of algorithms and datasets, and the permutation-based simulation is a sensible way to explore fold-order dependence. However, the current evidence is not sufficient to support the central claim: the stopping rule has no demonstrated connection between confidence-interval stabilization and accuracy of the early-stopped mean; the free parameter alpha is never reported; the energy claim is based on fold count rather than measured energy; and the simulation uses 10-CV scores rather than true e-fold scores, so it evaluates an early-stopped 10-CV rather than a genuine e-fold procedure. The paper also lacks error bars, a random-stopping baseline, and any statistical test for the reported averages. These are fixable in a revision, but they are load-bearing for the paper's main conclusion.","major_comments":[{"comment":"The stopping rule |c_{n-1} - c_n| <= alpha / c_n tests only that the confidence-interval width is changing slowly; it does not test whether the cumulative mean is close to the eventual 10-fold mean, and no derivation, error bound, or calibration study connects these two quantities. Moreover, the value of alpha is never reported, so the headline numbers (4.15 folds, 1.81% difference) are not reproducible and could reflect a favorable parameter choice. The authors should report alpha, provide a sensitivity analysis over alpha, and ideally replace or supplement the heuristic with a rule that has a formal or empirical convergence guarantee.","section":"Section 2, stopping rule"},{"comment":"The evaluation is a simulation of early stopping within a fixed 10-fold split, not of an e-fold cross-validation procedure. Each of the 10 scores used in the simulation comes from a model trained on 90% of the data, whereas a true e-fold run with e around 4 would train each model on only 75% of the data, changing both the scores and the stopping behavior. The reported 41.5% energy saving is therefore an estimate for early-stopped 10-CV and cannot be directly attributed to e-CV without retraining models with the selected number of folds (or at least a careful discussion of why the training-set-size mismatch is negligible).","section":"Section 2, simulation methodology"},{"comment":"The statement that e-CV 'only needed 41.5% of the energy that 10-fold cross validation would need' equates stopping after an average of 4.15 folds with using 41.5% of the energy, but no energy is measured. Energy per fold is not constant: it depends on model type, dataset size, training-set size, and hardware, and for the deep models considered the cost of training on 90% versus 75% of the data is not generally proportional to the number of folds. At most the paper can claim an average of 4.15 test folds, or about 41.5% of the number of model trainings in a 10-CV run, not 41.5% of energy.","section":"Section 3, energy claim"},{"comment":"Several dataset-algorithm cells in Figure 2 show percentage differences well above the 1.81% average, especially for MultiVAE and Pop, with some bars reaching roughly 5-6%. The 1.81% average across 30 cells and 5000 permutations hides a heavy upper tail, and no standard deviations, quantiles, or error bars are reported despite the 5000 permutations. The authors should report the distribution of percentage differences (e.g., median, 90th percentile, worst-case cell) rather than only the mean, to allow readers to assess reliability in the worst-performing settings.","section":"Figure 2 and Section 3, averaging"},{"comment":"There is no baseline comparison, such as a random stopping rule that halts at the same average fold count, and no statistical test comparing e-CV's 1.81% difference to that baseline. Without such a comparison, the CI-based stopping criterion cannot be distinguished from simply deciding to stop early after a fixed number of folds, and the claim that the criterion adds reliability beyond early stopping in general is not established.","section":"Section 3, no comparison baseline"}],"minor_comments":[{"comment":"The phrase \"it's results\" should be \"its results.\"","section":"Abstract"},{"comment":"\"Our intension was\" should be \"Our intention was.\"","section":"Section 1"},{"comment":"The confidence interval is never formally defined; the authors should give the exact formula for c_n (e.g., standard error times a critical value, or a bootstrap interval) and state what distributional assumption is used.","section":"Section 2"},{"comment":"The table uses commas as decimal separators inconsistently (e.g., \"0,0669\" vs. \"7,8049\") and the Density column lacks units or a clarifying caption; use a consistent decimal notation.","section":"Table 1"},{"comment":"The bars in Figures 2 and 3 would be easier to read if the average values were labeled, and since they are averages over 5000 permutations, the figures should include error bars or at least a stated measure of dispersion.","section":"Figures 2 and 3"},{"comment":"Reference [6] (Bergman et al., 'Don't waste your time: Early stopping cross-validation') is directly related and should be compared explicitly; the current text only cites it in passing, so the reader cannot see how e-CV differs from that prior early-stopping method.","section":"References"},{"comment":"The claim that rankings 'stayed consistent' would be strengthened by reporting a rank-correlation coefficient (e.g., Kendall's tau) and the fraction of permutations in which the algorithm ranking differs between e-CV and 10-CV.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short empirical paper whose central numbers rest on an unreported threshold, a fold-count proxy for energy, and a simulation that reuses 10-CV scores rather than true e-fold evaluations. The core idea is worth pursuing, and the issues are addressable with additional experiments and analysis, so I do not recommend rejection. However, the authors should be asked to reframe the method as early-stopped k-fold cross-validation or retrain with genuine e-fold splits, and to report alpha and variance. The paper also leans heavily on the authors' own prior work (refs [5], [21], [27]); the novelty relative to [21] should be clarified in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading. This is an application of the authors' own e-CV idea (refs [5],[21]) to recommender systems; the method isn't new, but the empirical study on 5 algorithms and 6 datasets with 5000 permutations is. And the headline numbers — 41.5% energy, 1.81% difference — are weaker than they look, because energy is never measured and the stopping threshold alpha is never reported.\n\nTo their credit, the study design is clean as far as it goes: standard datasets (Amazon, MovieLens, LastFM), standard algorithms, NDCG@10, and they check ranking consistency between e-CV and 10-CV. That last check is the most practically useful part of the paper, since for model selection you mainly care about whether early stopping preserves the ranking.\n\nSoft spots, in rough order of how much they bother me:\n\n1. The energy claim is a fold-count claim. They stop at 4.15 folds on average and call that 41.5% of the energy of 10-fold CV, which assumes energy scales linearly and identically per fold and ignores per-fold differences (e.g., training on 9/10 vs 3/10 of the data costs different amounts, and some algorithms dominate). No energy meter was used, even though the group has one (ref [28]). That matters because 'energy efficient' is the paper's whole pitch.\n\n2. Alpha is user-selectable but its value is never reported. The headline results could correspond to a favorable choice. Without the value, the study isn't reproducible.\n\n3. No error bars on the 1.81% average. The paper's own Figure 2 shows much larger deviations for MultiVAE/Pop on some datasets and for LastFM; averaging over 30 dataset/algorithm pairs and 5000 permutations hides a heavy tail.\n\n4. The stopping rule is ad hoc. Stopping when the CI width stabilizes does not guarantee the cumulative mean is close to the eventual 10-fold mean; no bound or calibration is given. The paper cites Bergman et al.'s early-stopping baseline [6] but doesn't compare against it. A random stopping rule that stops at the same mean fold count would be the key control, and it's absent.\n\n5. Circularity is mild but real: the same 10-fold scores serve both as the ground truth and as the source of the early-stopped estimate, so the 1.81% is partly a partial-average-vs-full-average comparison on identical numbers. Not dishonest, but it limits what the number tells you.\n\nWho this is for: the green-recSys crowd, and anyone thinking about adaptive fold counts. It's a reasonable workshop paper with a plausible idea, but the current evidence doesn't support the strong claims in the abstract.\n\nMy call: send it to peer review, but with the expectation of substantial revision. A serious referee should ask for the alpha value, error bars, a random-stopping control, and a measured or at least properly modeled energy estimate. The idea deserves the scrutiny; the numbers as presented don't yet stand.","headline":"A clean domain-extension study of the authors' own e-CV idea, but the headline energy and reliability numbers are not yet supported: energy is a fold-count proxy, alpha is unreported, and there is no control for random stopping.","tokens_in":7611,"tokens_out":2682,"would_cite":false,"duration_ms":22529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"e-fold cross-validation, which stops folding once the running score's confidence interval stabilizes, used 41.5% of the energy of 10-fold cross-validation while producing scores that differed by 1.81% on average.","keywords":["e-fold cross-validation","energy efficiency","recommender systems","k-fold cross-validation","early stopping","confidence interval","green recommender systems","NDCG@10"],"falsifier":"A direct test would be to run e-CV on a dataset/algorithm pair where the per-fold scores are non-stationary so that the confidence-interval width stabilizes early while later folds shift the mean; if on many permutations the final e-CV score differs from the 10-fold score by substantially more than 1.81%, the claimed reliability does not hold.","tokens_in":6657,"feed_emoji":"⚡","tokens_out":7970,"duration_ms":68110,"temperature":0.7,"pith_summary":"The paper proposes e-fold cross-validation (e-CV), an early-stopping version of k-fold cross-validation for evaluating recommender systems. The idea is to run folds one at a time and stop as soon as the confidence interval around the running mean score stabilizes, so that the number of folds is chosen by the data rather than fixed in advance. In experiments with five recommender algorithms and six datasets, e-CV stopped after 4.15 folds on average, used 41.5% of the energy of a full 10-fold run, and produced scores that differed from 10-fold results by 1.81% on average. The authors conclude that e-CV is a useful energy-saving alternative that keeps evaluation results and algorithm rankings largely intact.","feed_headline":"Folding early cuts evaluation energy by 58.5 percent","feed_subtitle":"An early-stop rule needed 4.15 folds instead of 10 and changed scores by just 1.81%.","key_machinery":"The mechanism is the e-CV stopping rule. After each fold, e-CV computes the cumulative mean and the confidence-interval width $c_n$ of the scores seen so far, and stops when $|c_{n-1} - c_n| \\leq \\alpha/c_n$, where $\\alpha$ is a user-selected parameter that trades energy saving against accuracy. The intuition is that once the width stops changing, additional folds are unlikely to move the mean much. The evaluation feeds precomputed 10-fold scores in 5000 different fold orders and records the stopping point and final score for each order.","core_discovery":"The central discovery is that a simple criterion on the width of the confidence interval can reliably decide when to stop folding. The paper simulates e-CV by feeding it precomputed scores from a fixed 10-fold split in 5000 random fold orderings, and finds that the stopping rule typically fires around fold 4, with a final NDCG@10 score only 1.81% away from the full 10-fold score. Algorithm rankings produced by e-CV nearly always match those produced by 10-CV. The paper takes this as evidence that e-CV is a viable energy-efficient alternative to standard 10-fold cross-validation.","pith_inferences":["The stopping rule is a heuristic, so a formal bound on the gap between e-CV and full k-CV would be needed before relying on it for leaderboard-critical comparisons; the paper does not supply one.","Because the experiments simulate e-CV from precomputed 10-fold scores, the reported 41.5% energy figure assumes savings scale linearly with fold count; a real deployment should measure end-to-end energy including confidence-interval computation.","An adversarial fold ordering, for example scores sorted so that the interval appears stable early, could break the rule, so worst-case performance remains an open question.","The same criterion could be tested on other evaluation metrics and other supervised-learning tasks, not only NDCG@10 in recommender systems."],"forward_implications":["If e-CV works as reported, researchers can cut the energy cost of recommender-system evaluation by more than half while staying within about two percentage points of the standard 10-fold result.","It would give practitioners a data-dependent choice of fold count instead of the conventional $k=10$, and the ranking of algorithms would remain stable on typical datasets.","The energy saving would grow with dataset and model size, because the avoided folds are the expensive training runs.","The observed variability across datasets suggests that the method's reliability should be checked per dataset before use."],"supporting_citations":[{"why":"Introduces the e-CV concept that the paper implements for recommender systems.","marker":"[5]"},{"why":"Provides the related early-stopping cross-validation approach this work builds on.","marker":"[6]"},{"why":"Supports the choice of k between 5 and 10 as the accuracy baseline.","marker":"[3]"},{"why":"Recommends k=10, the ground truth against which e-CV is compared.","marker":"[17]"},{"why":"Argues that smaller k is often sufficient, motivating early stopping.","marker":"[22]"},{"why":"Documents the energy consumption of recommender-system research that motivates the work.","marker":"[27]"},{"why":"Supplies the MovieLens ratings datasets used in the experiments.","marker":"[13]"},{"why":"Supplies the Amazon review datasets used in the experiments.","marker":"[14]"},{"why":"Supplies the Last.fm listening dataset used in the experiments.","marker":"[7]"}],"fun_headline_variants":["Early-stopping CV uses 58.5% less energy, 1.81% score shift","e-fold CV: 4 folds average, 41.5% energy, 1.81% difference","Cross-validation energy down 58.5% with early stopping rule","Stop folding early: CV keeps quality, cuts energy by 58.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a stabilizing confidence-interval width signals that the cumulative mean is close to the full 10-fold mean; the stopping criterion is an ad hoc heuristic with no formal error bound.","fun_headline_variants_meta":{"raw":{"variants":["Early-stopping CV uses 58.5% less energy, 1.81% score shift","e-fold CV: 4 folds average, 41.5% energy, 1.81% difference","Cross-validation energy down 58.5% with early stopping rule","Stop folding early: CV keeps quality, cuts energy by 58.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":3899,"prompt_tokens":792,"completion_tokens":3107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":3013}},"tokens_in":408,"tokens_out":3107,"duration_ms":21868,"temperature":1.0,"reasoning_tokens":3013,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:45:53.548472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run e-CV on a dataset/algorithm pair where the per-fold scores are non-stationary so that the confidence-interval width stabilizes early while later folds shift the mean; if on many permutations the final e-CV score differs from the 10-fold score by substantially more than 1.81%, the claimed reliability does not hold.","supporting_citations":[{"cited_title":"Don't Waste Your Time: Early Stopping Cross-Validation","cited_arxiv_id":"2405.03389","evidence_quote":"Provides the related early-stopping cross-validation approach this work builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recommends k=10, the ground truth against which e-CV is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues that smaller k is often sufficient, motivating early stopping."},{"cited_title":"https: //doi.org/10.31219/osf.io/exw3j","cited_arxiv_id":null,"evidence_quote":"Introduces the e-CV concept that the paper implements for recommender systems."}],"review_version":1}