{"id":"25f2ff05-48bd-4e48-8c4a-9a45b423ecb0","arxiv_id":"2607.06638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"UASPL selects training samples by an evidential loss that couples label fit with adaptive uncertainty-weighted KL regularization, yielding an easy-to-hard preference and stronger average classification results than loss-only SPL.","lead":"This paper proposes UASPL, a self-paced learning method that ranks training samples using both label-fitting loss and model-generated evidential uncertainty from Dirichlet evidence networks. It aims to stop treating low-loss samples as automatically easy, and reports better average accuracy and more stable early-sample learning than many SPL baselines on UCI and image benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The reliability claim rests on a hand-designed coeff that is not held fixed when the paper moves to deep image models.","rationale":"The Reader correctly isolates the hand-designed coeff and L_total ranking as the soft spot, and notes that image experiments change the selection rule. That is the single most load-bearing concern: the theory and interpretability argument (§3.2, Fig. 4) are about one operational definition of 'reliably easy,' while the strongest non-tabular evidence uses another. I do not invent a deeper inconsistency—the tabular ablations and regularizer variants still support a useful empirical method—so the verdict stays CONDITIONAL rather than REJECT. Agreement with the Reader is full on the weakest assumption; the concrete test simply makes that assumption falsifiable on the image suite. No formal proofs or machine-checked artifacts change this assessment.","tokens_in":19488,"tokens_out":626,"duration_ms":7432,"concrete_test":"Re-run the four image benchmarks under the exact tabular selection rule: pure ascending sort on L_total from Eq. 7 (no hybrid score, no extra correct-evidence term beyond what Eq. 7 already contains), same backbone and stage schedule. If mean Accuracy drops below the strongest non-meta baselines (e.g., MW-Net / active_bias) or the MRLV / four-category ordering of Fig. 4 collapses, the reliability claim does not transfer; if performance and ordering hold, the concern is largely defused.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that putting model-generated evidential uncertainty into the SPL objective via L_total = L_EMSE + coeff_i L_KL, with coeff_i = (1-c_i)(1-u_i)+c_i u_i (Eqs. 6–8), selects 'reliably simple' samples and thereby explains the gains over loss-only SPL. That claim is load-bearing for Contributions (1)–(2) and for the four-category preference derived in §3.2. The tabular results (Tables 2–5, Fig. 4) are consistent with that design. However, the image experiments that are needed to support generality beyond UCI MLPs abandon pure sorting on L_total: they add a correct-evidence regularizer and replace selection with the stage-dependent hybrid score_i = (1-r_t) Norm(L_total) + r_t Norm(1-u_i) (Eq. 18, §4.5.2). Once selection is no longer the same operational definition of reliability that the theory section analyzes, the empirical wins on CIFAR-10 / FashionMNIST / MNIST / SVHN no longer test the same mechanism. The weakest link is therefore not that uncertainty is useless, but that the paper treats the adaptive coefficient and pure L_total ranking as the reliability mechanism while reporting image results under a different selection rule.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes UASPL, a self-paced learning method that replaces pure loss-based sample ranking with an evidential objective. An ENN parameterizes a Dirichlet distribution over class probabilities; sample difficulty is measured by L_total = L_EMSE + coeff_i L_KL, where the adaptive coefficient coeff_i = (1-c_i)(1-u_i)+c_i u_i depends on prediction correctness and evidential uncertainty (Eqs. 6–8). Derivative analysis in §3.2 is used to argue that sorting on L_total induces an interpretable four-category easy-to-hard preference. The method is claimed to be the first SPL approach that puts model-generated evidential uncertainty inside the SPL objective, to be general across hard/linear/mixture regularizers, and to improve classification performance. Empirical support includes 25 UCI datasets (50 Monte Carlo splits), many SPL and reweighting baselines, ablations of Dirichlet modeling and KL weighting (Table 5), generality tables (Tables 3–4), and additional image-benchmark results under deep backbones.","tokens_in":19913,"tokens_out":1246,"duration_ms":13499,"significance":"If the reliability claim holds, the paper offers a concrete and reusable way to inject model-internal predictive reliability into SPL without external priors, with a clear selection story and code release. Strengths include a broad UCI evaluation with many baselines (including adapted uncertainty-SPL variants), ablations that isolate Dirichlet modeling and the adaptive KL coefficient (Table 5), generality checks across three regularizers, and supplementary noise/pretraining/Wilcoxon analyses. The contribution is incremental rather than foundational—it sits at the intersection of EDL and SPL—but it is practically useful if the same reliability mechanism is shown to drive gains under both tabular and deep-image settings.","major_comments":[{"comment":"The load-bearing reliability mechanism analyzed in §3.2 is pure ranking by L_total with the adaptive coeff_i of Eqs. 6–8. In §4.5.2 the image experiments abandon that operational definition: they add a correct-evidence regularizer and replace selection by the stage-dependent hybrid score_i = (1-r_t) Norm(L_total) + r_t Norm(1-u_i) (Eq. 18). Consequently, the CIFAR-10/FashionMNIST/MNIST/SVHN results (Table 6) do not test the same selection rule that Contributions (1)–(2) and the four-category preference claim. Either re-run image experiments under pure L_total ranking (or report both), or explicitly restate the image protocol as a separate practical variant and limit the theoretical reliability claim to the tabular setting.","section":null},{"comment":"The interpretability argument in §3.2 is partly by construction: L_total is designed so that sorting yields correct/low-u → correct/high-u → incorrect/high-u → incorrect/low-u, and Fig. 4 then visualizes that order. That is useful design analysis, but it does not by itself establish that the selected samples remain ‘reliably easy’ for later stages. The MRLV diagnostic (Figs. 1 and 3) is only a proxy. Strengthen the claim with an independent check—e.g., hold-out difficulty labels, later-stage loss trajectories stratified by the four categories, or a controlled comparison against a non-adaptive coeff—so that reliability is not identified solely with the designed ranking.","section":null},{"comment":"Table 6 shows UASPL is competitive on images but not uniformly superior (e.g., CIFAR-10 mean accuracy trails MW-Net; several methods are near saturation on MNIST). Given that the image protocol already differs from the analyzed objective, the paper should either (i) provide matched ablations of Eq. 18 and the correct-evidence term under deep backbones, or (ii) temper the generality claim so that ‘outperforms other SPL methods … and generality’ is not read as applying equally to the deep-image regime.","section":null}],"minor_comments":[{"comment":"Algorithm 1 is hard to follow: the first for-loop over epochs appears to only compute α/S, while selection and updates sit outside a clear outer training loop; clarify the pacing schedule and when λ is updated.","section":null},{"comment":"Notation for Dirichlet parameters and evidence (α_ij, e_ij, S_i) is introduced more than once with slight redundancy between §2.1 and §3.1; a single consistent block would help.","section":null},{"comment":"Table 1’s checkmarks for ‘Internal / Dynamic / Selection criterion’ are useful but slightly overloaded; a short footnote defining each column would reduce ambiguity.","section":null},{"comment":"Several detailed results are deferred to the Supplementary Material (F1/Precision/Recall tables, noise and Wilcoxon tests). Ensure the main text states the key numerical takeaways so the paper is self-contained for readers who do not open the appendix.","section":null},{"comment":"Minor prose issues: ‘this article’ vs ‘this paper’, occasional long sentences in §3.2, and arXiv ID / date formatting should be cleaned for journal submission.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core tabular idea is publishable after revision; the main risk is overclaiming a single reliability mechanism while the strongest non-tabular results use a different selection rule. I would not reject on novelty grounds—EDL+SPL is a reasonable contribution—but I would require the authors to align theory and image protocol before acceptance. Fit for a solid ML journal is fine if the revision is honest about scope."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit here is not “uncertainty + SPL” in the abstract—that already exists in SPUP and WSPLBF—but putting model-internal Dirichlet uncertainty into the SPL objective itself via an adaptive KL coefficient that depends on correctness and u. That is a concrete recipe, not just an external prior, and the paper is honest about the distinction in Table 1.\n\nWhat works: the tabular campaign is serious. Twenty-five UCI sets, 50 Monte Carlo splits, hard/linear/mixture regularizers, ablations of Dirichlet modeling and of the adaptive coeff vs fixed/annealed KL (Table 5), plus noise, pretraining, and Wilcoxon checks in the supplement. Gains are modest but consistent (accuracy ~0.8285, best rank and win count, lower variance). Fig. 4 and the derivative analysis make the four-category order readable, even if that order is partly designed into L_total. Code is linked. Citation pattern is fine; they engage the right priors rather than papering over them.\n\nSoft spots, in proportion. The interpretability story is mild design circularity: sort on a loss built to prefer correct/low-u first, then observe that order. That does not invalidate the accuracy numbers. The real soft spot is the image section. Once they move to ResNet-style backbones they add correct-evidence regularization and switch to a stage-dependent hybrid score (Eq. 18) instead of pure L_total ranking. So CIFAR/FashionMNIST/MNIST/SVHN do not test the same mechanism the theory section analyzes. That weakens the “generality” claim without killing the tabular result. Free knobs (pacing λ, pretrain length, coeff form, hybrid weight) are normal for this literature; they are design choices, not hidden fraud.\n\nWho it is for: people who actually train curriculum/SPL pipelines and care about sample selection under uncertainty. Not a theory paper. I would send it to peer review; a referee should demand a cleaner image protocol (same selection rule, or an explicit ablation of the hybrid change) and a clearer statement that the reliability claim is operational, not proven. Worth engaging if you work in this area; I would cite the tabular recipe and the comparison table, not the image generality claim as currently written.","headline":"Solid empirical SPL paper: internal evidential uncertainty in the selection loss is the real move; image results quietly change the selection rule, so treat generality as provisional.","tokens_in":20501,"tokens_out":553,"would_cite":true,"duration_ms":6946,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Self-paced learning selects more reliably simple samples when model-generated evidential uncertainty is folded into the loss, not just the training error.","keywords":["self-paced learning","evidential deep learning","uncertainty estimation","Dirichlet evidence","sample selection","Subjective Logic","curriculum learning"],"falsifier":"On held-out stages or datasets, check whether samples that UASPL ranks easiest in the first round keep smaller loss and higher correctness later than loss-only SPL selections; if first-round UASPL picks later become more unstable or less accurate than the baselines, the reliability claim fails.","tokens_in":20372,"feed_emoji":"🎯","tokens_out":625,"duration_ms":7215,"temperature":0.7,"pith_summary":"Self-paced learning trains models by starting with easy samples and adding harder ones, but \"easy\" is usually defined only by a small loss. The authors show that early low-loss samples can later become unstable, so small loss alone does not mean a sample is reliably simple for the current model. They propose UASPL: an evidential neural network that turns predictions into Dirichlet evidence and uncertainty, then builds a single sample loss that combines label fit with an uncertainty-weighted KL term. Sorting on that loss yields an interpretable order—correct low-uncertainty first, then correct high-uncertainty, then incorrect high-uncertainty, then incorrect low-uncertainty—and the same idea plugs into hard, linear, and mixture self-paced regularizers. Across many UCI tables and four image benchmarks, the method improves accuracy and stability over loss-only SPL, uncertainty-prior SPL, and several reweighting baselines, while the selection order matches the stated preference.","feed_headline":"Low loss is not enough: uncertainty picks truly easy samples","feed_subtitle":"Evidential self-paced learning ranks samples by reliability and beats loss-only curricula.","key_machinery":"The uncertainty-aware sample loss L_total = L_EMSE + coeff_i L_KL, with coeff_i = (1-c_i)(1-u_i)+c_i u_i, which couples prediction correctness and evidential uncertainty so that sorting samples by L_total induces the four-category easy-to-hard preference.","core_discovery":"UASPL claims that embedding model-generated evidential uncertainty together with the label-fitting loss inside the self-paced objective selects reliably simple samples, not merely low-loss ones, and that the resulting ranking is both more accurate and more interpretable than loss-only or externally prior-based self-paced methods.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Uncertainty not low loss picks truly easy samples","Evidential reliability ranks samples beyond mere loss","UASPL embeds uncertainty for interpretable easy-sample selection","Self-paced curricula guided by evidential predictive reliability","Uncertainty-aware loss selects reliable samples over low-loss ones"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the hand-designed coefficient and the sorted total loss really track what will stay easy for the model later, not just how confident the current evidence looks.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty not low loss picks truly easy samples","Evidential reliability ranks samples beyond mere loss","UASPL embeds uncertainty for interpretable easy-sample selection","Self-paced curricula guided by evidential predictive reliability","Uncertainty-aware loss selects reliable samples over low-loss ones"]},"model":"grok-4.5","effort":"low","cost_usd":0.00329,"raw_usage":{"total_tokens":1095,"prompt_tokens":724,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":32900000,"prompt_tokens_details":{"text_tokens":724,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":311,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":724,"tokens_out":60,"duration_ms":4035,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T00:47:29.546236+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On held-out stages or datasets, check whether samples that UASPL ranks easiest in the first round keep smaller loss and higher correctness later than loss-only SPL selections; if first-round UASPL picks later become more unstable or less accurate than the baselines, the reliability claim fails.","supporting_citations":[],"review_version":1}