{"id":"a62a2f53-52d8-45aa-8741-d7af410cf56e","arxiv_id":"2507.15381","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"PALM fits a four-parameter saturation curve to early active learning accuracy data and extrapolates it to predict the full learning trajectory.","lead":"This paper introduces PALM, a four-parameter mathematical curve that describes how a model's test accuracy grows as an active learning system labels more samples. It is meant to help researchers predict and compare active learning strategies using only a small labeling budget, without running full experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of accurate extrapolation is unsupported: fitted Amax swings from 32.6 to 100 depending on fitting points (Table 1) and BYOL is fitted as already plateaued (Section 4), with no held-out prediction error reported.","rationale":"The most load-bearing requirement for the central claim is that Eq. (13), fitted from a small number of early observations, can accurately predict the rest of the learning curve. The paper's evidence for this is visual (Figure 3) and lacks any quantitative measure of prediction error on held-out budgets. More importantly, the reported parameter estimates are unstable with respect to the number of fitting points: Table 1 shows Amax for CIFAR-10 Random ranging from 32.6 to 100 across fitting windows, and similar instability appears for delta and beta. If the fitted parameters change so dramatically with the fitting window, then predictions from limited data cannot be trusted, regardless of the model's descriptive fit. The BYOL case in Section 4 is a direct, acknowledged counterexample where the model interprets a slow early phase as a plateau, failing to anticipate later growth. These are not merely theoretical concerns about the derivation (e.g., the AUC ≈ 0 assumption in A.2.4); they are empirical signs that the parametric form is not sufficiently constrained by early data. The paper does substantial empirical work across datasets and strategies, and the code is available, which are commendable. But the central predictive claim is exactly what needs quantitative validation, and it is currently unsupported. The reader's verdict of CONDITIONAL is therefore appropriate; our stress-test does not move that verdict, but it sharpens the specific condition that must be met: a held-out extrapolation evaluation with stable parameter estimates. We agree with the reader that the weakest premise is the reliability of the parametric extrapolation, and we highlight the empirical validation gap as the concrete manifestation. This is not an attack on the authors; it is a request for the evidence that the headline claim promises.","tokens_in":22936,"tokens_out":4513,"duration_ms":48942,"concrete_test":"Compute held-out extrapolation error: for each AL strategy/dataset in Tables 1-11, fit Eq. (13) using only the first 6, 10, 20, and 50 budget points (or the earliest 2% of labeled budget), then measure MAE/RMSE between predicted and actual accuracy over all remaining budget points. Report these errors per fitting-window size. A minimal decisive variant: for CIFAR-10 Random (no embeddings), fit on the first 6 points and predict accuracy at the 500- and 1000-point budgets; compare to the actual values in the paper's data. If the held-out error is not small relative to the accuracy scale, or if predictions vary substantially across fitting windows, the predictive claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that PALM enables 'accurately predicting full learning curves from limited labeled data' (Abstract). The operative condition is that Eq. (13), A = Amax[1 - (1 - delta)^((B/b + alpha)^beta)], fitted to early budget points, extrapolates reliably to later budgets. This condition is not established and is contradicted by the paper's own data. Table 1 (CIFAR-10, no embeddings, mean of 5 runs) shows the fitted Amax for Random sampling jumping from 32.6 at 6 fitting points to 100.0 at 10-100 points and then settling to 90.7 at 1000 points; delta and beta also drift. Such non-identifiability means predictions depend on the arbitrary choice of fitting window, so 'prediction from partial observations' is not a well-defined output. Section 4 reports that for BYOL on ImageNet subsets, the early nearly-linear curve is fitted as a plateau ('PALM fits the curve as if the method has already plateaued'), a qualitative failure of extrapolation at exactly the limited-budget regime the paper claims to support. No quantitative prediction error on held-out budgets is reported anywhere; Figure 3 is visual only. Because the central claim is predictive, not merely descriptive, the absence of a held-out evaluation is load-bearing. This is an empirical validation gap, not a dispute about consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PALM, a four-parameter parametric model for active learning (AL) accuracy trajectories, A = Amax [1 - (1 - δ)^((B/b + α)^β)] (Eq. 13), where Amax is the asymptotic accuracy, δ is a coverage-efficiency parameter, α is an early-stage offset, β is a scaling exponent, and b is the budget per iteration. The authors claim that fitting these parameters to a limited number of early labeled-budget points allows accurate prediction of the full learning curve, as well as interpretable comparison of AL strategies across datasets, budgets, and self-supervised embeddings. They validate the model on CIFAR-10/100 and ImageNet-50/100/200 with several AL strategies and embeddings, reporting fitted parameters and visual curve comparisons.","tokens_in":23204,"tokens_out":2612,"duration_ms":31703,"significance":"If the predictive claim were substantiated, PALM would be a useful and interpretable tool for AL evaluation, allowing researchers to extrapolate performance from small annotation budgets and to compare strategies through meaningful parameters. The paper is commendable for targeting a real gap in AL benchmarking, for releasing code, and for reporting extensive experiments across diverse datasets and methods. The four-parameter form is simple and potentially generalizable. However, the current evidence does not establish the central predictive claim: the fitted parameters are unstable across fitting windows, no held-out prediction error is reported, the derivation contains an assumption that contradicts the paper's own motional motivation, and the BYOL experiments show a qualitative failure in the very regime the paper emphasizes. These issues are load-bearing because the paper's contribution is explicitly predictive, not merely descriptive.","major_comments":[{"comment":"The fitted parameters are highly unstable with respect to the number of points used for fitting. For Random sampling, Amax is 32.6 with 6 fitting points, 100.0 with 10, 20, 50, and 100 points, 93.8 with 500 points, and 90.7 with 1000 points; δ and β drift correspondingly. This non-identifiability means that predictions from partial observations depend on the arbitrary choice of the fitting window, so the claim that PALM predicts future performance from limited data is not well-defined unless a stable parameter regime or a principled fitting procedure is established.","section":"Supplementary Table 1 (CIFAR-10, no embeddings)"},{"comment":"No quantitative held-out evaluation is reported. Figure 3 compares predicted and actual curves visually only, and the 'within 2%' statement in the CIFAR-100 discussion refers to deviations of the full-curve fit, not to extrapolation from partial budgets. To support the headline claim, the authors should report prediction errors (e.g., RMSE or MAE in accuracy points) on held-out budget points after fitting only on early points, ideally broken down by strategy, dataset, and number of fitting points.","section":"Section 4, Limited Budgets; Figure 3"},{"comment":"The derivation of the generalized accuracy function assumes A_UC ≈ 0 to simplify Eq. (31) into Eq. (32). This directly contradicts Definition 5 and the main-text emphasis on the contribution of uncovered regions to generalization, and it is not a harmless simplification: the BYOL experiments (Section 4) show a regime where early accuracy is dominated by uncovered-region behavior and the model fits a plateau. The assumption should be stated as an explicit modeling limitation, or the derivation should be revised to include A_UC as a fitted parameter or an explicit approximation with stated validity conditions.","section":"Supplementary A.2.4, Eq. (31) to Eq. (32)"},{"comment":"The paper reports that for BYOL, whose early accuracy curve is slow and nearly linear, 'PALM fits the curve as if the method has already plateaued.' This is a qualitative failure of extrapolation exactly in the limited-budget regime that the abstract and introduction claim to support. The paper provides no quantitative error measure for this case and no guidance on when the model's extrapolation can be trusted. This issue must be addressed, for example by reporting held-out errors for slow-starting methods or by defining a validity region for predictions based on the fitted parameters.","section":"Section 4, Large-Scale Datasets (BYOL)"}],"minor_comments":[{"comment":"Many fitted parameter combinations are degenerate (e.g., δ = 0 with α ≈ 80 and β ≈ 3.7, or Amax = 100 with δ = 0.000). Reporting mean, min, and max across repetitions is helpful, but the high incidence of boundary solutions suggests that the optimization is hitting constraints; a discussion of parameter identifiability and of the quality of individual fits (e.g., R² or residual norm) would clarify which results are trustworthy.","section":"Supplementary Tables 2, 3, 5, 6, 7, 8"},{"comment":"The notation mixes B as the cumulative budget and b as the mean budget per iteration. In Definition 7, b is introduced as a known quantity, but in Section 4 the fitted parameter tables do not report b; it would be useful to state explicitly how b is determined and whether it is fixed or fitted for each experiment.","section":"Equations (13) and (14)-(15)"},{"comment":"The derivative-based comparison condition appears to be stated without derivation and has a dimensional inconsistency (δ multiplies a quantity that depends on 1/b and α/B). Since this is not needed for the main predictive claim, it should either be derived correctly or moved to a clearly labeled heuristic discussion.","section":"Supplementary A.2.7, Eq. (50)"},{"comment":"The figure's legend is hard to parse because the rows alternate between no-embedding and SimCLR-embedding settings but this is only noted in the caption. Adding row labels directly in the figure or a clearer subcaption would improve readability.","section":"Figure 3"},{"comment":"The authors write 'AUC' for accuracy in uncovered regions in Eq. (10), which collides with the standard area-under-the-curve abbreviation. A different symbol, such as A_U, would avoid confusion, especially since AUC is used in the introduction in its standard sense.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has an interesting idea and a broad experimental sweep, but the central predictive claim is not yet supported by the reported evidence. The parameter instability in Supplementary Table 1 and the BYOL plateau failure are not just presentation issues; they cast doubt on whether the model is identifiable from the limited-budget data that the paper says are sufficient. The authors should be given the opportunity to add a rigorous held-out evaluation and to discuss when the model is (and is not) reliable, but the current version is not ready for publication as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PALM is a four-parameter curve family that fits AL accuracy trajectories and compares strategies through interpretable parameters. The empirical sweep is broad: multiple datasets, strategies, SSL embeddings, with code. That is real work and the paper is readable. But the paper's central claim — accurate prediction of full learning curves from limited labels — is not established. The fitted parameters are unstable: on CIFAR-10 Random, Amax goes from 32.6 with 6 fitting points to 100 with 10–100 points. Predictions depend on the arbitrary fitting window. No quantitative held-out prediction error is reported anywhere; Figure 3 is visual. Section 4 itself admits that BYOL's early curve is fitted as already plateaued, which is a qualitative failure in the limited-budget regime the abstract highlights. The derivation in A.2.4 also assumes AUC≈0 after the text argued uncovered-region accuracy is crucial; that inconsistency matters because alpha is supposed to capture that contribution. These are not cosmetic issues; they undercut the predictive claim. What the paper does support is PALM as a compact, interpretable descriptor of observed AL curves, and as a way to compare strategies once you have enough data. That is worth having. The parameter comparison tables, despite instability, suggest real differences between embeddings and strategies. I would send this to peer review: the problem is meaningful, the experiments are substantial, and the flaws are addressable with a held-out evaluation, a stability analysis, and a clearer justification (or reparameterization) of the model. As is, I would not cite the predictive claim, but I would point to the empirical survey. Serious thinker: yes — the paper is coherent and honestly reports its own failure case, which is more than many papers do.","headline":"A useful descriptive model of AL curves, but the headline claim of accurate extrapolation from limited data is not supported by the evidence.","tokens_in":23785,"tokens_out":2062,"would_cite":false,"duration_ms":23479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that active learning accuracy trajectories follow a four-parameter curve, Eq. (13), which can be fit from a few early labeled-budget points and then extrapolated to predict the full learning curve.","keywords":["active learning","learning curve prediction","parametric model","coverage efficiency","self-supervised embeddings","annotation budget","sample efficiency"],"falsifier":"Fit Eq. (13) to the first 10% of the labeled budget for a method with a slow, nearly linear early curve (the paper's own BYOL case) and then compare the predicted accuracy at the full budget with the observed value; the central claim fails if the extrapolation departs from the measured curve by more than the roughly 2% error band that the paper reports for its fits.","tokens_in":22649,"feed_emoji":"📈","tokens_out":6510,"duration_ms":68421,"temperature":0.7,"pith_summary":"This paper proposes PALM, a parametric equation intended to describe how a model's accuracy grows as an active learning budget increases. The central claim is that fitting the equation $A = A_{\\max}[1 - (1-\\delta)^{((B/b)+\\alpha)^{\\beta}}]$ to accuracy measurements at a few early cumulative budgets predicts the full learning curve, and that the fitted parameters carry named meanings: achievable accuracy, coverage efficiency, early-stage performance, and scalability. If that claim holds, practitioners could compare active learning strategies and plan annotation budgets without labeling large fractions of a dataset. The paper validates the claim on CIFAR-10/100 and ImageNet subsets across several active learning strategies, with and without self-supervised embeddings.","feed_headline":"Four parameters predict an active learning curve from early labels","feed_subtitle":"PALM fits accuracy, coverage, early gain, and scaling from a small budget slice, then maps the whole trajectory.","key_machinery":"The central object is Eq. (13), the normalized accuracy function. It grows out of a random-covering calculation: if $B$ labeled samples are placed independently and uniformly, the expected covered fraction of the space is $1-(1-\\delta)^B$, which the paper turns into an accuracy law by assuming accuracy in covered regions is $A_{\\max}$ and then generalizing the exponent to $((B/b)+\\alpha)^{\\beta}$. The added parameters let a handful of fitted points reproduce both fast-converging and delayed learning curves, and normalizing $B$ by the per-iteration budget $b$ keeps the exponent numerically stable. This equation carries the paper's predictive claim: once its four parameters are estimated from partial observations, the whole curve, final accuracy, and label requirements for a target accuracy are determined.","core_discovery":"The paper's core discovery is that active learning accuracy curves can be summarized and extrapolated by four parameters through the normalized accuracy function $A = A_{\\max}[1 - (1-\\delta)^{((B/b)+\\alpha)^{\\beta}}]$ (Eq. 13), where $B$ is the cumulative number of labeled samples, $b$ is the mean budget per iteration, $\\delta$ is the expected fraction of the data space covered by one labeled sample, $\\alpha$ shifts the effective start of learning, $\\beta$ controls how quickly accuracy gains scale with budget, and $A_{\\max}$ is the asymptotic accuracy. Fitting these parameters by nonlinear regression to accuracy observations at a few budget points yields a predicted trajectory for the rest of the annotation process. The paper presents this as a unified and interpretable model of active learning dynamics, and reports that it generalizes across datasets, budgets, and strategies, with fitted parameters that distinguish sample-efficient methods from slow starters.","pith_inferences":["The functional form of Eq. (13) is a stretched-exponential-style growth curve, so if the parameters are stable across settings, PALM could serve as a prior or a stopping-time predictor for active learning scenarios the paper does not test, such as noisy-label or semi-supervised pipelines.","Because the derivation sets uncovered-region accuracy to zero, the parameter $\\alpha$ is left to absorb all generalization from unlabeled regions; a cleaner extension would reintroduce an explicit uncovered-region accuracy term, changing how $\\alpha$ should be interpreted.","A testable extension is to use $\\delta$ or $\\beta$ as a selection signal: an active learning strategy that greedily maximizes the estimated per-sample coverage might accelerate learning if PALM's coverage interpretation holds."],"forward_implications":["Annotation budgets can be planned from a small initial labeling run: given a few early accuracy readings, PALM predicts how many labels a target accuracy needs.","Active learning strategies can be compared by interpretable parameters ($A_{\\max}$, $\\delta$, $\\alpha$, $\\beta$) instead of final accuracy alone, exposing differences in early learning and coverage.","Self-supervised embeddings measurably raise sample efficiency under PALM, with higher $\\delta$ and lower $\\alpha$ values, giving a quantitative basis for choosing embeddings in a pipeline.","For slow-starting methods such as BYOL, PALM needs more early budget points to avoid mistaking a nearly linear rise for a plateau, as the paper itself observes.","Under tight budgets, PALM lets a practitioner pick the strategy with the best predicted return per label rather than the best end-of-run accuracy."],"supporting_citations":[{"why":"Supplies the active-learning pool notation and the dynamic-coverage perspective that PALM modifies.","marker":"[25]"},{"why":"Random covering result that motivates the expected coverage fraction $1-(1-\\delta)^B$ used as the model's starting point.","marker":"[17]"},{"why":"Introduces TypiClust, an embedding-based active learning strategy that the paper evaluates.","marker":"[14]"},{"why":"Provides the experimental settings and reproducibility framework for the active learning strategies.","marker":"[27]"},{"why":"SimCLR, the self-supervised embedding method used for feature extraction in the CIFAR experiments.","marker":"[5]"},{"why":"BYOL, a self-supervised embedding method whose slow early curve stresses PALM's extrapolation behavior.","marker":"[13]"},{"why":"The covering-lens view of active learning that underlies the coverage-accuracy relationship in the model.","marker":"[38]"},{"why":"Contributes the effective-sample-size concept that the paper links to coverage estimation.","marker":"[8]"}],"fun_headline_variants":["Four parameters forecast active learning curves early","PALM predicts full learning curves from few labels","Active learning curve forecast from a small sample slice","Fit four parameters, predict the whole annotation curve","Predict active learning performance from a partial budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that an active learning accuracy curve follows the parametric shape $A = A_{\\max}[1 - (1-\\delta)^{((B/b)+\\alpha)^{\\beta}}]$, with parameters fitted from a few early observations that remain trustworthy for extrapolation, even though the derivation assumes uncovered regions contribute negligible accuracy and labeled samples are placed independently and uniformly.","fun_headline_variants_meta":{"raw":{"variants":["Four parameters forecast active learning curves early","PALM predicts full learning curves from few labels","Active learning curve forecast from a small sample slice","Fit four parameters, predict the whole annotation curve","Predict active learning performance from a partial budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1364,"prompt_tokens":989,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":605,"tokens_out":375,"duration_ms":4042,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:33:30.845066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit Eq. (13) to the first 10% of the labeled budget for a method with a slow, nearly linear early curve (the paper's own BYOL case) and then compare the predicted accuracy at the full budget with the observed value; the central claim fails if the extrapolation departs from the measured curve by more than the roughly 2% error band that the paper reports for its fits.","supporting_citations":[{"cited_title":"Random coverings in several dimensions","cited_arxiv_id":null,"evidence_quote":"Random covering result that motivates the expected coverage fraction $1-(1-\\delta)^B$ used as the model's starting point."},{"cited_title":"Towards robust and reproducible active learning using neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the experimental settings and reproducibility framework for the active learning strategies."},{"cited_title":"A simple framework for contrastive learning of visual representations","cited_arxiv_id":null,"evidence_quote":"SimCLR, the self-supervised embedding method used for feature extraction in the CIFAR experiments."},{"cited_title":"Bootstrap your own latent-a new approach to self-supervised learning","cited_arxiv_id":null,"evidence_quote":"BYOL, a self-supervised embedding method whose slow early curve stresses PALM's extrapolation behavior."},{"cited_title":"Active learning through a covering lens","cited_arxiv_id":null,"evidence_quote":"The covering-lens view of active learning that underlies the coverage-accuracy relationship in the model."},{"cited_title":"Class-balanced loss based on effective number of samples","cited_arxiv_id":null,"evidence_quote":"Contributes the effective-sample-size concept that the paper links to coverage estimation."}],"review_version":1}