{"id":"0a20fd39-93ec-4665-a993-fa8cbfcd93f4","arxiv_id":"2508.19659","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"SCAR estimates a foundation data size from fitted concept-growth curves and uses it to guide multimodal data completion, with mixed results against random sampling.","lead":"A new scheme called SCAR scores datasets on four axes (scale, coverage, authenticity, richness) and estimates a 'foundation data size' it says is the minimum needed to preserve generalization. The paper claims this guides data collection across image, text, video, and audio, but the estimates are fitted curves and several experiments contradict the core claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 6 inverts the confidence term and drops the dataset-size factor, so the FDS estimator does not follow from Eq. 5.","rationale":"The paper aims to derive a model-agnostic dataset property from PAC bounds. The reader's central concern is that SCAR measures are plugged into Eq. 6 without derivation. My analysis shows the problem is worse than a missing derivation: Eq. 6 is algebraically inconsistent with Eq. 5. §3.2 defines I_c = 1-δ, so using I_c in place of δ flips the confidence. The missing n* factor is also consequential: since I_s = n/n*, the exponent in Eq. 5 is 2 n ε^2 = 2 n* I_s ε^2, not 2 I_s ε^2. Therefore the bound in Eq. 6 can be violated even when all SCAR assumptions hold. This directly undermines the 'theoretical framework for estimating Foundation Data Size' claimed in the abstract and contributions. The set-level FDS (Eq. 9) inherits the same error via t = exp(-2 n ε^2) and via the asserted correlated-failure model. The empirical completion results in Table 2 are plausible as a class-balancing heuristic, but they do not validate the theory. Given no formal verification or independent code inspection, the theoretical claim should be treated as unsupported. The reader's REJECT verdict is appropriate.","tokens_in":11523,"tokens_out":6335,"duration_ms":62175,"concrete_test":"Re-derive Eq. 6 from Eq. 5 using the definitions in §3.2.1–3.2.4 and check the direction and factors on the released code: for the Flickr30k/CLIP text configuration, compute I_c, I_s, I_a, I_r from the probe outputs, evaluate both the published RHS I_c exp(2 I_s (I_r-I_a)^2) and the correct RHS (1-I_c) exp(2 n* I_s (I_r-I_a)^2), and compare them with |H_j| fitted from Eq. 7. If the published bound is violated or the foundation size changes by >10% when the correct RHS is used, the FDS estimator is an artifact of the algebraic error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's Eq. 6 is the linchpin linking SCAR measures to Foundation Data Size, but it does not follow from Eq. 5. Eq. 5 requires |H_j| ≤ δ exp(2 n ε^2). §3.2.2 defines I_c ≜ 1-δ, §3.2.1 defines I_s = n/n*, and §3.2.3–3.2.4 give I_a ≜ 1-ε_j, I_r ≜ 1-ε'_j, so ε = ε_j - ε'_j = I_r - I_a. Substituting into Eq. 5 yields |H_j| ≤ (1-I_c) exp(2 n* I_s (I_r-I_a)^2). Eq. 6 instead writes |H_j| ≤ I_c exp(2 I_s (I_r-I_a)^2): the confidence term is replaced by its complement and the n* factor is missing. The complement error reverses the logic—larger coverage (smaller δ) loosens the bound instead of tightening it—and the omitted n* changes the exponent by orders of magnitude for real dataset sizes. Because Eq. 6 is the only theoretical bridge from the probe-derived SCAR values to FDS, the estimated foundation sizes are not supported by the PAC bound; the same algebraic error propagates to the set-level estimate in Eq. 9 and the completion weights in §4.2. The empirical FDS thus reduces to inverting the fitted exponential |H_j| = α_j(1-e^{-λ_j n}) from Eq. 7, making it a property of the linear probe's accuracy curve rather than of the dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAR, a set of four dataset characteristics (Scale, Coverage, Authenticity, Richness), and uses them to estimate a 'Foundation Data Size' (FDS) — the minimal data scale needed for generalization. The authors derive FDS from a PAC-style bound (Eq. 5), an exponential model of concept-space growth (Eq. 4/7), and linear-probe accuracies on downsampled subsets. They extend the estimate to a set of step functions via a Bonferroni inequality and use the resulting per-function and set-level FDS to guide a data-completion strategy in multimodal settings. Experiments cover image, text, video, and audio datasets with multiple pretrained encoders, and code is released.","tokens_in":11984,"tokens_out":9451,"duration_ms":95292,"significance":"If the derivation were sound, FDS would be a useful, model-agnostic diagnostic for data acquisition and subset selection. The paper is also commendable for its empirical breadth and for releasing code. However, the central theoretical bridge from the PAC bound to the SCAR-based FDS is algebraically incorrect (Eq. 6), the FDS formula is misstated, and the estimator ultimately inverts a curve fitted to the same probe accuracies that define the SCAR measures. As a result, the paper does not establish FDS as an intrinsic dataset property, and the experimental results are mixed rather than consistently supportive. The contribution as stated is therefore not validated by the manuscript.","major_comments":[{"comment":"Eq. (6) does not follow from Eq. (5). With the paper's own definitions I_c ≜ 1−δ, I_s = n/n*, and (I_r−I_a) = ε_j−ε'_j, Eq. (5) gives |H_j| ≤ (1−I_c) exp(2 n* I_s (I_r−I_a)^2). Eq. (6) instead uses I_c as the confidence factor and drops the n* factor. The complement substitution reverses the direction of the bound: larger coverage (smaller δ) should tighten the bound, but Eq. (6) loosens it. The missing n* changes the exponent by orders of magnitude for realistic dataset sizes. Because Eq. (6) is the only link between probe-derived SCAR values and the FDS estimate, the FDS estimates in §3.3 and §4.1 are not consequences of the PAC bound.","section":"§3.3, Eq. (6)"},{"comment":"The stated lower bound 'n_j ≥ ln(|H_j|*/δ*) 2ε*²' is algebraically wrong. Solving |H_j| ≤ δ exp(2 n ε²) for n gives n ≥ ln(|H_j|/δ) / (2 ε²). As printed, the inequality is dimensionally inconsistent and would not yield the claimed lower bound. This formula is the actual FDS estimate used in the experiments, so the numerical results are not reproducible from the equations as written.","section":"§3.3, after Eq. (7)"},{"comment":"Definition 1 requires Pr(err_{D,H}(f,D) ≥ ε) = 0 for all N > n. The practical approximation with δ=0.01 and ε=0.01 is a PAC statement that bounds the probability of error, not zero error. These are not interchangeable: the PAC bound in Eq. (2) only controls the probability of a large error, and a nonzero δ is not an 'approximation' of δ=0. The FDS is therefore not the minimal size satisfying Definition 1, and the conceptual link between the abstract definition and the estimator is missing.","section":"Definition 1 and footnote 2"},{"comment":"The 'concept space size' |H_j| is not an observed or independently estimated quantity; it is the output of the parametric curve α_j(1−e^{−λ_j n}) fitted to the same linear-probe accuracies that define the SCAR measures. FDS is then obtained by inverting this fitted curve at chosen δ* and ε*. Consequently, FDS reduces to a reparameterization of the probe's own accuracy curve. This may be a reasonable heuristic, but it contradicts the paper's claim that FDS is 'model-agnostic' and derived from a theoretical framework. The circular relationship between the fitted curve and the resulting FDS should be acknowledged and validated against an independent ground truth.","section":"§3.1 and §3.3, Eq. (7)"},{"comment":"The set-level FDS is under-specified. The text introduces 'a threshold t* at which the bound is attained with equality in step function series' but does not define how t* is computed. The subsequent instruction to 'solve exp(−2 n* ε_E^{*2}) = t* numerically' is meaningless without t*. Moreover, the positive-correlation assumption in Eq. (8) is not used in Eq. (9); the displayed Bonferroni expression is just the union bound from Eq. (1) with Pr(E_j) substituted. The set-level FDS estimate therefore lacks a clear derivation.","section":"§4.1, Eq. (8)-(9)"},{"comment":"The claim that 'SCAR consistently delivers superior or comparable performance' is not supported by Table 2. For AudioCaps, SCAR is worse than random for both text and audio in every reported encoder row (e.g., AudioCaps text with CLAP: 40.47 vs 40.97; with Pengi: 7.24 vs 7.69; AudioCaps audio with CLAP: 25.99 vs 26.27; with Pengi: 11.04 vs 11.91). For MSR-VTT video with X-CLIP, SCAR is below random (41.30 vs 43.10). These cases are not 'parity'; they are losses. The experimental section therefore does not establish the effectiveness of FDS-guided completion, and the large standard deviations (e.g., 31.30, 23.20) make several other differences statistically fragile.","section":"Table 2 and §5.2"}],"minor_comments":[{"comment":"Coverage I_c is first defined as a Jensen-Shannon divergence between empirical logit distributions and a fitted Gaussian, then set equal to 1−δ. The mapping from a distributional-divergence value to a PAC confidence parameter is asserted without justification; please clarify the probabilistic model.","section":"§3.2.2"},{"comment":"The notation I^{(k')}_r is not formally defined; the reader cannot reproduce the Bonferroni expansion without additional explanation of the index set.","section":"Eq. (1)"},{"comment":"The symbols I_s, I_c, I_a, I_r are redefined with different formulas in §4.3 than in §3.2. This is confusing; please use different names or explicitly state that these are aggregate-level counterparts.","section":"§4.3"},{"comment":"Axes and color-bar legends are missing in both panels. The prose makes quantitative-sounding claims (e.g., 'high authenticity but differ in coverage') that cannot be verified from the figure.","section":"Figure 3"},{"comment":"The caption does not define the baselines: 'rand.' and 'avg.' are used but not explained, and 'Pri. Acc.'/'Ext. Acc.' need a precise definition (accuracy on what split, after what training?).","section":"Table 2 caption"},{"comment":"The statements that 'computing n and r yields numerically equivalent results with λ' and that 'λ_j* = λ_j × n is numerically equivalent to λ_j' are unclear. λ_j is a fitted parameter; multiplying by n changes the quantity being regressed. This should be clarified.","section":"§3.2.1 and §3.3"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The load-bearing issue is not a matter of style: Eq. (6) is algebraically inconsistent with Eq. (5), the FDS formula is misstated, and the FDS estimate is effectively a fit of the probe's own accuracy curve. These issues cannot be resolved by local edits without redefining the central claims. The code release and multi-modal experimental sweep are positive, but the theoretical core needs fundamental revision before the paper can be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know two things. First, the paper's central theoretical bridge, Eq. 6, is algebraically wrong. Starting from Eq. 5 and the definitions I_c=1−δ, I_s=n/n*, and ε=I_r−I_a, the correct bound is |H_j| ≤ (1−I_c) exp(2 n* I_s (I_r−I_a)^2). Eq. 6 writes |H_j|≤ I_c exp(2 I_s (I_r−I_a)^2) — the confidence term is flipped from δ to 1−δ, and the dataset size n* disappears from the exponent. That breaks the PAC justification for the FDS estimator, and the same error propagates into the set-level bound and the completion weights. Second, Table 2 contains several rows where SCAR-guided extension lowers accuracy relative to the primary set (MSR-VTT with X-CLIP, AudioCaps with CLAP and Pengi), undercutting the prose claim that it 'avoids performance degradation.'\n\nWhat is genuinely new is the packaging: four scalar descriptors (Scale, Coverage, Authenticity, Richness) computed from a frozen-encoder linear probe, combined with an exponential growth model to estimate per-modality data needs, and cross-modal pseudo-labels to extend that to paired image-text/video/audio datasets. The experiments are broad, and the code is promised.\n\nThe soft spots are in the theory and the evaluation protocol. Definition 1 demands zero generalization error but is approximated with δ=ε=0.01, a different, weaker condition. The set-level Bonferroni polynomial is asserted, not derived. The FDS estimate is a fitted-curve inversion, so it is a property of the probe-and-dataset pair, not a model-agnostic dataset property. And because the probe is trained on k-means pseudo-labels, the reported 'accuracies' are cluster agreement, not task performance.\n\nWho is this for? Practitioners wanting a cheap, relative sense of how much more data helps a frozen-encoder linear classifier might find the SCAR indices useful. As a foundation for data acquisition theory, it does not hold up.\n\nMy recommendation: I would send this to reviewers with an explicit request to check the Section 3.3 algebra and the Table 2 claims. It is not acceptable as is, but the scaling-law heuristic could survive a major rewrite. I would not cite it in its current form.","headline":"Eq. 6 inverts the confidence term and drops the dataset-size factor, so the FDS estimator is not supported by the PAC bound; the experiments also show cases where SCAR-guided completion hurts accuracy.","tokens_in":12441,"tokens_out":5921,"would_cite":false,"duration_ms":56602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCAR is a four-axis dataset characterization scheme that estimates the minimal 'foundation data' size needed for generalization and uses it to guide multimodal data completion.","keywords":["dataset characterization","data quality","foundation data size","multi-modal learning","data-centric AI","generalization bound","cross-modal supervision","scaling behavior"],"falsifier":"Compute SCAR and FDS for a dataset with the paper's frozen-encoder linear probe, then train the same task with a different-capacity model family on random subsets of increasing size; if the predicted foundation size does not match where accuracy actually saturates, FDS is probe-specific rather than a dataset property.","tokens_in":1678,"feed_emoji":"📊","tokens_out":1963,"duration_ms":83231,"temperature":0.7,"pith_summary":"The paper tries to establish that a dataset's usefulness for a foundation model can be summarized by four stable numbers—Scale, Coverage, Authenticity, and Richness—that do not change when the dataset is scaled up or down. From these numbers it derives an estimate of the Foundation Data Size (FDS): the smallest amount of data needed to reach a target generalization error, computed with a linear probe on frozen pretrained embeddings rather than by retraining models. If this is right, data teams could audit a corpus, identify which modality or class is under-supplied, and add only samples that actually move the generalization needle. The scheme is tested across image, text, video, and audio datasets with several pretrained encoders, and the guided completion step consistently matches or beats random and class-average baselines.","feed_headline":"Four numbers size up how much data a model really needs","feed_subtitle":"SCAR scores scale, coverage, authenticity, and richness to estimate a dataset's minimal foundation size.","key_machinery":"The mechanism is a step-function task model: each binary label is a Heaviside threshold on a fixed linear projection of a lossless latent code K = φ*(X), with tasks being sets of such step functions. Concept-space growth is modeled as |H_j| = α_j(1 − e^{−λ_j n}), a saturating exponential whose fitted α_j is the asymptotic concept-space size. The SCAR-to-bound mapping sets I_s = r, I_c ≜ 1 − δ, I_a ≜ 1 − ε_j, and I_r ≜ 1 − ε′_j, producing |H_j| ≤ I_c exp{2 I_s (I_r − I_a)^2} and a lower bound n_j ≥ ln(|H_j|*/δ*)/(2ε*²). A Bonferroni-style union bound over correlated step-function errors converts per-function FDS into a set-level FDS n*. Cross-modal pseudo-supervision uses k-means clusters in","core_discovery":"SCAR models each task as a set of step functions H_j applied to a shared latent representation, and estimates the size |H_j| of the concept space behind each step function as an exponential saturation curve |H_j| = α_j(1 − e^{−λ_j n}), an assumption motivated by the exponential form of a PAC-style generalization bound. The four measures are mapped onto the terms of that bound: Scale is the downsampling ratio, Coverage plays the role of confidence δ, Authenticity plays 1 − ε (strict error), and Richness plays 1 − ε′ (loose empirical error). Plugging these into a Bonferroni-style inequality gives a per-function and set-level lower bound on the sample count—the Foundation Data Size. The paper's","pith_inferences":["A testable extension would be to compute SCAR and FDS before training, then check whether adding data beyond the predicted foundation size actually stops improving generalization; the paper's framework implies this should hold.","Because SCAR is computed from a specific frozen encoder, FDS estimates are relative to that representation; comparing datasets across encoder families may require re-normalizing the measures.","Synthetic datasets with known step-function concept spaces could directly validate whether FDS tracks true saturation, since the paper's experiments use real corpora where the ground-truth concept space is unknown.","The exponential growth model predicts diminishing returns after a certain scale, which could be used as an early-stopping rule for data acquisition even if the four SCAR indices are treated only as heuristics."],"forward_implications":["Datasets can be scored without retraining: a frozen pretrained encoder plus a linear probe yields four SCAR indices and a Foundation Data Size estimate for any modality.","If FDS is correct, data collection and pruning can target known shortfalls instead of adding random samples, because the minimal sufficient scale is estimated in advance.","Per-step foundation-size gaps give a concrete allocation rule for data completion, and the paper reports that this rule outperforms random and class-average sampling on Flickr30k, COCO Captions, MSR-VTT, and AudioCaps.","Cross-modal pseudo-supervision lets one modality's cluster structure supervise another, yielding separate FDS estimates for paired modalities and exposing cross-modal alignment gaps.","Large-scale corpora such as ImageNet, Wikipedia, and DBPedia meet their estimated Foundation Data Size, while smaller multimodal datasets fall short, indicating uneven class-level fitting."],"supporting_citations":[{"why":"Supplies the PAC-style generalization bound |H_j| exp(−2nε²) that the SCAR measures plug into.","marker":"[27]"},{"why":"CLIP provides the shared image-text embedding space used for cross-modal pseudo-supervision and FDS estimation.","marker":"[18]"},{"why":"BERT is the text encoder used to extract embeddings for text-modality SCAR estimation.","marker":"[6]"},{"why":"Confident Learning motivates the correlated-error assumption for noisy labels used in the set-level Bonferroni bound.","marker":"[16]"},{"why":"Instance-dependent label noise provides the dependency structure behind the positive correlation of step-function errors in Eq. (8).","marker":"[10]"},{"why":"ResNet-50 is the image encoder used in CIFAR experiments to compute SCAR measures.","marker":"[9]"},{"why":"The data-centric AI perspective frames the paper's goal of improving data rather than model architecture.","marker":"[24]"}],"fun_headline_variants":["SCAR: four metrics size up the data a model truly needs","Four numbers reveal a dataset's minimal foundation size","SCAR scores scale, coverage, authenticity, richness to cut data","Data needs? SCAR maps four properties to minimal sample count"],"cache_read_input_tokens":13952,"weakest_assumption_plain":"The whole estimate rests on treating three SCAR scores—coverage, authenticity, richness—as the confidence and error terms in a statistical learning bound, and on assuming that the space of learnable concepts grows as a saturating exponential curve; if either mapping is wrong, the predicted foundation size just reflects the probe's own accuracy curve rather than a property of the dataset.","fun_headline_variants_meta":{"raw":{"variants":["SCAR: four metrics size up the data a model truly needs","Four numbers reveal a dataset's minimal foundation size","SCAR scores scale, coverage, authenticity, richness to cut data","Data needs? SCAR maps four properties to minimal sample count"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1113,"prompt_tokens":784,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":528,"tokens_out":329,"duration_ms":4183,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:35:40.219543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute SCAR and FDS for a dataset with the paper's frozen-encoder linear probe, then train the same task with a different-capacity model family on random subsets of increasing size; if the predicted foundation size does not match where accuracy actually saturates, FDS is probe-specific rather than a dataset property.","supporting_citations":[{"cited_title":"Mathematical analysis of machine learning algorithms","cited_arxiv_id":null,"evidence_quote":"Supplies the PAC-style generalization bound |H_j| exp(−2nε²) that the SCAR measures plug into."},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., et al","cited_arxiv_id":null,"evidence_quote":"CLIP provides the shared image-text embedding space used for cross-modal pseudo-supervision and FDS estimation."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"BERT is the text encoder used to extract embeddings for text-modality SCAR estimation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Confident Learning motivates the correlated-error assumption for noisy labels used in the set-level Bonferroni bound."},{"cited_title":"An in- formation fusion approach to learning with instance-dependent label noise","cited_arxiv_id":null,"evidence_quote":"Instance-dependent label noise provides the dependency structure behind the positive correlation of step-function errors in Eq. (8)."},{"cited_title":"Deep residual learning for image recog- nition","cited_arxiv_id":null,"evidence_quote":"ResNet-50 is the image encoder used in CIFAR experiments to compute SCAR measures."},{"cited_title":"P., Lai, K.-H., Y ang, F., and Hu, X.Data-centric ai: Perspectives and challenges","cited_arxiv_id":null,"evidence_quote":"The data-centric AI perspective frames the paper's goal of improving data rather than model architecture."}],"review_version":1}