{"id":"5d738495-5c45-4031-bd6b-051a4b0bc054","arxiv_id":"2607.21615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FrED attributes generated images and forecasts to training samples by multiplying latent similarity with a knowledge-graph rank boost, reporting LDS gains over black-box baselines on ArtBench.","lead":"Researchers propose FrED, a black-box method that traces AI outputs back to training examples by combining visual similarity scores with domain knowledge graphs. It outperforms simple similarity baselines on art attribution and shows feasibility for weather-flood analog retrieval, without needing model weights.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KG grounding is the least secure component: Table 7 shows the domain boost adds ~0.6 LDS over the sharpened visual baseline, and the contrastive projection's OOD generalization is unreported.","rationale":"The paper's central empirical claim — that FrED achieves strong black-box LDS on ArtBench — is plausibly supported by the benchmark table and by adherence to the D-TRAK protocol. The reader's conditional verdict captures the need for additional evidence around the contrastive visual-to-graph projection. My stress-test sharpens this: the domain KG's measured contribution is so small (≈0.6 LDS over the sharpened visual baseline) that even a perfectly reliable projection would not justify the framing. This does not falsify the LDS result, but it does mean the method-specific novelty ('external data influence estimation via KG grounding') is the least secure component. The reported W and k are also absent, leaving Eq. (6) non-reproducible. Since the reader already recommended a conditional verdict, my read does not move the verdict; it reinforces it with a more precise condition: the authors must show the KG term is non-negligible and the projection generalizes before the central claim is accepted.","tokens_in":21731,"tokens_out":5183,"duration_ms":57636,"concrete_test":"Recompute ArtBench-2 validation LDS using Eq. (5) with the learned domain ranks R_dom,j replaced by random integer ranks of the same distribution, or with W=0, keeping the visual-latent pipeline and all hyperparameters fixed. If LDS remains at 29.1 ± 0.65, the KG term is not contributing. In addition, report the anchor subset size and a held-out synthetic-image rank correlation to test whether the contrastive projection generalizes to generated images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FrED's headline ArtBench LDS is presented as evidence for the full dual-space method, but the paper's own ablation undercuts the KG-grounding claim. In Table 7, the visual-only pipeline (ViT-g-14 + p(c|xj)^10 + distinctiveness prior) reaches 28.5 LDS on ArtBench-2 validation, while the proposed Asymmetric Rank Boost reaches 29.1 — a gain of only 0.6, within the reported ±0.65 CI. The large jump from CLIP cosine (8.9) to 29.1 is driven by the non-linear power scaling of latent similarities (Tables 6 and 7), not by Eq. (6). Meanwhile, the domain rank R_dom,j is produced by a contrastive projection (§4.1.1, Appendix B) whose anchor subset size, sampling method, coverage, and generalization to generated images are not reported. If this projection is unreliable for synthetic out-of-distribution targets, R_dom is essentially arbitrary, and a +0.6 gain from an arbitrary rank carries no evidential weight. Moreover, W and k in Eq. (6) are never reported, so the boost term is under-specified and cannot be independently checked or tuned. Thus the load-bearing condition for the central claim — that KG grounding is what makes FrED effective — is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FrED is a black-box training-data attribution method. It fuses a continuous latent-space score (ViT-g-14 cosine similarity, raised to a power p, multiplied by a distinctiveness prior) with a discrete domain-Knowledge-Graph rank obtained via a learned contrastive projection from images to graph embeddings. The two signals are combined by an asymmetric rank boost: α_j = S_lat,j × (1 + W/(k + R_dom,j)). On ArtBench, FrED is evaluated under the D-TRAK protocol using the Linear Datamodeling Score; the headline result is 29.1% on ArtBench-2 validation and 21.3% on generated targets, compared with 8.9% for CLIP cosine and 27.61%/26.53% for D-TRAK. A second, smaller study uses the same scoring idea for historical analog retrieval in flood forecasting. The paper claims that grounding latent similarity in domain KG structure is what makes the method effective, while stressing the black-box advantage over gradient-based estimators.","tokens_in":22075,"tokens_out":8003,"duration_ms":82700,"significance":"If the empirical claims hold, FrED would be a practically valuable black-box attribution tool: it uses only model outputs and external metadata, follows the established D-TRAK protocol, shares the exact target artifacts with the baselines, and provides an open-source code link. The paper also includes useful ablation tables and a cross-domain demonstration, which are strengths. However, the central contribution — that knowledge-graph grounding is responsible for the reported gains — is not established by the current evidence. The visual-only variant already reaches 28.5 LDS, and the full KG-boosted method reaches 29.1, a difference well within the reported confidence intervals. In addition, the boost hyperparameters W and k are never reported, and the contrastive projection that produces the domain rank is under-specified. The Bayesian framing is also explicitly a heuristic surrogate. These issues are load-bearing for the paper's main claim and require careful revision.","major_comments":[{"comment":"The fusion rule is central to FrED, but W and k in Eq. (6) are never reported anywhere in the manuscript, and S_lat,j is not defined as a formula. A reader cannot reproduce the reported 29.1 LDS or independently check whether the boost term is meaningful. Please report the exact values of W, k, and the full definition of S_lat,j, together with the procedure used to select them.","section":"§3.4, Eq. (5)-(6)"},{"comment":"The ablation does not support the claim that KG grounding is the source of the improvement. The visual-only pipeline p(c|xj)^10 p(xj) reaches 28.5 ± 0.60 on ArtBench-2 validation, while the proposed Asymmetric Rank Boost reaches 29.1 ± 0.65 — a difference of 0.6, smaller than either confidence interval. The same pattern holds for generation (20.8 vs. 21.3). The large jump from CLIP cosine (8.9) to 28.5 is caused by the power scaling and the distinctiveness prior, not by Eq. (6). The paper should either report a significance test on the KG increment or substantially temper the claim that grounding in the domain KG drives the method.","section":"Table 7, §4.1.3"},{"comment":"The contrastive visual-to-graph projection is under-specified. The text says a 'representative subset of training images with rich metadata' is used as ground-truth anchors, but the subset size, sampling method, metadata coverage, and train/validation split are not reported. More importantly, generalization to generated (out-of-distribution) images is not evaluated. If the projection is unreliable for synthetic images, the domain rank R_dom,j is an arbitrary learned quantity and the small boost from Eq. (6) carries no evidential weight. Please report these details and add a quantitative OOD validation.","section":"§4.1.1, Appendix B"},{"comment":"The Bayesian derivation is presented as a probabilistic framework, but the manuscript itself notes that Eq. (2) is a 'ranking-oriented surrogate.' The likelihood P(C|X_j) is a latent cosine similarity and the prior P(X_j) is a hand-designed distinctiveness/sparsity term; the final α_j in Eq. (5) is a rank-fusion score, not a posterior probability. This is not necessarily an empirical flaw, but it is an overclaim in the abstract and introduction. Please either replace the Bayesian language with a direct statement of the proposed scoring heuristics or provide a genuine derivation from the stated generative model.","section":"§3.2, Eqs. (2)-(4)"},{"comment":"Table 1 is internally inconsistent. The header promises columns for ArtBench-2 and ArtBench-5, each with Validation and Generation and both 10 and 100 settings, but the non-parametric rows and the FrED row contain only four numeric entries each. In particular, FrED has no ArtBench-5 values, despite the text claiming results on both splits. Please correct the table, clarify what the '10' and '100' columns mean, and report the missing numbers.","section":"Table 1"}],"minor_comments":[{"comment":"Typo: 'KGa' should be 'KGs'.","section":"§2"},{"comment":"The paper-structure paragraph ends with '6.' — remove the stray numeral.","section":"§1"},{"comment":"The conclusion says FrED is 'competitive with several parametric estimators,' which is fair, but the abstract and introduction should make clear that the headline gain over simple baselines comes primarily from latent-space sharpening, not from the KG component, given the ablation evidence.","section":"§5"},{"comment":"The environmental study is clearly labeled as analog retrieval rather than causal attribution. This is a reasonable framing, but the abstract's phrase 'training data attribution' could mislead readers; consider using 'domain-grounded analog retrieval' in the abstract as well.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful empirical protocol and a potentially interesting black-box attribution idea, but the central KG-grounding claim is currently supported by a statistically negligible increment in Table 7, and the method is under-specified because W, k, and the anchor-projection details are missing. Major revision is warranted. If the authors can supply the missing hyperparameters, add significance testing, and validate the contrastive projection on generated images, the contribution could become publishable; otherwise the core claim should be substantially reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know the headline result is real but mis-assigned. FrED hits 29.1 LDS on ArtBench-2 validation, beating CLIP cosine (8.9) and D-TRAK (27.61), all without model weights. The evaluation follows the D-TRAK protocol and uses the same generated targets and training subsets, so the benchmark is solid. What the abletions show, though, is that almost all of that gain comes from the stacked latent pipeline: ViT-g-14 features, p(c|xj)^10 sharpening, and a distinctiveness prior get you 28.5. The proposed Asymmetric Rank Boost adds 0.6, inside the reported CI. So the paper's framing — that knowledge-graph grounding is what makes FrED work — does not survive its own Table 7.\n\nThat is the load-bearing weakness. W and k in Eq. (6) are never reported, so the boost term cannot be independently checked or tuned. The contrastive projection that produces the domain rank is described only as 'a representative subset of training images with rich metadata,' with no size, sampling method, coverage, or test on synthetic out-of-distribution images. If that projection fails for generated art, R_dom is arbitrary and the +0.6 is noise. Hyperparameters like p=10 and W, k appear to be selected on the ArtBench validation LDS and then reported on that same set, so some selection bias is likely.\n\nI don't want to overstate the problems. The latent-only result is genuinely interesting and worth reporting: a black-box method that matches or beats D-TRAK on validation LDS is a useful finding, even if the KG component fizzles. The weather case is honestly framed as feasibility analog retrieval, not training-data attribution, and the limitations section is candid. The relationship to the authors' own [23] is acknowledged but not benchmarked, which is a missed opportunity rather than a fatal flaw.\n\nIf the paper were reframed as 'a strong black-box baseline plus a small KG refinement,' I'd be comfortable with it after minor revision. As it stands, the central claim is overstated and the key equations are under-specified. A good referee can force the missing details and get an honest contribution out of this. I'd send it to peer review, but with the expectation that the authors either show the KG term matters outside the noise or dial back their title.\n\nFor the reading group: maybe. It's a clean case study in how ablations can undercut a paper's narrative, and the latent baseline result is worth knowing. I'd cite the work if the final version reports W, k and the anchor details; the numbers are already being circulated.","headline":"Strong black-box LDS numbers driven by a sharpened latent baseline; the KG boost is small, under-specified, and the central 'grounding' claim is not yet established.","tokens_in":22569,"tokens_out":2794,"would_cite":true,"duration_ms":30623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box estimator that fuses latent similarity with domain-graph evidence nearly matches gradient-based training-data attribution, without touching model internals.","keywords":["training data attribution","black-box influence estimation","knowledge graphs","domain grounding","generative diffusion models","linear datamodeling score","analog retrieval","weather forecasting"],"falsifier":"Permute the node labels of the domain knowledge graph while keeping its structure and the latent scores identical, and rerun the method on the same benchmark; if the linear datamodeling score does not drop substantially from the reported 29%, the domain boost is not carrying structural information. Alternatively, hold out a set of generated images with known style lineage and check whether the learned projection's domain ranking of those images agrees with ground-truth artist and style annotations.","tokens_in":21573,"feed_emoji":"🎨","tokens_out":5736,"duration_ms":58442,"temperature":0.7,"pith_summary":"The paper tries to establish that training-data influence can be estimated from outside a model, without accessing weights or gradients, by combining continuous feature similarity with a discrete domain knowledge graph. The method scores each training sample by a sharpened latent similarity to the target output, then multiplies that score by an asymmetric boost derived from the sample's rank in a curated graph of artistic or environmental relations. On an art-generation benchmark, this reaches a linear datamodeling score of 29.1% on validation, where a standard black-box similarity baseline scores 8.9% and a gradient-based estimator scores 27.6%, narrowing most of the gap to parametric methods. The same pipeline, applied to flood forecasts, improves geographic precision of retrieved historical analogs from 50.9% to 65.7%. If the claims hold, closed or proprietary generative models can be audited post-hoc for influence and provenance.","feed_headline":"Knowledge-graph boost lifts black-box attribution near gradient methods","feed_subtitle":"Fusing latent similarity with domain graphs reaches ~29% LDS on art, vs ~9% for similarity-only baselines.","key_machinery":"The central object is the asymmetric boost: a sample's final attribution score is S_lat × (1 + W / (k + R_dom)), where S_lat is the latent-engine probability (feature similarity raised to a high power times a distinctiveness prior) and R_dom is the sample's integer rank in a domain knowledge graph. The graph rank is obtained by a contrastively learned projection that maps visual features onto graph-embedding coordinates, and the boost uses reciprocal-rank decay so that graph evidence re-ranks only top latent candidates while never discarding samples with missing metadata. This lets the continuous representation carry the attribution while the discrete graph supplies historical or physical co","core_discovery":"FrED's central claim is that an attribution weight can be formalized as a posterior probability whose ranking is governed by the product of a latent-space likelihood and an informational-scarcity prior, and that this continuous score should be the primary evidence while a knowledge-graph rank acts only as a contextual boost. The reported art-benchmark results show this fusion substantially outperforms all black-box similarity baselines and approaches gradient-based estimators that require internal access: on the generation split the paper reports 21.3% versus 26.5% for a leading gradient-based method, and on validation 29.1% versus 27.6%, while a stronger parametric method reaches 38%. In a","pith_inferences":["The fairness of the comparison rests on the unstated size, sampling method, and coverage of the anchor subset used to train the visual-to-graph projection; if that subset is small or cherry-picked, the reported gains may partly reflect the projection's fit to the benchmark rather than genuine structural grounding. A direct test would vary the anchor subset and re-measure LDS.","The ablation shows that raising the similarity score to a high power, p(c|x)^10, is the largest single driver of accuracy; this hints that the method's gain may be substantially a calibration effect concentrating influence, and the knowledge graph contributes a smaller refinement. If so, the method's advantage might generalize only when influence in generative models is genuinely sparse.","The same asymmetric-boost fusion could extend to text-to-image provenance and retrieval-augmented generation, where domain graphs already exist or can be induced automatically; the main barrier is the 'expert bottleneck' of knowledge-graph construction, not the scoring mechanism.","The environmental case study stops short of causal influence; if a counterfactual evaluation protocol for weather models becomes feasible, the analog-retrieval signal could be upgraded into a true training-data attribution claim, which would be a much stronger validation of the framework."],"forward_implications":["Closed or proprietary generative models can be audited post-hoc for which training samples most plausibly shaped a given output, without requiring weight or gradient access.","Attribution responses can carry an interpretable historical or physical explanation, not just a similarity vector, because the domain graph supplies structural context.","The asymmetric fusion is robust to incomplete knowledge graphs: samples absent from the graph still receive their full latent score, so attribution never breaks due to missing metadata.","In weather forecasting, the same pipeline can retrieve physically consistent historical analogs, improving geographic localization over latent-only retrieval for decision support.","The reported art-benchmark numbers suggest black-box attribution can narrow the gap to parametric estimators on stylized image generation, opening a practical route to transparency for large deployed models."],"fun_headline_variants":["Black-box attribution nears gradient methods via knowledge graphs","Knowledge-graph grounding boosts black-box data attribution","FrED: black-box attribution closes gap to gradient methods","Domain knowledge graphs sharpen black-box attribution"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the contrastively learned mapping from images to graph coordinates, trained on a small unstated set of metadata-rich anchor paintings, reliably places unseen and synthetic images near their correct art-historical nodes; if that alignment fails, the domain rank in the boost is an arbitrary learned projection and the reported gains from knowledge-graph grounding collapse.","fun_headline_variants_meta":{"raw":{"variants":["Black-box attribution nears gradient methods via knowledge graphs","Knowledge-graph grounding boosts black-box data attribution","FrED: black-box attribution closes gap to gradient methods","Domain knowledge graphs sharpen black-box attribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1065,"prompt_tokens":748,"completion_tokens":317,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":492,"tokens_out":317,"duration_ms":3904,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:59:12.549821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the node labels of the domain knowledge graph while keeping its structure and the latent scores identical, and rerun the method on the same benchmark; if the linear datamodeling score does not drop substantially from the reported 29%, the domain boost is not carrying structural information. Alternatively, hold out a set of generated images with known style lineage and check whether the learned projection's domain ranking of those images agrees with ground-truth artist and style annotations.","supporting_citations":[],"review_version":1}