{"id":"4e9a36b3-28e7-479a-be4b-e427cb6e3257","arxiv_id":"2507.03004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A collaborative data-selection method that scores each private sample's influence on a public anchor set and filters by a global threshold before federated learning or model merging.","lead":"CLUES proposes to filter low-quality training data in collaborative LLM fine-tuning by scoring each sample's influence on a public anchor dataset, then applying a single global threshold across all clients. The method is tested on medical, multilingual, and financial QA tasks and shows gains over perplexity, IFD, and DataInf baselines in some setups.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The global threshold—mean of 10 anchor scores—is the least secure link: Algorithm 1 does not specify how the server computes anchor scores, and no anchor-size or scale sensitivity is reported; the 96%-of-upper-bound claim rests on this calibration.","rationale":"The reader's weakest assumption correctly identified the global threshold as the fragile component. I refine this by pointing out an even more basic issue: the paper never defines what model or checkpoints the server uses to score anchor data. In Algorithm 1, Stage 3 instructs the server to compute S(z) for anchor data, but the server does not have access to the per-checkpoint models produced during private client training. Without a precise recipe, the threshold tau is not well-defined, and the selection rule could vary arbitrarily depending on implementation choices. The paper's own Table 3 shows that the threshold rule is decisive, so this is not a minor implementation detail. The absence of anchor-size sensitivity compounds the problem: 10 anchors drawn from a held-out pool may not represent the score distribution across heterogeneous clients, and no error bars or repeated draws are provided. I do not treat the missing TracIn citation or the Oracle-beaten cells as the primary objection because those concern novelty and noise, respectively; the threshold calibration issue is more fundamental because it determines whether the method has a well-defined output at all. I keep the reader's CONDITIONAL verdict: the pipeline is plausible and the released code may allow the authors to supply the missing specification and sensitivity analysis. If the proposed sweep shows that selection quality is stable over anchor size and that server-side anchor scoring can be implemented without scale mismatch, then this concern would be resolved and the empirical claim would be substantially strengthened.","tokens_in":17859,"tokens_out":10649,"duration_ms":129513,"concrete_test":"Implement Algorithm 1 with an explicit server-side anchor scoring recipe and run a controlled sweep: for the MMedBench federated setting and the MedicalQA setting, draw bootstrap anchor sets of size 1, 2, 5, 10, 20, and 50 from the held-out high-quality pool (at least 10 draws each), recompute tau from the average anchor score, and report selection accuracy/F1 plus final downstream GPT-4 scores. Also run a scale-mismatch control: compute anchor scores with the server's initial global model theta0 while client scores use the locally fine-tuned checkpoints, which is the only specification Algorithm 1 currently supports. If selection F1 changes by more than 5 points or the comparison with baselines reverses, the global threshold, and with it the main claim, is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on the CLUES selection rule: keep every sample whose influence score is at least tau, where tau is the average influence score of only 10 public anchor samples (Section 3.3). This rule is load-bearing because the method's only improvement over per-client scoring baselines is the global threshold; Table 3 shows that changing the threshold rule changes selection accuracy from 75% to 97.9%. The weakness is twofold. First, the anchor-score computation is underspecified: Algorithm 1 Stage 3 tells the server to compute S(z) for anchor data using per-checkpoint gradients, but in the model-merging setting the server has no checkpoints from private client training, and in the federated setting the paper does not state which global-model checkpoints are used. If anchor scores are computed from a different model or checkpoint set than client scores, their scales are not comparable, and tau can retain or discard nearly arbitrary fractions of each client's data. Second, no sensitivity analysis is given for anchor size or anchor sampling; with n=10 and no error bars, a single unrepresentative anchor can shift tau enough to change selection decisions. The claimed over-96%-of-theoretical-upper-bound performance is therefore contingent on an unvalidated calibration constant.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CLUES, a data-selection pipeline for collaborative fine-tuning of LLMs under model merging or federated learning. Client models are trained on mixed-quality private data, per-sample influence scores are computed from gradient inner products against a public validation set, and a global threshold is determined on the server from the average score of a small set of public anchor samples. Samples below the threshold are discarded, and the clients retrain on the filtered data before merging or federated aggregation. The authors evaluate on medical, multilingual, and financial QA tasks with Llama-2 and Mistral, comparing against perplexity, IFD, and DataInf baselines, and they report that CLUES often outperforms the baselines and approaches the Oracle upper bound. The manuscript also includes ablations on threshold rules, merging techniques, gradient layer choice, and varying pollution ratios, with code released publicly.","tokens_in":18067,"tokens_out":6100,"duration_ms":66818,"significance":"If the empirical claims were reliable, CLUES would be a useful contribution to collaborative data curation: it is a general, scalable influence-based selection method that respects data-sharing constraints, and it provides a unified treatment of model merging and federated learning. The release of code and the bi-level optimization interpretation are strengths. However, the current empirical support is not yet convincing because several reported results violate the paper's own theoretical upper bound, the 96%-of-Oracle claim is contradicted by the authors' tables, and the global threshold mechanism, which is the key novel component, is underspecified and lacks sensitivity analysis. These issues are load-bearing for the central claim and must be resolved before the results can be accepted.","major_comments":[{"comment":"The Oracle is defined in Section 4.1 as the theoretical upper bound obtained by training only on the remaining high-quality data, yet CLUES exceeds the Oracle in multiple cells. In Table 1 (Llama2, GPT-4 Scoring) CLUES achieves 0.210 while the Oracle achieves 0.099; in Table 2 (Llama2, Federated) CLUES achieves 0.477 while the Oracle achieves 0.449; and in Table 1 (Mistral, GPT-4 Scoring) CLUES achieves 0.161 while the Oracle achieves 0.160. Under the paper's own definition, this is internally inconsistent, and it indicates either that the Oracle construction is flawed (for example, the 'low-quality' pollution may not be harmful under the GPT-4 metric) or that the evaluation metric has enough variance to invert rankings. The paper reports no variance or significance measures for the GPT-4 scores, so the two possibilities cannot be distinguished. This directly undermines the 'over 96% of the theoretical upper bound' claim in Section 4.2.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The claim that CLUES achieves over 96% of the theoretical upper bound is not supported by Table 1. On MedicalQA with Llama2, the Knowledge Avg score for CLUES is 0.356 versus an Oracle of 0.440, which is 80.9%; with Mistral, CLUES is 0.309 versus Oracle 0.233, which is 132.6%. The 96% figure appears to apply only to the multilingual setting in Table 2, and even there the Llama2 federated row shows 0.477 versus an Oracle of 0.449, exceeding 100%. The authors should either revise the claim to state precisely which table and which metric support it, or report corrected numbers if the Oracle or the evaluation is erroneous.","section":"Section 4.2, Table 1"},{"comment":"The computation of the global threshold is underspecified in a way that affects comparability of scores. In the model-merging setting (communication round cr=1), clients send only their final LoRA adapters to the server, but Algorithm 1 Stage 3 instructs the server to compute anchor scores using per-checkpoint gradients; no checkpoints are available on the server in this setting. In the federated setting, the paper does not state which global checkpoints are used for anchor scoring or how their time indices align with the client-side checkpoints used for sample scoring. If the anchor scores and client scores are computed under different model checkpoints or different training trajectories, their scales are not comparable, and the threshold τ can retain or discard nearly arbitrary fractions of each client's data. This is a load-bearing issue because the global threshold is the only mechanism that distinguishes CLUES from per-client scoring baselines.","section":"Section 3.3, Algorithm 1"},{"comment":"No sensitivity analysis is provided for the anchor set size or anchor sampling, even though the global threshold is the mean score of only 10 public anchor samples. Table 3 demonstrates that the choice of threshold rule changes selection accuracy from 75.00% (select-by-ratio) to 95.00% (pre-determined score) to 97.91% (global threshold), so the threshold is the primary determinant of success. With n=10, a single unrepresentative anchor can shift τ enough to change selection decisions for many samples, and no confidence intervals are reported. The authors should include experiments varying the anchor set size and multiple random anchor draws to show that the threshold is stable.","section":"Section 3.3, Table 3"}],"minor_comments":[{"comment":"The caption contains the typo 'Webold' instead of 'We bold', which should be corrected.","section":"Table 1 caption"},{"comment":"The baseline citations in the Table 1 caption are inconsistent with the reference list and with the text: IFD is cited as [25] in Section 2 but as [20] in the caption, and DataInf is cited as [22] in Section 2 but as [25] in the caption. The caption should use [25] for IFD and [22] for DataInf.","section":"Table 1 caption and references"},{"comment":"The notation for the learning-rate weight is inconsistent: Equation (3) uses \\bar{\\eta}_i while Equation (6) uses \\eta_i, and the summation limits differ between T and earlier definitions of T. The authors should unify the notation and clarify that T denotes the number of checkpoints.","section":"Section 3.2, Equations (3) and (6)"},{"comment":"The text says the gradient computation process is 'based on one single checkpoint,' but Algorithm 1 loops over checkpoints t in T for each training sample. This apparent contradiction should be resolved by stating explicitly whether the score aggregates over multiple checkpoints.","section":"Section 3.2, Algorithm 1"},{"comment":"The selection condition is written as 'D'(k)_train = {zi in D(k)_train, zi >= tau}', which compares a data sample directly to a score threshold. It should read 'S(zi) >= tau' to compare the score of the sample, not the sample itself.","section":"Algorithm 1, Stage 4"},{"comment":"No standard deviations or repeated-run statistics are reported for any of the main tables, so the reader cannot assess whether the differences between CLUES and the baselines are within evaluation noise. Please add at least three seeds or bootstrap confidence intervals for the key comparisons.","section":"Section 4.1 and Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the code release is a positive feature, but the empirical contradictions around the Oracle bound and the underspecified global threshold currently prevent acceptance. I recommend major revision: the authors should correct or re-run the Oracle experiments, provide a concrete specification with sensitivity analysis for the threshold, and report variance for the GPT-4 evaluations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: CLUES is a sensible pipeline for collaborative data selection with public anchors, but the results as presented don't support the abstract's claims. The core scoring is TracIn without credit, and the Oracle being beaten in several cells means either the upper bound isn't one or the evaluation is noisy. Still, the system-level idea is worth a serious look.\n\nWhat's actually new and good: the framing of data selection as a collaborative problem where a server sets a global threshold using a small public anchor set, then clients filter locally before merging or federated averaging, is a useful contribution. The experiments span medical, multilingual, and financial QA with two base models, and the ablations on layer choice, merging technique, and threshold rule are informative. The writeup honestly lists limitations, and the code is available.\n\nSoft spots: First, the Oracle inconsistencies. Table 1 Llama2 shows CLUES at 0.210 vs Oracle 0.099, and Table 2 Llama2 federated shows 0.477 vs 0.449. If Oracle is truly training on the high-quality subset, exceeding it shouldn't happen. The authors owe an explanation—maybe the Oracle definition leaks through the anchor/validation loop, or the selected data actually helps via interference reduction. Without error bars, these differences could be noise. Second, the global threshold: mean of 10 anchor scores—no sensitivity analysis, and the algorithm doesn't specify which checkpoints are used on the server in the merging setting. The stress-test note gets this right. Third, Eq. (3) is the TracIn score without citation or comparison. That's a scholarly-acceptability issue, not a technical one. Fourth, the quality definition is circular relative to validation loss reduction, but this is a common pragmatic stance in influence-based selection; less severe.\n\nWho benefits: researchers working on federated data curation or model merging, and anyone evaluating training-dynamics-based selection. The paper is plausible enough to warrant peer review, but as it stands the central '96% of upper bound' claim is not established. I'd send it to review with a strong request to fix the Oracle contradictions and add threshold sensitivity analysis before acceptance.","headline":"Useful collaborative data-curation pipeline, but the headline numbers overclaim: the scoring is uncited TracIn, the Oracle is beaten, and the anchor-based threshold is unvalidated.","tokens_in":18658,"tokens_out":2919,"would_cite":false,"duration_ms":34197,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that collaborative fine-tuning of LLMs can be quality-controlled by scoring each private sample's influence on a small public anchor set, and that this beats local filtering baselines.","keywords":["data selection","training dynamics","data influence","collaborative fine-tuning","federated learning","model merging","low-rank adaptation","anchor data"],"falsifier":"Take a client whose private domain is far from the public anchor domain and whose low-quality data are adversarial paraphrases, then vary the anchor set from 1 to 100 samples and across domains; record selection precision/recall and downstream QA score. If the global threshold swings enough to drop selection F1 below the best per-client baseline, the claim that one small anchor set defines a reliable unified standard fails.","tokens_in":17623,"feed_emoji":"🎯","tokens_out":6918,"duration_ms":73174,"temperature":0.7,"pith_summary":"CLUES tries to establish that data quality for collaborative LLM fine-tuning can be judged without sharing raw data, by measuring how each private training sample moves the model along the same trajectory as a small public 'anchor' set. The method scores every sample as the learning-rate-weighted inner product of its gradient updates with anchor-sample gradient updates across saved checkpoints, then applies a single global threshold equal to the average anchor score, so clients with different contamination levels keep different fractions of their data. In medical, multilingual, and financial QA experiments, training on the selected samples often outperforms per-client perplexity, IFD, and DataInf baselines, reaching over 96% of the oracle upper bound that trains only on known-clean data in federated settings. If correct, this gives a privacy-preserving data curation pipeline that fits into both model merging and federated learning with only server-side changes.","feed_headline":"Influence scores pick the best data for private LLM fine-tuning","feed_subtitle":"CLUES filters each client's data by one global anchor threshold, reaching 96% of the oracle in federated QA.","key_machinery":"The load-bearing object is the training-dynamics influence score, $S(z)=\\sum_{z'}\\sum_{t=1}^{T}\\bar{\\eta}_t\\,\\mathcal{L}(z',\\theta_t)\\cdot\\mathcal{L}(z,\\theta_t)$, where $\\mathcal{L}$ is the Adam/AdamW gradient transform used for the update, $\\theta_t$ are checkpoints saved during local fine-tuning, and the inner product is taken between the update direction of a private sample $z$ and that of a public validation/anchor sample $z'$. Gradients come from the first low-rank layer of LoRA adapters, which the paper argues avoids the cancellation effect that weakens last-layer influence scores. The second load-bearing piece is the global threshold: the server computes the same score for 10 public anchor samples and uses their average as a unified quality bar that each client applies independently, turning data selection into a server-side decision that needs no raw data and no client-specific selection ratio.","core_discovery":"The central claim is that a training sample's quality is readable from its training dynamics relative to a public anchor set: high-quality private data move the model in directions that resemble the anchor data's directions during optimization, while corrupted or mismatched samples diverge. Concretely, CLUES computes a score for each private sample $z$ by summing, over checkpoints and over anchor/validation samples $z'$, the learning-rate-weighted inner product of the AdamW update direction for $z$ and for $z'$, and the paper uses this score $S(z)$ as the quality signal. The server sets the global cutoff as the average score of 10 public anchor samples; every client drops private samples below that cutoff before collaborative fine-tuning. Across medical, multilingual, and financial QA, the authors report that this pipeline outperforms local baselines, retains medical knowledge better than the baselines, adapts its threshold to different contamination ratios, and approaches the oracle that trains only on the known-clean subset.","pith_inferences":["Because the score is computed against a public anchor set, CLUES could double as a federated data-quality audit: clients report only scores, and the server can rank domains by how far their data drift from the anchor.","The 10-sample anchor rule is a free parameter the paper does not vary; a direct extension would measure selection stability as anchor size and anchor domain change, and pick per-client thresholds from score quantiles instead of a single global mean.","The gradient-trace score also ranks samples by influence on the anchor loss, so the same machinery could identify poisoned or mislabeled samples for removal before and after training, not just filter before fine-tuning."],"forward_implications":["In the federated setting, selected data reaches more than 96% of the oracle upper bound on GPT-4 scoring across medical and multilingual QA, and more than 91% in model-merging settings.","CLUES beats per-client perplexity, IFD, and DataInf baselines on QA quality metrics while keeping knowledge-benchmark performance in an acceptable range.","The global threshold adapts to contamination level: selection accuracy stays above 99% with 20%, 50%, and 80% low-quality data.","On quality-heterogeneous FiQA, the anchor-derived global threshold beats ratio-based and pre-determined-score selection, with 98.39% F1 versus 95.84% and 79.17%.","Because filtering happens before training and only the threshold broadcast changes on the server, CLUES drops into existing federated averaging and model-merging pipelines."],"supporting_citations":[{"why":"Supplies LoRA, the parameter-efficient fine-tuning method whose low-rank gradients the influence score is computed on.","marker":"[11]"},{"why":"TIES-Merging is the model-merging technique used for the main collaborative fine-tuning results.","marker":"[43]"},{"why":"Task Arithmetic is the alternative merging method compared in the merging-technique ablation.","marker":"[13]"},{"why":"DataInf is a baseline influence-estimation method that CLUES is compared against on data selection and QA performance.","marker":"[22]"},{"why":"IFD is a baseline data-selection method for instruction tuning that CLUES is compared against.","marker":"[25]"},{"why":"Federated learning is the collaborative training paradigm used in one of the two experimental settings.","marker":"[21]"},{"why":"Dataset cartography supplies the training-dynamics view that motivates scoring samples by their effect on model prediction.","marker":"[35]"},{"why":"Influence functions provide the conceptual grounding for treating per-sample gradient effects as data quality.","marker":"[20]"},{"why":"GPT-4 is the judge used to score QA responses, defining the main performance metric.","marker":"[31]"}],"fun_headline_variants":["Training dynamics identify the best private data for LLMs","CLUES: Collaborative high-quality data selection via training dynamics","Rate private data by how it moves LLM training","One global anchor threshold filters private LLM data","Federated LLM fine-tuning with training-dynamics data selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method hinges on the global cutoff, which is the average influence score of only 10 public anchor samples; if those anchors are unrepresentative of a client's domain or contain one noisy sample, the threshold shifts and can silently discard good data or keep bad data.","fun_headline_variants_meta":{"raw":{"variants":["Training dynamics identify the best private data for LLMs","CLUES: Collaborative high-quality data selection via training dynamics","Rate private data by how it moves LLM training","One global anchor threshold filters private LLM data","Federated LLM fine-tuning with training-dynamics data selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3566,"prompt_tokens":968,"completion_tokens":2598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2518}},"tokens_in":584,"tokens_out":2598,"duration_ms":19527,"temperature":1.0,"reasoning_tokens":2518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:52:38.297153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a client whose private domain is far from the public anchor domain and whose low-quality data are adversarial paraphrases, then vary the anchor set from 1 to 100 samples and across domains; record selection precision/recall and downstream QA score. If the global threshold swings enough to drop selection F1 below the best per-client baseline, the claim that one small anchor set defines a reliable unified standard fails.","supporting_citations":[{"cited_title":"TIES-merging: Resolving interference when merging models","cited_arxiv_id":null,"evidence_quote":"TIES-Merging is the model-merging technique used for the main collaborative fine-tuning results."},{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"Task Arithmetic is the alternative merging method compared in the merging-technique ablation."},{"cited_title":"Gpt-4 technical report, 2023","cited_arxiv_id":null,"evidence_quote":"GPT-4 is the judge used to score QA responses, defining the main performance metric."}],"review_version":1}