{"id":"2c3e2b7a-b0da-450d-ad88-a96ea9217cdd","arxiv_id":"2412.00420","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Targeted data selection via whitened feature distance and optimal transport minimization consistently outperforms influence-based greedy heuristics across vision and language tasks.","lead":"TAROT selects training data for a target task by minimizing optimal transport distance between gradient-derived features of selected and target data, using whitened distances to avoid dominant-feature bias. It reports consistent gains over prior selection methods across semantic segmentation, motion prediction, and instruction tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is confounded: TAROT changes both the distance metric (WFD) and the selection rule (OT) relative to baselines, so the reported gains cannot be attributed to OT-based selection; additive greedy on the same WFD features is never tested.","rationale":"The reader's weakest assumption focuses on whether whitened gradient features faithfully reflect sample usefulness for the target domain. That is a legitimate concern, but the more directly load-bearing issue is that the paper never isolates the contribution of the OT selection rule. The central claim is about OT-based selection beating additive influence heuristics, yet TAROT differs from LESS/DsDm in both the feature embedding and the selection objective. The reported cross-architecture transfer and consistent wins provide real empirical support that TAROT works, so I am not arguing for rejection. However, the mechanism claimed in the title and abstract remains untested: an additive greedy rule applied to the same WFD features might recover all of the gains, in which case the optimal transport framing is not the operative ingredient. This is a concrete experimental-design gap, not a matter of external consensus, and it can be closed by the fixed-feature control described above. The secondary inconsistency between Eq. (14) and Algorithm 2's stopping criterion is a separate correctness risk for the OTM results, but the primary reason for keeping the verdict at CONDITIONAL is the missing control ablation that would attribute the gains to OT rather than to WFD.","tokens_in":16638,"tokens_out":8086,"duration_ms":92552,"concrete_test":"Hold the feature representation fixed and compare selection rules. In the instruction-tuning and motion-prediction setups, compute WFD scores (Eq. 10) for the candidate pool, then select subsets of matching sizes by (a) TAROT's Algorithm 1/2 and (b) an additive LESS-style rule that ranks candidates by average WFD to the target and takes the top S. Train and evaluate the same downstream models as in Table 1 and Figure 7; if (b) matches or beats (a), the OT selection objective is not responsible for the reported gains. Separately, instrument Algorithm 2 with the Eq. (14) stopping criterion versus the printed line 11 to check whether OTM ratios and downstream performance change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that OT-based subset selection outperforms influence-based additive greedy heuristics (LESS, DsDm). But the experiments change two variables at once: (i) the features/distance are replaced by whitened gradient features (Eqs. 6-10), and (ii) the selection rule is replaced by a greedy kNN/OT-potential procedure (Algorithms 1-2). The LESS and DsDm baselines use unwhitened projected-gradient influence with additive top-k selection. Consequently, any observed improvement could be caused entirely by the improved WFD features, with the OT formulation contributing little or nothing. No experiment isolates the selection rule by running an additive greedy heuristic on the same WFD features. The conceptual distinction is further blurred because TAROT's fixed-size selection is itself a greedy nearest-neighbor heuristic rather than an actual OT minimizer. There is also an internal inconsistency in the OTM stopping criterion: Section 3.3, Eq. (14) states that OT distance is evaluated against Dt \\ D_fold, but Algorithm 2, line 11 uses D_fold as the target measure. If the implemented code follows the algorithm rather than the main text, the reported OTM selection ratios and downstream gains may reflect a different protocol than the one described.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAROT, a targeted data selection framework that uses optimal transport with whitened gradient features. The method first computes per-sample gradient features from a pretrained model, whitens and normalizes them (WFD), then selects subsets either by a fixed-size greedy nearest-neighbor rule or by an OT-distance-minimization rule with k-fold early stopping, and optionally reweights selected samples via OT potentials. Experiments cover semantic segmentation (GTA5 to Cityscapes), motion prediction (multi-source to nuScenes), and instruction tuning (FLAN/CoT/Dolly/OpenAssistant to MMLU/BBH), reporting gains over LESS, DsDm, TSDS, random selection, and full-data training, as well as cross-architecture transfer.","tokens_in":16780,"tokens_out":4703,"duration_ms":45875,"significance":"If the central claim holds, TAROT is a practically valuable framework: it consistently improves targeted data selection across vision and language tasks, can select very small subsets (e.g., 0.13% of the candidate pool for BBH transfer), and exhibits transfer across architectures (Autobots-to-Wayformer and LLaMA-to-Qwen). The paper's strengths include extensive multi-task evaluation, a released codebase, ablations isolating the whitening and normalization components (Table 2), cross-architecture transfer experiments, and computational complexity analysis. However, the load-bearing attribution of the gains to OT-based selection over influence-based heuristics is not fully isolated, and the OTM protocol is internally inconsistent between the main text and the algorithm listing. These issues are fixable with additional experiments and clarification, but they are central to the paper's contribution.","major_comments":[{"comment":"The central claim that OT-based selection outperforms influence-based additive greedy heuristics is confounded: TAROT changes both the feature/distance (whitened gradient features, Eqs. 6–10) and the selection rule (greedy kNN plus OT potential, Algorithms 1–2), while the LESS and DsDm baselines use unwhitened projected-gradient influence with additive top-k selection. No experiment runs an additive greedy heuristic on the same WFD features or runs TAROT's selection rule on unwhitened features, so the reported gains could be entirely due to WFD rather than to the OT formulation. Please add ablations that isolate the selection rule, e.g., top-k additive selection with WFD cost, or OT-potential selection with the original projected-gradient distance, at least on one segmentation and one instruction-tuning setting.","section":"§4.2–§4.4 vs. §3.2–§3.3"},{"comment":"There is a direct inconsistency between the stopping-criterion description and the provided algorithm. Eq. (14) states that the OT distance is evaluated against Dt \\ D_fold (the held-out portion), while Algorithm 2, line 11 compares dOT(D_s_fold ∪ D_k, D_t_fold), i.e., against the selection fold D_fold. Additionally, the text says '1/k of Dt is used for selection, while the OT distance is evaluated against the remaining (k−1)/k data,' but Algorithm 2 sets D_tar ← Dt \\ D_fold and uses D_tar for the kNN candidate search, while line 11 evaluates against D_fold. These two readings are opposite. Please align the text and pseudocode and state which protocol was actually implemented, since the reported OTM selection ratios and downstream gains depend on this choice.","section":"§3.3, Eq. (14) and Algorithm 2"},{"comment":"The comparison is not controlled for data weighting. In the segmentation experiments (Section 4.2), TAROT applies OT-potential weights with R = N + M while baselines use uniform repetition; in the instruction-tuning experiments (Section 4.4), TAROT-OTM is weighted with R = 0.5%N. Thus the gains of TAROT over baselines in those sections may stem from the weighting scheme rather than from the selection rule. The motion-prediction experiments (Section 4.3) do not apply weighting and still show gains, which partially mitigates this concern, but the paper should either apply the same repetition/weighting protocol to all methods or report the contribution of weighting explicitly.","section":"§3.3–§3.4 and §4.2/§4.4"},{"comment":"The fixed-size selection scheme is described as minimizing the OT distance, but Algorithm 1 is in fact a greedy k-nearest-neighbor procedure: it adds the k-th nearest candidates for each target point, and only in the final partial iteration does it use the OT dual potential (line 9). This is a heuristic approximation, not an empirical OT minimizer, and the paper does not characterize how close the resulting subset is to the OT-optimal subset. Given that the paper's title and framing emphasize optimal transport, please either provide a theoretical or empirical justification for this kNN greedy approximation, or soften the claim that the selected subset minimizes the OT distance.","section":"§3.3, Algorithm 1"}],"minor_comments":[{"comment":"Eq. (2) uses N as the summation limit for checkpoint indexing, but N is also used for the candidate dataset size earlier in the paper; Eq. (4) correctly uses T for the number of checkpoints. Please make the notation consistent.","section":"§2.1, Eq. (2)"},{"comment":"The input line states 'Desired subset size S = N', but the algorithm is intended for S ≤ N. This is presumably a typo; please correct it.","section":"Algorithm 1, line 3"},{"comment":"The whitening procedure does not specify over which data the centered gradients and covariance matrix are computed: the candidate pool Dc, the target set Dt, or their union. This affects the interpretation of the whitened distances and should be stated explicitly.","section":"§3.2, Eqs. (6)–(8)"},{"comment":"The text says TAROT 'consistently outperforms all baseline methods,' but Table 1 shows TAROT-5% is slightly below LESS-5% on Qwen-2.5-7B MMLU (74.1 vs. 74.3) and TAROT-OTM matches LESS-5% on LLaMA-3.1-8B MMLU (65.7 vs. 65.7). Please qualify the 'consistently outperforms' claim accordingly.","section":"Table 1 and §4.4"},{"comment":"The experimental sections do not report standard deviations or multiple seeds. Given the small reported improvements in some comparisons (e.g., less than 1 point on MMLU), adding variance estimates or at least stating the number of runs would strengthen the claims.","section":"§4.2–§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a plausible and potentially useful contribution, and the experimental breadth is impressive. The main concern is attribution: because the method changes both the feature distance and the selection rule relative to baselines, the paper's headline claim about OT-based selection is not yet fully supported. The OTM protocol inconsistency between Eq. (14) and Algorithm 2 is also a correctness issue that must be resolved. These are fixable with additional ablations and clarification, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this: TAROT is a solid, well-tested contribution to targeted data selection. The new bits are the whitened gradient distance (WFD) and the greedy OT-flavored selection with a data-driven stopping rule. Experiments span segmentation, motion prediction, and instruction tuning, and the cross-architecture transfer results are genuinely encouraging. The LDS gain over TRAK is a concrete win, and the ablations show whitening and normalization matter.\n\nThe soft spots are fixable but need attention. The central attribution problem: the paper changes both the distance metric and the selection rule at once. It never runs an additive-greedy baseline on the same WFD features, so the marginal value of the OT machinery is not demonstrated. That's an experimental gap, not a deep flaw.\n\nNext, the OTM stopping criterion is inconsistent: Eq. (14) evaluates the OT distance on Dt \\ D_fold, while Algorithm 2 line 11 checks it on D_fold. These are different protocols. If the code follows the algorithm, the reported OTM results come from a different procedure than the text describes. This needs fixing.\n\nSmaller issues: no error bars or multiple seeds; the \"consistently outperforms\" claim is too strong given a few ties/losses (e.g., MMLU on Qwen); and the greedy selection is a heuristic, not a real OT minimizer, so the OT theory is used loosely. The code link is stated; I did not verify it.\n\nWho it's for: data selection researchers and practitioners who want to reduce training data for a target domain. It deserves serious peer review. My recommendation: send it out, and ask for (1) the additive-greedy-on-WFD control, (2) a consistent OTM protocol, (3) variance estimates on the main tables.","headline":"Solid empirical paper with a genuine new distance metric, but the OT selection claim is not isolated and the OTM protocol has an internal inconsistency.","tokens_in":17439,"tokens_out":3999,"would_cite":true,"duration_ms":38894,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TAROT claims targeted data selection should minimize optimal-transport distance in whitened gradient space, outperforming influence heuristics and full-data training with far less data.","keywords":["targeted data selection","optimal transport","whitened feature distance","data influence","gradient features","instruction tuning","motion prediction","semantic segmentation"],"falsifier":"Construct a target domain that is a simple unimodal shift of one candidate cluster, with a second candidate cluster of near-duplicate, low-gradient samples placed at large whitened distance: additive influence methods should select the shifted cluster easily, and if TAROT's OT minimization over nearest neighbors still selects fewer useful samples or stops too early (higher OT distance to held-out data than a random subset of the same size), the monotonicity assumption is falsified.","tokens_in":16305,"feed_emoji":"🎯","tokens_out":4085,"duration_ms":38668,"temperature":0.7,"pith_summary":"TAROT argues that targeted data selection should be posed as distribution matching: pick the subset of a candidate pool whose optimal-transport distance to the target data, measured in a whitened gradient-feature space, is smallest. It identifies two failure modes in influence-based heuristics — dominant feature components swamping high-dimensional distance estimates, and the assumption that per-sample influences add linearly — and replaces them with a whitened-feature distance and a greedy OT-minimizing selection. Across semantic segmentation, motion prediction, and instruction tuning, subsets chosen this way match or beat full-data training while using a fraction of the data; in the LLM setting OTM selects as little as 0.13% of the candidate pool and improves BBH from 63.8 to 68.9. The paper also introduces an early-stopping rule based on when OT distance starts to increase, which estimates the optimal selection ratio without a validation sweep.","feed_headline":"0.13% of data beats full training on BBH","feed_subtitle":"TAROT selects subsets by matching target distributions in whitened gradient space — less data, better scores.","key_machinery":"Whitened Feature Distance (WFD): $\\phi(z)=\\sum_i \\nabla L(z;\\theta_i)$ summed over checkpoints, projected, decorrelated by Cholesky (or ZCA) whitening, then L2-normalized, giving pair distance $d^w_{\\mathcal{Z}}(z,z')=\\|\\hat\\phi(z)-\\hat\\phi(z')\\|_2$. This feeds the OT cost in $d_{OT}(D_s,D_t)=\\min_{\\pi}\\int d^w_{\\mathcal{Z}}\\,d\\pi$, solved by Sinkhorn; the greedy selection uses nearest-neighbor candidates and, for OTM, stops when $d_{OT}$ to held-out folds rises. The machinery does two jobs: it repairs the metric so dominant gradient directions do not dominate selection, and it converts selection into a distribution-alignment problem with a built-in stopping criterion.","core_discovery":"The central claim is that the whitened gradient-feature distance (WFD) is a more reliable measure of a sample's usefulness for a target distribution than raw projected-gradient influence, and that minimizing the optimal-transport distance between the selected subset and the target, rather than greedily accumulating high-influence points, yields subsets that generalize better on complex, multimodal target data. TAROT embeds each example by summing loss gradients over checkpoints of a model trained on the candidate pool, projects, whitens, and normalizes to the unit sphere, then uses the resulting Euclidean distances as the OT cost. Its greedy selection iterates nearest neighbors of target points and, in the OTM variant, stops when the OT distance to held-out target folds increases, producing both a subset and an estimated selection ratio. The paper reports consistent gains over LESS, DsDm, TSDS, and full-data training across three task families, including cross-architecture transfer.","pith_inferences":["If whitening is the key repair, then applying the same normalization to other embedding spaces, such as feature activations, may extend the method to settings where gradients are unavailable or prohibitively expensive.","The early-stopping rule suggests a data-efficiency law: the optimal subset size is where marginal transport cost turns upward, a quantity that could be used to compare candidate data sources before any training.","Because OTM selects nearest neighbors in whitened gradient space, it may concentrate on dominant target modes and under-select rare edge cases; this testable risk could be checked by measuring performance on out-of-distribution target splits.","Combining WFD with explicit diversity regularization should recover the generalization that the authors themselves flag as a limitation, offering a concrete extension of the current objective."],"forward_implications":["Training on OT-selected subsets can match or surpass full-dataset training while using a small fraction of data; for example, BBH rises to 68.9 with 0.13% of the candidate pool versus 63.8 with the full dataset.","OTM's early-stopping rule estimates an optimal selection ratio automatically, avoiding the budget sweeps that influence-based methods require.","WFD outperforms TRAK on the LDS influence-estimation score across CIFAR-10 and nuScenes for every ensemble size tested, indicating a stronger correlation with model behavior.","Selection transfers across architectures and model scales (AutoBots to Wayformer, LLaMA to Qwen) and does not require the subtask labels that LESS relies on."],"supporting_citations":[{"why":"Supplies the trajectory-influence formulation that TAROT extends by summing loss gradients over checkpoints.","marker":"Pruthi et al., 2020"},{"why":"Provides the random projection technique, the TRAK baseline for influence estimation, and the LDS evaluation protocol.","marker":"Park et al., 2023"},{"why":"Defines the LESS baseline, the instruction-tuning experimental setup, and the candidate/target datasets used for LLM evaluation.","marker":"Xia et al."},{"why":"Provides the DsDm baseline, a model-aware datamodel selection method that TAROT compares against.","marker":"Engstrom et al., 2024"},{"why":"Gives the Kantorovich formulation of optimal transport that TAROT uses to define dataset-level distance.","marker":"Kantorovitch, 1958"},{"why":"Supplies the Sinkhorn algorithm that makes OT distance computation tractable at the scale used in the experiments.","marker":"Cuturi, 2013"},{"why":"Supports using OT potentials to rank the benefit of adding individual candidate points during greedy selection.","marker":"Just et al., 2023"},{"why":"Documents the failure of additive influence assumptions even in linear regression, motivating the distribution-matching approach.","marker":"Hu et al., 2024"},{"why":"Establishes the practice of computing OT distances between datasets using pairwise sample distances.","marker":"Alvarez-Melis & Fusi, 2020"}],"fun_headline_variants":["TAROT picks 0.13% of data and beats full training","Optimal transport finds the data that matters most","Whitened feature distance improves data selection","TAROT uses optimal transport for smarter data subsets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pretrained model's whitened gradient features are assumed to capture, monotonically, how much a training example will help the target task, so that smaller OT distance in this space means better downstream performance.","fun_headline_variants_meta":{"raw":{"variants":["TAROT picks 0.13% of data and beats full training","Optimal transport finds the data that matters most","Whitened feature distance improves data selection","TAROT uses optimal transport for smarter data subsets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2341,"prompt_tokens":957,"completion_tokens":1384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1320}},"tokens_in":573,"tokens_out":1384,"duration_ms":10026,"temperature":1.0,"reasoning_tokens":1320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:23:00.142944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a target domain that is a simple unimodal shift of one candidate cluster, with a second candidate cluster of near-duplicate, low-gradient samples placed at large whitened distance: additive influence methods should select the shifted cluster easily, and if TAROT's OT minimization over nearest neighbors still selects fewer useful samples or stops too early (higher OT distance to held-out data than a random subset of the same size), the monotonicity assumption is falsified.","supporting_citations":[],"review_version":1}