{"id":"a6dcbf2c-9f72-4c22-ae8d-049683518915","arxiv_id":"2504.12471","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"D2FT dynamically schedules attention subnets to full, forward-only, or skipped operations during fine-tuning, cutting compute 40% and communication 50% with 1-2% top-1 accuracy loss on three vision datasets, plus a LoRA variant.","lead":"A new training scheme, D2FT, skips most attention computations during fine-tuning of a vision transformer, cutting training compute by 40% and communication by 50% with a 1-2% accuracy drop on three image datasets. It decides per image batch which attention heads to update, forward-only, or skip, and shows the same scheduling works for LoRA fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 40% compute-saving claim excludes the Fisher scoring pre-pass (Sec. II-A3); with it included, the true saving is at most 40% − 1/E, so without knowing the number of fine-tuning epochs the headline is unsupported.","rationale":"The reader identified the static proxy as the weakest assumption, and that is a real concern. But the paper's headline is a quantitative efficiency claim, and the one number that would immediately falsify it is the excluded cost of the scoring pre-pass. The paper explicitly describes the pre-pass in Section II-A3 and then defines the computational-cost metric in Section IV-A using only pf/po execution times, so the omission is unambiguous. Without knowing E, the 40% reduction is not merely unverified; for common fine-tuning budgets of 2-5 epochs, it is mathematically inconsistent with the paper's own cost model. This makes it the most load-bearing concern because it attacks the abstract's central number directly. I agree with the reader's general conditional verdict, and their rationale already lists the cost-accounting gap, but I would elevate it over the proxy-correlation issue. A corrected accounting might still show meaningful savings, so the verdict remains conditional rather than reject; however, the paper should not be accepted until this accounting is resolved.","tokens_in":17451,"tokens_out":6253,"duration_ms":69892,"concrete_test":"Ask the authors to report the number of fine-tuning epochs E for each dataset and to rerun Figures 1-2 measuring total wall-clock time and FLOPs for (a) D2FT including the Fisher pre-pass and (b) standard full fine-tuning, over the same E epochs on the same hardware and batch sizes. Compute the true ratio as (E * scheduled_ratio + 1) / E and compare it with the reported 60%. If at the reported E the saving is below 40% (or negative for E < 2.5), the headline claim must be revised; if E is large and the pre-pass is relatively small, the claim may survive with a corrected number.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-A3 states that computing the Fisher contribution scores requires feeding 'all samples for forward and backward propagation without updating weights' before fine-tuning. That is one full forward+backward pass over the training set, equal in cost to one epoch of standard full fine-tuning. The computational-cost metric used in Figures 1-2 and Section IV-A counts only pf/po operations during fine-tuning and never adds this pre-pass. At the headline 60% budget, if fine-tuning runs E epochs, D2FT's true compute relative to standard fine-tuning is 0.6 + 1/E, so the true saving is 0.4 − 1/E. For E=5 the saving is 20%, not 40%; for E=2 D2FT costs 10% more than standard fine-tuning. The paper never states E for any experiment. This is not a proxy-quality objection; it is an omitted term in the exact quantity the abstract claims to reduce. The LoRA experiments inherit the same omission if the Fisher scores are computed for the LoRA matrices.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes D2FT, a distributed fine-tuning framework for transformer-based foundation models. During fine-tuning, each attention-head subnet is assigned one of three operations per micro-batch: full forward and backward, forward-only, or skip, with residual routes preserving signal flow. The assignment is formulated as a multiple-knapsack problem using proxy contribution scores (weight magnitude for backward selection and Fisher information for forward selection), decoupled into per-device bi-level knapsacks solved by dynamic programming. Experiments on CIFAR-10, CIFAR-100, and Stanford Cars with ViT-small compare D2FT against random scheduling, dynamic pruning baselines, an MoE-style baseline, and standard full fine-tuning, and additionally extend the framework to LoRA. The paper reports 40% computational-cost savings and 50% communication-cost savings with small accuracy drops.","tokens_in":17606,"tokens_out":7578,"duration_ms":80197,"significance":"If the headline results hold, D2FT would be a useful contribution to efficient fine-tuning and distributed workload balancing: it converts a training-time sparsity observation into a concrete scheduling algorithm that exactly satisfies per-device budget constraints. The workload-variance result in Table I and the execution-time comparison in Table II are concrete strengths, as is the extension to LoRA with rank-matched and reduced-rank baselines. The algorithmic description, including the dynamic-programming scheduling tables, is detailed enough to be reimplemented, although no code is provided. The main risks are the omission of the Fisher scoring pre-pass from the computational-cost accounting, the post hoc selection of scoring metrics, and the lack of direct validation of the proxy objective; these issues affect the magnitude and generality of the central savings claims.","major_comments":[{"comment":"The reported computational-cost savings omit the cost of the Fisher-scoring pre-pass. Section II-A3 states that computing the scores requires feeding all samples for forward and backward propagation without updating weights, which costs the same as one epoch of standard fine-tuning. The cost metric in Section IV-A counts only the pf and po operations during fine-tuning. If D2FT fine-tuning runs E epochs, its true relative cost at the headline 60% scheduling budget is 0.6 + 1/E, so the true saving is 0.4 − 1/E, not 0.4. For E = 5 the saving is 20%, and for E = 2 D2FT is 10% more expensive than standard fine-tuning. The paper never reports E for any experiment, and the LoRA results inherit the same issue because Fisher scores are also computed over all samples for the LoRA matrices. The abstract and Figures 1–3 should be revised to include this pre-pass in the cost accounting, or the paper should report E and restate the savings for realistic epoch counts.","section":"§II-A3, §IV-A; abstract"},{"comment":"The scoring metrics are selected on the same dataset used for the headline results. Table III reports that the choice of Weight Magnitude as the backward score and Fisher Information as the forward score was made by testing eight combinations on Stanford Cars, and Stanford Cars is one of the three datasets in Figures 1 and 3 and in the abstract's 1–2% accuracy-drop claim. The reported Stanford Cars results are therefore in-sample with respect to metric selection. The paper should select the scores on a held-out validation set or on a different dataset, or should report all eight combinations on all three datasets, so that the headline numbers are not biased by the post hoc choice.","section":"§III-B3, §III-B, Table III"},{"comment":"The optimization objective assumes that per-subnet and per-sample contribution scores are additive and that maximizing their sum maximizes fine-tuning accuracy, but this proxy is not validated. The large accuracy gap between D2FT and random scheduling in Table II (89.4% versus 44.4%) indicates that the scheduling mechanism carries signal, but it does not establish that the Fisher/weight-magnitude proxy is the source of the signal rather than the knapsack structure itself. Section II-B also concedes that the scores are hard to determine in practice. The paper should add a controlled experiment in which the same DP scheduler is run with random contribution scores, and/or a correlation analysis between the proxy scores and per-sample loss reduction, to support the use of this objective as a surrogate for fine-tuning performance.","section":"§II-B, Eqs. (4)–(8), Table II"}],"minor_comments":[{"comment":"The backtracking loop updates the sample index with \"i = 1 − 1\"; this should be \"i = i − 1\", otherwise the loop will not terminate as intended.","section":"Algorithm 2, line 27"},{"comment":"The sentence \"where l and h denotes the index of attention heads and index of blocks\" reverses the definitions: earlier in the same paragraph k = (l, h) with l the block index and h the attention-head index. This should be corrected.","section":"§II-A1"},{"comment":"The first computational-cost setting for LoRA states that 3 micro-batches performing pf and 2 performing po reduce cost to 95% of standard LoRA. Given the paper's own cost model in Section IV-A that po costs 40% of pf, the resulting cost is 3 + 2×0.4 = 3.8 out of 5, i.e., 76%, not 95%. Please clarify whether the cost model differs for LoRA or correct the reported percentage and the corresponding point in Figure 3.","section":"§III-B2"},{"comment":"The experimental settings list datasets, model, hardware, and partition details but omit the number of fine-tuning epochs and optimizer hyperparameters other than the optimizer name. Reporting these is necessary both for reproducibility and for interpreting the cost-savings claims.","section":"§III-A"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is the omitted Fisher pre-pass in the computational-cost accounting; if the savings are recomputed for typical epoch counts, the headline contribution may become substantially smaller, so this should be addressed before the paper is considered further. The post hoc metric selection on Stanford Cars is also a correctness concern for the empirical claims, but it is fixable with a held-out selection procedure or full cross-dataset reporting. I do not see signs of questionable research practice; the issues concern accounting and experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about arXiv:2504.12471. First, there is a genuinely useful idea here: during fine-tuning, schedule each attention subnet into full, forward-only, or skipped operations, and balance the workload across devices with a bi-level knapsack. The three operation modes plus residual shortcuts are a clean way to turn head redundancy into a distributed scheduling problem. Second, the headline '40% compute saving' is not actually supported by the paper's own accounting, because the Fisher-scoring pre-pass—a full forward and backward pass over the entire training set—is excluded from every reported cost figure. As the stress-test notes, the true saving is 0.4 − 1/E, where E is the number of fine-tuning epochs. The paper never states E anywhere. For E=5 you're at 20%, not 40%; for E=2 you're over budget.\n\nWhat's new: the three-mode schedule, the variance-zero assignment, and the LoRA extension are not in the cited literature, and the measured tradeoff curves are new. The paper also does some things well: the ablations on micro-batch size, subnet granularity, and device heterogeneity are thoughtful; the workload variance of zero is a strong result; and D2FT beats random scheduling by a huge margin, which tells you the schedule matters.\n\nSoft spots beyond the pre-pass. Several of the reported cost percentages (95%, 75%, 90%, 64%) don't follow from the paper's own cost model, which sets forward-only at 40% of full cost. For instance, 3 full + 2 forward-only gives 76% of standard compute, not 95%. So the tables are internally inconsistent. The scoring rule (weight magnitude + Fisher) was selected by testing eight combinations on Stanford Cars, the same dataset used for the headline numbers—post-hoc selection on the evaluation set, with no held-out check. There are no error bars, seeds, hyperparameters, or code. The distributed claims come from single-GPU timing measurements, not a real multi-device run. And the motivating observation about attention-head redundancy is established in uncited prior work; the novelty is in the integration.\n\nVerdict: this deserves a serious referee, but with a request for major revision. The framework is interesting, the flaws are fixable, and the paper doesn't try to hide its limitations—it openly says contribution scores are hard to determine in practice. I'd want to see epochs stated, the pre-pass included in the cost model, the cost tables corrected, and ideally code and seeds. With those, the 40% claim might survive; as written, it doesn't.","headline":"Interesting scheduling framework with a solid core idea, but the headline compute savings are inflated because the Fisher pre-pass is excluded from the accounting.","tokens_in":18201,"tokens_out":4969,"would_cite":true,"duration_ms":45102,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"D2FT's central claim: most attention subnets can be skipped or run forward-only during fine-tuning, cutting compute 40% and communication 50% with only 1–2% accuracy loss.","keywords":["distributed fine-tuning","attention sparsity","multiple knapsack","forward-only propagation","LoRA","workload balancing","ViT fine-tuning","training efficiency"],"falsifier":"A direct test: on CIFAR-100, replace D2FT's contribution scores with random values but keep the same knapsack schedule structure; if accuracy stays near 89%, the score proxy is not doing the work. A stronger test: compute leave-one-subnet-out loss changes on a validation batch and compare their ranking with the paper's weight-magnitude and Fisher-information ranking; if the rank correlation is near zero, the central claim fails.","tokens_in":17112,"feed_emoji":"⚡","tokens_out":10413,"duration_ms":98682,"temperature":0.7,"pith_summary":"The paper proposes D2FT (Distributed Dynamic Fine-Tuning), which argues that fine-tuning a pretrained transformer does not need every attention module for every training sample. Instead, each attention-head subnet is assigned one of three operations per sample—full forward and backward, forward-only, or skip—and a distributed scheduler chooses these operations to cut training computation by 40% and communication by 50% while losing only 1–2% top-1 accuracy on CIFAR-10, CIFAR-100, and Stanford Cars. The method scores each subnet using pretrained weight magnitudes and one-shot Fisher information, then casts operation selection as a multiple-knapsack problem solved per device by dynamic programming. If the claim holds, memory-constrained commercial devices could fine-tune foundation models at a fraction of the usual training traffic, with device workloads balanced exactly. The same scheduling shell extends to LoRA, where the 40%/50% savings cost 4–6% accuracy on Stanford Cars.","feed_headline":"Skip 40% of fine-tuning compute, lose 1–2% accuracy","feed_subtitle":"A knapsack scheduler tells each attention subnet whether to train, pass forward only, or skip—and cuts device workload imbalance to zero.","key_machinery":"The load-bearing object is the operation-selection table $T_{opt}$, produced by two nested knapsack solves: an outer solve picks which samples get full forward and backward updates by maximizing summed weight-magnitude scores under each device's full-operation cost, and an inner solve picks which samples get forward-only passes by maximizing summed Fisher-information scores under the forward-only cost. Residual shortcut routes carry activations and gradients around skipped subnets so the rest of the model still trains. This decoupling converts the NP-hard multiple-knapsack orchestration into per-device dynamic programs that jointly balance workload and maximize contribution.","core_discovery":"D2FT partitions a transformer into subnets, each one attention head plus a slice of the feed-forward network, and gives each subnet a per-sample instruction from {full, forward-only, skip}. The instructions come from a bi-level heuristic that first fills each device's full-operation budget using weight-magnitude backward contribution scores, then fills the forward-only budget using empirical Fisher information scores, merging the two into a single schedule table. The paper's central empirical discovery is that this scheduled sparsity costs almost nothing: relative to standard fine-tuning, D2FT reports 40% lower training computation and 50% lower communication with 1–2% top-1 accuracy drops on CIFAR-10, CIFAR-100, and Stanford Cars; with LoRA, the same savings cost 4–6% on Stanford Cars, while beating small-rank LoRA at equal compute. The schedule also drives device workload variance to zero, so no subnet acts as a straggler.","pith_inferences":["Beyond the paper: the schedule is computed once from static pretrained scores, so refreshing the contribution scores during training could improve accuracy if subnet importance shifts; the paper does not test this.","Beyond the paper: since the method's edge over random scheduling rests entirely on the contribution-score proxy, the knapsack shell could be reused with other cheap saliency signals or a learned predictor of per-sample loss reduction as a direct test of where the gain comes from.","Beyond the paper: if attention redundancy in language models is as high as in vision transformers, the same 40%/50% savings might transfer to LLM fine-tuning; that is an extrapolation, not a claim in the paper.","Beyond the paper: because inference still uses all parameters, D2FT is purely a training-time distribution strategy, and combining it with post-training pruning could compound savings, but that combination is unexplored here."],"forward_implications":["A device with a tight compute budget can fine-tune ViT-small at 60% of full compute and keep about 89.4% top-1 accuracy on CIFAR-100, roughly 35 points above random scheduling and well above both dynamic pruning baselines.","Because skipped and forward-only subnets keep residual routes, the schedule can change per micro-batch without breaking gradient flow, so the framework works across different batch-to-micro-batch splits.","D2FT reduces workload variance to zero across 74 subnets, which corresponds to shorter per-batch execution time than random scheduling and dynamic pruning.","Applied to LoRA, the same scheduling saves 40% compute or 50% communication on Stanford Cars while losing 4–6% top-1 accuracy relative to full-rank standard LoRA, and it outperforms simply shrinking LoRA rank at equal cost.","Heterogeneous device pools, with mixed memory sizes or compute speeds, keep the same accuracy as homogeneous pools because the knapsack capacities are set per device."],"supporting_citations":[{"why":"Defines the multi-head attention transformer blocks that D2FT partitions into per-head subnets.","marker":"[16]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning method D2FT extends by co-scheduling low-rank matrices with frozen attention heads.","marker":"[18]"},{"why":"Dynamic pruning baseline with feedback; its sparse-training idea motivates the Forward-Skip operation.","marker":"[21]"},{"why":"Supplies the empirical Fisher information metric used as the forward contribution score in the knapsack.","marker":"[23]"},{"why":"Provides the knapsack problem formulation that the operation-selection optimization is cast as.","marker":"[24]"},{"why":"Establishes multi-knapsack NP-hardness, motivating the paper's bi-level decoupled heuristic.","marker":"[25]"},{"why":"Supplies the pretrained ViT-small model used for every reported experiment.","marker":"[28]"},{"why":"Gshard mixture-of-experts baseline whose workload-balancing behaviour is compared against D2FT's.","marker":"[30]"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The schedule's value over random selection rests on the claim that a subnet's weight size and a one-time gradient-based importance estimate predict how much that subnet will actually help on each training sample; if those proxies do not track real loss reduction, the knapsack optimizer is just maximizing a number that does not matter.","fun_headline_variants_meta":{"error":"DeepSeek 429: {\"error\":{\"message\":\"Too many requests. Your current concurrency is 153, which exceeds your concurrency limit of 117 based on your remaining balance. Please top up your balance to restore your concurrency.\",\"type\":\"rate_limit_error\",\"param\":null,\"code\":\"invalid_request_error\"}}"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:33:29.866361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: on CIFAR-100, replace D2FT's contribution scores with random values but keep the same knapsack schedule structure; if accuracy stays near 89%, the score proxy is not doing the work. A stronger test: compute leave-one-subnet-out loss changes on a validation batch and compare their ranking with the paper's weight-magnitude and Fisher-information ranking; if the rank correlation is near zero, the central claim fails.","supporting_citations":[{"cited_title":"Resource- efficient transformer pruning for finetuning of large models,","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical Fisher information metric used as the forward contribution score in the knapsack."},{"cited_title":"Martello and P","cited_arxiv_id":null,"evidence_quote":"Provides the knapsack problem formulation that the operation-selection optimization is cast as."},{"cited_title":"A class of generalized greedy algorithms for the multi-knapsack problem,","cited_arxiv_id":null,"evidence_quote":"Establishes multi-knapsack NP-hardness, motivating the paper's bi-level decoupled heuristic."}],"review_version":1}