{"id":"ff5a2116-40f0-4ef4-b4b3-dc65e5f8132b","arxiv_id":"2505.05130","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CacheFL improves CLIP classification in federated settings by aggregating client updates to a lightweight cache of visual features that is initialized from DALL-E-generated class-balanced images.","lead":"This paper proposes CacheFL, a federated learning method that fine-tunes only a small cache model attached to a frozen CLIP vision-language model, instead of fine-tuning the full model. It reports higher accuracy than several baselines on 11 image classification datasets while transmitting far fewer parameters per round.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 does not apply to the trained objective: Assumption 2 asserts µ-strong convexity for a loss that is non-convex in the cache weights W1 because of the RBF term inside cross-entropy.","rationale":"The reader's weakest_assumption identifies the same load-bearing gap that I find: Section IV-B Assumption 2 asserts strong convexity of every local objective, but the loss in Eqs. (6)-(8) is not proved, and is generally not, µ-strongly convex in W1 because the RBF/exponential term enters the logits before softmax cross-entropy. Since Lemma 1 and Theorem 1 both depend on Eq. (25), the stated O(1/t) rate does not cover the algorithm actually run. I also considered the privacy claim (only a single gradient-inversion demo, no formal guarantee), missing error bars, and the mismatch between the theoretical decreasing learning rate and the fixed experimental LR, but these are either already in the reader's rationale or secondary to the theorem's applicability. The empirical results may still be useful and the conditional verdict is appropriate; the paper needs either a nonconvex convergence analysis or a clear statement that no convergence guarantee is claimed for the implemented objective.","tokens_in":20324,"tokens_out":6486,"duration_ms":70926,"concrete_test":"On a fixed logged local batch (e.g., ImageNet, iid, client 0), freeze f_vision, W_text, and W_2, and compute the Hessian of the per-sample cross-entropy loss in Eqs. (7)-(8) with respect to the trainable cache weights W1 at the DALL-E initialization and at round-1, round-5, and round-20 checkpoints. If any eigenvalue of this Hessian is negative, the local objective is not µ-strongly convex and Assumption 2 is false, so Theorem 1 cannot be applied to the executed algorithm; as a cross-check, re-derive Lemma 1 without invoking Eq. (25) and confirm that the (1 - η_t µ) contraction term no longer follows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The convergence claim in Theorem 1 (Section IV-C, Eq. (60)) is proved under Assumptions 1-4, and Lemma 1 specifically uses Assumption 2 through Eq. (25) to obtain the contraction term (1 - η_t µ)||c_k^t - c*||^2. However, the objective actually minimized on clients (Algorithm 1 lines 17-18, Eqs. (6)-(8)) is cross-entropy over logits = f_vision W_text^T + α exp(-β(1 - f_vision W1)) W_2^T. The exp/RBF term makes logits a nonlinear function of W1, and the paper provides no proof that the resulting empirical risk is µ-strongly convex in W1; in fact it is generally non-convex. Thus the theorem bounds a different, assumed-strongly-convex objective, not the CacheFL training procedure that produced the reported results. The same gap also means the theoretical learning-rate schedule η_t = 2/(µ(t+γ)) is not connected to the constant LR=0.001 used in the experiments. This does not invalidate the empirical accuracy comparisons, but it does mean the advertised O(1/t) convergence guarantee is unsupported for the implemented method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CacheFL, a federated fine-tuning method for CLIP in which only a lightweight Tip-Adapter-style cache model is trained and communicated. The cache model is initialized from class-balanced synthetic images generated by DALL·E; clients update the visual-feature cache by SGD on a cross-entropy loss over fused CLIP and cache logits, and the server aggregates the cache updates with FedAvg. The paper reports accuracy improvements over FedAvg, FedProx, CLIP, AdapterFL, and PromptFL on 11 datasets under IID, Dirichlet, and extreme non-IID splits, reduced communication and computation costs, qualitative resistance to gradient inversion, and a convergence theorem claiming O(1/t) convergence.","tokens_in":20551,"tokens_out":4859,"duration_ms":49408,"significance":"If the empirical results are reproducible, the underlying idea is practically valuable: it is simple, communication-light, and improves on zero-shot CLIP in federated settings. The paper includes broad dataset coverage, ablations of synthetic initialization and of the number of synthetic images, and a resource-consumption comparison. However, the theoretical section overstates the guarantees: Theorem 1 relies on strong convexity that is not satisfied by the objective actually trained on clients. The empirical claims also need stronger support, in the form of error bars, validation-based hyperparameter selection, and a quantitative privacy evaluation, before they can be taken at face value.","major_comments":[{"comment":"The convergence proof uses Assumption 2 in Eq. (25) to obtain the contraction term (1−η_t μ)||c_k^t − c*||^2, and Theorem 1 inherits this strong-convexity requirement. However, the objective actually minimized on clients (Algorithm 1 lines 17–18; Eqs. (6)–(8)) has logits = f_vision W_text^T + α exp(−β(1−f_vision W_1)) W_2^T, and cross-entropy of these logits is generally non-convex in W_1 because of the exponential/RBF term. The manuscript asserts strong convexity without proof, saying only that Assumptions 1 and 2 are \"canonical assumptions\" for linear regression with softmax classification. Consequently, the O(1/t) bound in Eq. (60) is not established for CacheFL as implemented. The authors should either prove the required property, replace Theorem 1 with a nonconvex FL convergence analysis (e.g., convergence to stationarity under bounded-gradient assumptions), or explicitly restrict the theorem to a simplified convex surrogate and state that the experiments do not satisfy its assumptions.","section":"§IV-B, Assumption 2 and §IV-C, Theorem 1"},{"comment":"The hyperparameters α and β are selected as the values that maximize accuracy (α=0.5, β=1.0) and these values are then used in the headline comparisons of Table I. Table III does not state which dataset is used or whether a validation split was held out, so the tuning appears to be performed on the evaluation set. This can inflate the reported gains, especially because the sensitivity to α is sharp (accuracy drops from 64.1 to 51.4 as α goes from 0.5 to 2.0). Please use a validation split for hyperparameter selection, report which dataset Table III refers to, and ideally show sensitivity across all 11 datasets.","section":"§V-C, Table III"},{"comment":"All accuracy numbers in Table I are reported without error bars, standard deviations, or the number of random seeds. In the small-data regime considered (16 images per class) with random non-IID partitioning, a single run per cell is insufficient to support claims such as \"consistently achieves superior performance.\" Report means and standard deviations over at least three seeds and random partitions, at minimum for the averaged table and for the datasets with the largest variance.","section":"§V-A and §V-B, Table I"},{"comment":"The privacy evaluation is qualitative and compares a full-model FedAvg baseline against CacheFL's cache-model updates. Gradient inversion success depends strongly on model dimension and structure, so this comparison does not isolate the privacy protection of CacheFL. Provide quantitative reconstruction metrics (e.g., attack success rate, mean squared error, or learned perceptual similarity) and compare against baselines with comparable parameter counts, such as PromptFL or AdapterFL. Also clarify what the server observes about local data through the gradients of W_1 with respect to f_vision, since these gradients may carry information about local class statistics.","section":"§V-B, Privacy Preservation and Figure 7"}],"minor_comments":[{"comment":"Assumption 2 contains a typo: strong convexity should have (μ/2)||v−w||^2, not (L/2)||v−w||^2, since the proof and the rest of the paper use μ.","section":"§IV-B, Assumption 2"},{"comment":"The dataset name is misspelled as \"StandfordCars\" in Table I(j) and in Figure 8; the correct spelling is \"StanfordCars.\"","section":"Table I and Figure 8"},{"comment":"The labels in Figure 6 are garbled: \"Ca cheFL\" appears in panel (a), and the number \"12712932\" looks like an artifact. Please clean the figure labels and ensure the caption matches the reported values.","section":"Figure 6"},{"comment":"Several panels in Figure 5 contain stray numbers near the y-axis (e.g., \"22\", \"28\", \"16\", \"38\", \"40\") that appear to be residual axis labels or editing artifacts; please regenerate the figure.","section":"Figure 5"},{"comment":"The communication cost for CacheFL in Figure 6a is reported as 102,400 parameters per round, but the cache matrix W_1 has dimension C×N×K; for ImageNet (N=1000, K=16) this would be substantially larger than 102,400. Please specify which dataset and which parameter count are used so that the resource comparison is reproducible.","section":"§V-A, Communication Overhead and Figure 6a"},{"comment":"The ablation \"w.o. Training\" is compared with PromptFL in the text, but the table shows only PromptFL and CacheFL rows; it would be clearer to include the full comparison or to state explicitly that PromptFL values are taken from Table I.","section":"§V-C, Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for a distributed-systems or federated-learning venue and does not appear to be a duplicate of prior work. The main risk is the mismatch between the theoretical convergence claim and the objective actually trained, which is serious but fixable either by correcting the theory or by clearly labeling the theorem as applying only to a simplified surrogate. The empirical method itself is plausible and deserves a fair evaluation after the experimental rigor issues (error bars, validation-based hyperparameters, quantitative privacy metrics) are addressed. I would also encourage the editor to request the authors release code, since no reproducibility artifacts are mentioned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper is worth reading for its empirical recipe, not for its theory. The authors take the Tip-Adapter cache model, initialize it with DALL-E generated class-balanced images, and fine-tune only the cache in a federated loop. That combination is new, and the results are genuinely suggestive—average gains of +7 to +9 points over PromptFL across 11 datasets, with communication reduced to the size of the cache matrix. The ablations show both the synthetic initialization and the federated training contribute. Those are real, useful contributions for anyone working on parameter-efficient federated fine-tuning of VLMs.\n\nThe weak spot is the convergence analysis, and it is load-bearing. Theorem 1 claims O(1/t) convergence under Assumptions 1-4, and the proof of Lemma 1 uses Assumption 2 (µ-strong convexity of every local objective) to get the contraction term in Eq. (25). But the actual loss, Eq. (7) with Eq. (6), is cross-entropy over logits that include an RBF term exp(-β(1 - f_vision W1)). That makes the objective non-convex in W1; strong convexity is asserted, not proved, and it is generally false. So the theorem bounds a different, assumed-strongly-convex objective than the one Algorithm 1 actually optimizes. The theoretical learning-rate schedule η_t = 2/(µ(t+γ)) also never appears in the experiments, which use a constant 0.001. The empirical accuracy comparisons may well hold, but the advertised convergence guarantee is unsupported for the implemented method.\n\nOther issues are more standard. Table I reports no error bars or seeds, and the α/β values in Table III are tuned on the same evaluation metric and then used in the headline results—overfitting, not circularity, but still too convenient. The privacy claim rests on a single gradient-inversion picture with no attack code or formal analysis, so it should be described as preliminary. The resource-cost numbers are simple and clear.\n\nWho is this for? Researchers in federated learning for vision-language models who want a lightweight adapter baseline. It deserved a serious referee, but not acceptance as-is. I would ask the authors to either prove convergence for the actual non-convex objective (hard) or honestly remove the theorem and keep the paper as an empirical contribution, which is strong enough on its own. With that fix, the paper would be publishable.","headline":"A useful empirical recipe for federated cache-model fine-tuning of CLIP, undermined by a convergence theorem that does not apply to the trained (non-convex) objective.","tokens_in":21109,"tokens_out":1753,"would_cite":true,"duration_ms":19298,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that federated fine-tuning of CLIP can be reduced to updating a lightweight cache matrix, and that this cache-only approach beats prompt- and adapter-based federated baselines on 11 image-classification datasets.","keywords":["federated learning","cache model","vision-language models","CLIP","parameter-efficient fine-tuning","non-IID data","privacy preservation","synthetic data initialization"],"falsifier":"Compute the smallest eigenvalue of the Hessian of the local cross-entropy loss with respect to $W_1$ at several points along a training run; if it is ever zero or negative, Assumption 2 fails and Theorem 1 does not apply to CacheFL as run.","tokens_in":1790,"feed_emoji":"🖼️","tokens_out":3751,"duration_ms":102038,"temperature":0.7,"pith_summary":"CacheFL is a federated fine-tuning method for vision-language models like CLIP that trains only a lightweight cache model instead of the full backbone. The cache stores visual features $W_1$ and one-hot labels $W_2$ built from a server-generated class-balanced synthetic dataset; clients update only $W_1$ on private data and the server aggregates updates with federated averaging. The paper argues this design preserves privacy, cuts communication and computation to a small fraction of full-model fine-tuning, and mitigates non-IID data through the class-balanced initialization. Experiments on 11 datasets report average gains of roughly 7 to 10 points over PromptFL, and a convergence theorem claims an $O(1/t)$ rate under standard assumptions.","feed_headline":"Cache-only federated tuning beats prompt-based CLIP on 11 datasets","feed_subtitle":"Training and sharing only the cache matrix cuts communication to about 100k parameters while raising non-IID accuracy.","key_machinery":"The central object is the cache model, composed of $W_1$, a $C \\times NK$ matrix of CLIP image features from $NK$ synthetic images, and $W_2$, an $N \\times NK$ matrix of one-hot labels for those images. Predictions combine the frozen CLIP logits $f_{\\text{vision}} W_{\\text{text}}^T$ with the cache logits $\\exp(-\\beta(1 - f_{\\text{vision}} W_1)) W_2^T$, where the exponential term converts feature distance into similarity weights, so the cache acts as a retrievable class-balanced memory. During federated training only $W_1$ is updated locally by SGD on cross-entropy loss, with $W_2$ frozen, and the server aggregates client updates by federated averaging. Theorem 1 then bounds the objective gap by $O(1/t)$ under smoothness, strong convexity, and bounded-gradient assumptions.","core_discovery":"The central claim is that a frozen CLIP backbone plus a trainable cache model is a sufficient and superior adapter for federated fine-tuning. The cache model stores visual feature weights $W_1$ and frozen one-hot label weights $W_2$ built from class-balanced synthetic images; predictions fuse CLIP's text-image logits with a distance-weighted cache lookup, $\\text{logits} = f_{\\text{vision}} W_{\\text{text}}^T + \\alpha \\exp(-\\beta(1 - f_{\\text{vision}} W_1)) W_2^T$, and only $W_1$ is trained locally and aggregated. The paper shows this object, initialized from synthetic data and then refined federated, outperforms FedAvg, FedProx, AdapterFL, PromptFL, and zero-shot CLIP across 11 datasets and three data-partition settings, with the largest margins in non-IID regimes, while sending about 100,000 parameters per round and resisting gradient-inversion attacks.","pith_inferences":["Editorial inference: the cache model is effectively a learned nearest-prototype memory, so the same federated protocol could be applied to any frozen encoder whose features are meaningful under cosine or Euclidean similarity, not only CLIP.","Editorial inference: because only the cache matrix is exchanged, standard secure-aggregation or differential-privacy techniques would be much cheaper to add than for full-model federated learning, potentially giving stronger privacy guarantees than the gradient-inversion test alone.","Editorial inference: the choice of the text-to-image model is a testable bottleneck; replacing the generative model with a different one, or using a public feature bank from another domain, should change accuracy in a predictable way if the class-balanced synthetic prior is what carries the non-IID robustness."],"forward_implications":["Per-round communication drops to the size of the cache matrix, about 100,000 parameters, making federated fine-tuning of CLIP practical on bandwidth-limited clients instead of the millions of parameters required by full-model or adapter-based baselines.","On the 11 datasets, CacheFL beats PromptFL by an average of +9.6 points in the IID setting, +7.8 points under Dirichlet non-IID partitioning, and +7.1 points under extreme non-IID partitioning, while also exceeding zero-shot CLIP on every dataset.","The federated training stage carries the largest accuracy gain: on UCF101, accuracy rises from 57.9 percent with only the synthetic initialization to 77.6 percent after federated fine-tuning.","Under the stated assumptions, the federated cache update converges at rate $O(1/t)$, so most of the accuracy improvement appears within the first few communication rounds.","The transmitted cache-model updates resist a standard gradient-inversion attack, with reconstructed images resembling noise rather than the original training images."],"supporting_citations":[{"why":"Supplies the frozen CLIP image and text encoders whose features the cache model is built on and whose logits are fused with the cache logits.","marker":"[2]"},{"why":"Introduces the cache-based logit computation that CacheFL adapts, including the exponential distance weighting and one-hot label cache.","marker":"[15]"},{"why":"Provides the federated averaging rule used to combine client updates to the cache model.","marker":"[23]"},{"why":"The main federated prompt-tuning baseline that CacheFL compares against and outperforms.","marker":"[24]"},{"why":"The generative text-to-image model used to create the class-balanced synthetic dataset that initializes the cache model.","marker":"[14]"},{"why":"A federated optimization baseline with a proximal term that CacheFL compares against on non-IID data.","marker":"[25]"},{"why":"The gradient-inversion attack used to evaluate whether CacheFL's transmitted updates protect local images.","marker":"[40]"}],"fun_headline_variants":["Federated cache tuning cuts communication to 100k parameters","CacheFL: efficient privacy-preserving fine-tuning for CLIP","Lightweight cache model boosts federated CLIP accuracy","Federated cache fine-tuning beats prompt methods on 11 sets","CacheFL: private, efficient federated learning for VLMs"],"cache_read_input_tokens":23168,"weakest_assumption_plain":"The proof's claimed speed of convergence relies on every client's loss being strongly convex, but the actual loss includes an exponential distance term inside the cross-entropy, so strong convexity is asserted rather than proved and the bound does not follow if this assumption fails.","fun_headline_variants_meta":{"raw":{"variants":["Federated cache tuning cuts communication to 100k parameters","CacheFL: efficient privacy-preserving fine-tuning for CLIP","Lightweight cache model boosts federated CLIP accuracy","Federated cache fine-tuning beats prompt methods on 11 sets","CacheFL: private, efficient federated learning for VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"total_tokens":2270,"prompt_tokens":1029,"completion_tokens":1241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":1156}},"tokens_in":645,"tokens_out":1241,"duration_ms":9565,"temperature":1.0,"reasoning_tokens":1156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:12:35.341756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the smallest eigenvalue of the Hessian of the local cross-entropy loss with respect to $W_1$ at several points along a training run; if it is ever zero or negative, Assumption 2 fails and Theorem 1 does not apply to CacheFL as run.","supporting_citations":[{"cited_title":"Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model,","cited_arxiv_id":null,"evidence_quote":"The main federated prompt-tuning baseline that CacheFL compares against and outperforms."}],"review_version":1}