{"id":"9f19019b-94d7-45aa-a93f-eaf904e613c6","arxiv_id":"2502.06355","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MPSL applies parallel split learning with a single aggregated server backward pass to fine-tune multimodal transformers, matching federated learning accuracy while reducing client compute by about 250x.","lead":"This paper proposes MPSL, a way to fine-tune large multimodal AI models across many edge devices by running only lightweight tokenizers on clients and offloading the heavy transformer to a server. The authors report about 250x lower client-side computation than federated learning, with accuracy that is comparable or better on several vision, audio, and text benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The server-side update in MPSL is undefined: Eq. 4's aggregated scalar loss cannot yield gradients for FS without per-sample output gradients or labels.","rationale":"The reader's weakest assumption is precisely the one that matters. The manuscript's training loop is not missing a minor detail; it posits an impossible gradient computation. This is an internal inconsistency, not a matter of disagreeing with community consensus. I considered whether 'loss' could mean a tensor carrying a graph or a loss-gradient, but the text explicitly says 'only sends its loss' and emphasizes label privacy. If the implementation actually sends per-sample gradients or labels, the abstract's empirical claims could be reproducible, but the paper as written fails to describe the algorithm that generated them. Consequently, the rejection is justified. The other issues, such as post-hoc selection of fusion type and batch size and the fact that FedAvg beats MPSL on most classification rows, are secondary: they would weaken the 'matches or outperforms' wording but would not by themselves be fatal. The central algorithmic gap is sufficient.","tokens_in":11923,"tokens_out":4162,"duration_ms":39426,"concrete_test":"In the released repository (github.com/Nousphera/MPSL), locate the server update and record exactly what each client sends: scalar loss, per-sample gradient of LCn with respect to yhat, labels, or something else. Then for one Table 1 configuration (e.g., COCO-QA, N=25), compare the server-side gradients actually used by the code against the true gradient obtained by augmenting communication with per-sample output gradients. If the code only uses scalar losses and the gradients differ, the reported results cannot be produced by the described MPSL mechanism; if the code sends additional information, the paper's algorithm description is inaccurate and must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is in Section 3.2: the server receives only the scalar loss LCn(yhat,y) from each client (Eq. 4, text: 'Since client n only sends its loss to the server'), while the labels stay on-device. To update FS by backpropagation, the server needs dLS/dyhat for each sample, or equivalently dLCn/dyhat; a scalar loss value does not determine this gradient. For cross-entropy, dL/dz = softmax(z) - one_hot(y), which depends on the label; for the contrastive loss used in retrieval, it depends on the full set of labels/similarities. Knowing yhat and the scalar L permits infinitely many consistent gradients. Hence the described 'single backward pass on FS from LS' is not a well-defined computation: Eq. 4 cannot be minimized by gradient descent with only the communicated quantities. This breaks the central claim: MPSL's reported performance cannot follow from the stated algorithm, regardless of the empirical tables. This is an internal inconsistency in the algorithm, not a disagreement with community consensus about what is practical.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MPSL, a parallel split learning framework for fine-tuning multimodal transformers on edge devices. Clients run lightweight modality-specific tokenizers, send activations to a server that processes them through a shared encoder and task head, and the server supposedly performs a single backward pass using an aggregated loss computed from scalar client losses. The authors report evaluations on seven multimodal datasets, claiming that MPSL matches or outperforms federated learning baselines while reducing client-side computation by roughly 250x and improving communication scalability with model size. The paper also includes ablations on batch size, number of fine-tuned blocks, encoder depth, and fusion type.","tokens_in":12139,"tokens_out":3360,"duration_ms":31341,"significance":"The problem addressed is relevant: distributed fine-tuning of large multimodal transformers on resource-constrained clients is an important practical challenge, and the reported efficiency numbers (about 1.0M client-side trainable parameters and 0.1 GFLOPs per sample) would be valuable if the training method were sound. The manuscript is also commendable for its breadth of evaluation across seven datasets, multiple modality pairs, and several ablations, and for releasing code. However, the central training mechanism is not well-defined: the server cannot compute gradients of the aggregated loss with respect to its own model from scalar client losses alone. Because this under-specification affects the core learning loop, the empirical results cannot be attributed to the described algorithm, and the significance of the claimed contributions cannot be assessed without a corrected and re-evaluated training procedure.","major_comments":[{"comment":"The server-side update is not a well-defined computation. The text states that 'client n only sends its loss to the server' and Eq. (4) defines LS as a weighted sum of scalar client losses. To update FS by gradient descent, the server would need the gradient of LCn with respect to the server's output y_hat (or equivalently with respect to the activations an) for each sample. A scalar loss value LCn(y_hat, y) does not determine this gradient: for cross-entropy, the gradient is softmax(z) - one_hot(y), which depends on the unknown label y; for the contrastive loss used in image-text retrieval, it depends on the full set of positive and negative similarities in the batch. Therefore the claim that the server 'performs a single backward pass on FS' using only aggregated scalar losses is not implementable as written, and the results in Tables 1-4 cannot follow from the stated algorithm.","section":"Section 3.2, Eq. (4)"},{"comment":"The same issue breaks the client-side update as well. The paper says that after backpropagation the server sends 'cut-layer gradients' to the clients, which the clients use to update FCn. These gradients are dLS/dan, and computing them requires dLS/dy_hat, which in turn requires labels or per-sample output gradients. Since neither is communicated under the described protocol, the client-side training step is also undefined. The training loop as a whole therefore cannot be executed as described.","section":"Section 3.2, client update paragraph"},{"comment":"The claim that MPSL 'eliminates label sharing' while still updating the server model is internally inconsistent. The only way for the server to compute the needed gradients is for clients to send either their labels or per-sample gradients of the loss with respect to the server's outputs. Sending the latter would preserve label privacy but would change the communication protocol and the stated communication-cost analysis; sending the former would contradict the privacy claim. The manuscript should specify which quantity is actually transmitted and should re-derive the communication and privacy statements accordingly.","section":"Section 3.2, label privacy claim"}],"minor_comments":[{"comment":"Please fix the wording: 'computational efficient' should be 'computationally efficient' in the abstract, and 'splitted' should be 'split' in Section 3.3.","section":"Abstract and Section 3.3"},{"comment":"The caption says entries are 'normalized accuracy and top-1 recall vs. centralized fine-tuning', but the surrounding text describes raw percentages (e.g., accuracy improving from 50 to 500). Please clarify whether the reported numbers are raw or normalized, and if normalized, specify the normalization formula.","section":"Table 3 caption"},{"comment":"Unlike Tables 1 and 2, the encoder-depth experiments in Figure 6 do not report error bars or the number of runs. Adding this information would make the scalability claim easier to assess.","section":"Figure 6"}],"recommendation":"reject","confidential_remarks":"The central flaw in Section 3.2 is load-bearing and cannot be fixed by local editing: the described communication protocol does not allow the server to compute any gradient. If the released code actually transmits per-sample gradients or labels, the manuscript text should be corrected and the experiments re-interpreted; as written, the internal inconsistency is sufficient grounds for rejection. The paper otherwise fits the scope of cs.DC and has a useful experimental setup, so a substantially revised version could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical study of parallel split learning for multimodal transformers, and the client-side cost numbers are worth having. But the training loop as written cannot compute the server update. Section 3.2 says each client sends only its scalar loss to the server, labels stay on-device, and the server then does a single backward pass on FS from the aggregated loss LS = sum |Bn|/|B| LCn. For that backward pass you need dLS/dŷ per sample. For cross-entropy that is softmax(z) − one_hot(y), which requires y; for the contrastive retrieval loss it requires the relative label structure. A scalar loss value plus the logits does not determine those gradients—infinitely many gradients are consistent with the same loss. So Eq. (4) is not a computable objective with the communicated quantities. This is not a peripheral detail; it is the central training mechanism. If the released code transmits per-sample output gradients (or the server has labels, or something else), the paper needs to say so. As written, the reported results do not follow from the stated algorithm.\n\nWhat is good: the first mapping of PSL onto Meta-Transformer-style unified encoders, with seven datasets, three modality pairs, and ablations on fusion, batch size, and encoder depth. The concrete client-side numbers (0.1 GFLOPs, ~1M trainable params) are useful. The communication argument—smashed data rather than model parameters—holds up as model size grows. The ablations and convergence analysis are the most valuable part.\n\nSoft spots beyond the main one: 'matches or outperforms Federated Learning' overstates things. On several classification tasks FedAvg is clearly ahead (COCO-QA 59.1 vs 53.3, Kinetics 78.2 vs 72.2 with N=25). The claim survives only if you read 'matches' loosely or put all weight on the efficiency win. Also, fusion type and batch size are chosen per task after seeing results; the paper is transparent about this, but it weakens the 'MPSL excels' claims. The FedCLIP comparison is a little apples-to-oranges since FedCLIP is a PEFT method with a different efficiency profile.\n\nWho is this for: systems people working on split learning or edge fine-tuning of multimodal transformers. If the gradient issue is resolved, the empirical tables are worth a serious look. As it stands, send to peer review with a clear request to fix or clarify the training loop; accept only after that is resolved.","headline":"Careful empirical mapping of split learning to multimodal transformers, but the core server update as described cannot be computed from scalar client losses; the paper needs major revision before the results can be trusted.","tokens_in":12651,"tokens_out":7540,"would_cite":false,"duration_ms":56749,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A parallel split-learning design fine-tunes multimodal transformers on edge devices with 250x less client-side compute than federated learning.","keywords":["multimodal transformers","split learning","parallel split learning","federated learning","edge computing","parameter-efficient fine-tuning","image-text retrieval","modality fusion"],"falsifier":"Run the protocol exactly as written—clients send only their scalar loss, never labels or per-sample gradient vectors—and check whether the server-side encoder weights change after a training step. For the usual classification loss, knowing only the single loss number does not tell you how each logit contributed, so the server cannot compute its own gradient unless extra information is sent; this can be settled by instrumenting the message contents and the resulting gradients.","tokens_in":11719,"feed_emoji":"⚡","tokens_out":6545,"duration_ms":51622,"temperature":0.7,"pith_summary":"Multimodal transformers are too heavy to fine-tune directly on phones and sensors. This paper proposes MPSL, a parallel split-learning setup that cuts the model at the tokenizer/encoder boundary: edge clients run only lightweight tokenizers, the server runs the transformer encoder and task head, and training proceeds through a single aggregated backward pass. The central claim is that this setup matches or beats standard federated learning on seven multimodal datasets while cutting client-side computation by roughly 250x and keeping labels on the client. If right, it makes distributed fine-tuning of large vision-language-audio models practical on resource-constrained edge devices and gives a concrete scaling story for communication cost as encoders grow.","feed_headline":"Edge devices fine-tune multimodal transformers with 250x less compute","feed_subtitle":"Parallel split learning matches federated learning accuracy while cutting client-side training cost and label sharing.","key_machinery":"The load-bearing object is the split point and the aggregated-loss backward pass. MPSL partitions a multimodal transformer into client-side modality-specific tokenizers and a server-side unified encoder plus task head; the tokenizers convert each modality into a sequence of embeddings that are concatenated early or late, global-average-pooled, and classified on the server. The server then computes one weighted loss $L_S = \\sum_n \\frac{|B_n|}{|B|} L_{C_n}$ over all clients' scalar losses and backpropagates once, which is what removes per-client sub-models, client synchronization, and label sharing from the loop while keeping client-side work at a few tokenizer forward passes.","core_discovery":"The paper's central discovery is that multimodal transformers can be fine-tuned in a split-learning topology without requiring clients to hold a copy of the encoder or synchronize their local models. Clients tokenize their own image, audio, or text inputs, send only the resulting activations to the server, and receive the server's predictions; the client computes its own loss against its local label and sends that scalar loss to the server. The server weights the client losses by batch fraction, performs one backward pass on the whole encoder-head stack, and sends the cut-layer gradients back to clients, so coordination emerges from a shared global gradient signal rather than from FedAvg-style parameter averaging. Across seven datasets covering vision-text, vision-audio, and audio-text pairs, MPSL matches or exceeds the federated baselines on classification tasks while using about 1.0 million trainable client parameters and 0.1 GFLOPs per sample, and it improves communication-cost scaling relative to FedAvg as encoder size grows.","pith_inferences":["The protocol as written sends only scalar losses to the server; if the server is to update its own layers, it needs gradient information that a single scalar cross-entropy value does not determine, so the practical implementation likely must transmit per-sample output gradients (or equivalent) alongside the loss—an implicit re-scoping of the label-privacy claim.","The same single-backward-pass mechanism might transfer to other split points, such as parameter-efficient adapters on frozen backbones, and to heterogeneous edge hardware, though client dropouts and missing modalities are expressly left to future work.","A direct test of the batch-size hypothesis would be to train MPSL with replay buffers or gradient accumulation to simulate large batches on clients that hold very few samples; the paper predicts this should close much of the retrieval gap with centralized training.","If the scalar-loss update can be made to work, MPSL's communication and compute profile makes it a candidate for on-device personalization of large multimodal models, since the post-training model can be reassembled per client or via FedAvg-style aggregation."],"forward_implications":["Client-side computation becomes nearly independent of transformer encoder size, since clients only execute tokenizers and receive gradients.","Communication cost per client is tied to activation (smashed-data) size rather than model parameter count, so MPSL overtakes FedAvg's communication cost as encoders grow toward ViT-L and ViT-H scale.","The shared server-side backward pass acts as a global alignment signal, which the paper reports as faster convergence and more stable training under high client counts and non-IID data.","Label privacy is preserved in principle because the ground-truth label never leaves the client, though the server still sees all intermediate representations.","Retrieval tasks remain the weakest point: distributed training underperforms centralized fine-tuning, and MPSL's gains there depend on large effective batch sizes."],"supporting_citations":[{"why":"Supplies the server-side single-loss aggregated backward pass that MPSL extends from unimodal to multimodal settings.","marker":"[Lyu et al., 2023]"},{"why":"Provides the unified Meta-Transformer encoder and modality-specific tokenizer design and the pretrained weights used in all experiments.","marker":"[Zhang et al., 2023]"},{"why":"Introduced split learning and the label-sharing concern that MPSL claims to eliminate.","marker":"[Gupta and Raskar, 2018]"},{"why":"Introduced parallel split learning, the line of work MPSL directly extends for multimodal transformers.","marker":"[Jeon and Kim, 2020]"},{"why":"Defines FedAvg, the primary federated baseline MPSL compares against and the aggregation used for post-training model construction.","marker":"[McMahan et al., 2017]"},{"why":"Defines FedCLIP, the parameter-efficient federated baseline that MPSL compares against for performance and communication.","marker":"[Lu et al., 2023]"},{"why":"Supplies the ViT backbone and pretrained weights scaled across Tiny through Huge in the encoder-depth experiments.","marker":"[Dosovitskiy et al., 2020]"},{"why":"Supplies the text tokenizer and the contrastive alignment rationale for large batch sizes used in retrieval tasks.","marker":"[Radford et al., 2021]"}],"fun_headline_variants":["Split learning: 250x less edge compute, no label sharing","MPSL: match FedAvg with 250x lighter client devices","Parallel split learning slashes edge compute 250x","Edge multimodal fine-tuning: MPSL skips label sync, cuts compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire training loop rests on the assumption that the server can correctly update its encoder and head from only the scalar loss values sent by clients, with labels never leaving the devices.","fun_headline_variants_meta":{"raw":{"variants":["Split learning: 250x less edge compute, no label sharing","MPSL: match FedAvg with 250x lighter client devices","Parallel split learning slashes edge compute 250x","Edge multimodal fine-tuning: MPSL skips label sync, cuts compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1200,"prompt_tokens":928,"completion_tokens":272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":544,"tokens_out":272,"duration_ms":4215,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:42:10.072277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the protocol exactly as written—clients send only their scalar loss, never labels or per-sample gradient vectors—and check whether the server-side encoder weights change after a training step. For the usual classification loss, knowing only the single loss number does not tell you how each logit contributed, so the server cannot compute its own gradient unless extra information is sent; this can be settled by instrumenting the message contents and the resulting gradients.","supporting_citations":[{"cited_title":"Scalable aggregated split learning for data-driven edge intelligence on internet-of-things","cited_arxiv_id":null,"evidence_quote":"Supplies the server-side single-loss aggregated backward pass that MPSL extends from unimodal to multimodal settings."},{"cited_title":"Dis- tributed learning of deep neural network over multiple agents,","cited_arxiv_id":null,"evidence_quote":"Introduced split learning and the label-sharing concern that MPSL claims to eliminate."},{"cited_title":"Privacy-sensitive parallel split learning","cited_arxiv_id":null,"evidence_quote":"Introduced parallel split learning, the line of work MPSL directly extends for multimodal transformers."},{"cited_title":"Communication-Efficient Learning of Deep Networks from De- centralized Data","cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, the primary federated baseline MPSL compares against and the aggregation used for post-training model construction."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the text tokenizer and the contrastive alignment rationale for large batch sizes used in retrieval tasks."}],"review_version":1}