{"id":"88fc249f-0bdc-4341-ac58-5afcbeea5d6b","arxiv_id":"2412.20064","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VELoRA applies modality-specific and modality-shared LoRA adapters to a frozen CLIP ViT for RGB-event recognition, reaching 57.99% on PokerEvent and 50.89% on HARDVS.","lead":"This paper introduces VELoRA, a method that fine-tunes a frozen vision transformer for recognizing objects and actions from both regular video and event camera data, updating only a small set of low-rank adapter parameters. It reports small accuracy gains over prior work while reducing trainable parameters from 1.7 GB to 7 MB.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an unspecified, self-admittedly static event-image representation; without a reproducible conversion or a test of an alternative representation, the small reported gains over SAFE are not anchored.","rationale":"I reviewed the paper in good faith. The central claim is that VELoRA is the first PEFT method for RGB-event recognition and achieves SOTA accuracy with far fewer trainable parameters. The most load-bearing condition is the event input representation: the method's three branches all encode a static 'event image' whose construction is never specified, and the frame-difference branch uses an undefined mean of consecutive differences. The authors' own limitation statement (Section IV-G) weakens the motion-cue claim. The accuracy advantages over SAFE are small enough (0.36 pp and 0.01–0.72 pp) that representation sensitivity could reverse them. This agrees with the reader's weakest_assumption. I also note the FLOPs contradiction (Table VIII shows 41.35 GB for VELoRA vs 30.98 GB for full tuning while the text claims 'lower FLOPS') and the absence of reported full fine-tuning accuracy, but those are secondary to the representation issue. The appropriate verdict remains CONDITIONAL: release code and specify the event representation, add error bars, and include a same-backbone full fine-tuning baseline. My read does not change the reader's conditional verdict.","tokens_in":16543,"tokens_out":12529,"duration_ms":127950,"concrete_test":"Re-run the full VELoRA pipeline on PokerEvent and HARDVS with the event input replaced by a standard 5-bin voxel grid (timestamps quantized into 5 bins per frame) and, separately, by the authors' event-image conversion, while keeping the frozen CLIP ViT, LoRA rank/placement, fusion, and loss identical. If the top-1 accuracy difference between the two event encodings exceeds the reported margins over SAFE (0.36% on PokerEvent, 0.72% on HARDVS), the SOTA claim is representation-dependent and the missing specification in Section III-C is load-bearing; if the difference is below 0.1%, the omission is cosmetic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—state-of-the-art RGB-event recognition via PEFT—rests on the input encoding described in Section III-C: 'we convert the event stream E into event images aligned with the timestamps of the video frames.' No algorithm, binning scheme, or normalization is given, and Td for the frame-difference branch is never defined ('We calculate the mean of the differences between consecutive frames and process it through the aforementioned vision backbone network'). All three branches (RGB, event, frame difference) feed this static representation into a frozen CLIP ViT, so the representation is the foundation of every reported accuracy number. The authors concede in Section IV-G that 'the static representation of event streams may limit the effectiveness of temporal information.' This matters because the claimed margins are tiny: 57.99% vs. 57.63% on PokerEvent and 50.89% vs. 50.17%/50.88% on HARDVS. If a standard alternative representation (e.g., a voxel grid or point cloud preserving event timestamps) shifted accuracy by even 0.5%, the ranking against SAFE could reverse, and the claim of a general PEFT framework for RGB-event recognition would not be established. Without the event-image construction, the experiments cannot be reproduced or compared.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VELoRA, a parameter-efficient fine-tuning (PEFT) method for RGB-event recognition. The method freezes a CLIP ViT-B/16 backbone and inserts LoRA adapters in two regimes: modality-specific LoRA in low-level Transformer blocks for RGB, event, and frame-difference branches, and modality-shared LoRA in the final high-level Transformer block for fusion. A reconstruction loss between RGB and event features is added. The authors report state-of-the-art top-1 accuracy on PokerEvent (57.99% vs SAFE's 57.63%) and HARDVS (50.89% vs SAFE's 50.17% and C3D's 50.88%), while reducing trainable parameters from 1719MB to 7.02MB in their comparison with full fine-tuning. The paper claims this is the first PEFT approach for RGB-event recognition based on pre-trained foundation models.","tokens_in":16782,"tokens_out":4122,"duration_ms":43350,"significance":"If the claims are robust, the paper would make a useful contribution: it demonstrates that a low-rank adaptation strategy can be organized into modality-specific and modality-shared parts for multi-modal RGB-event data, and the reduction in trainable parameters is substantial and practically relevant. The idea of using frame differences as an auxiliary modality within a frozen-backbone PEFT framework is reasonable. However, the headline accuracy gains over prior methods are very small (0.36 points on PokerEvent; 0.01-0.72 points on HARDVS depending on the baseline), and the current manuscript does not provide enough statistical or procedural support for these margins. The efficiency claims are also partially inconsistent with the reported table. The central ideas are plausible, but the evidence as presented is not yet sufficient to support state-of-the-art and efficiency claims at the level of rigor expected for a journal publication.","major_comments":[{"comment":"The input encoding is underspecified. The text states that \"we convert the event stream E into event images aligned with the timestamps of the video frames,\" but no algorithm, binning scheme, normalization, polarity handling, or resolution mapping is given. In addition, the frame-difference branch is defined only as \"the mean of the differences between consecutive frames,\" and the symbol T_d used in Eq. (3) is never defined. Since all three branches (RGB, event, frame difference) feed this static representation into the frozen CLIP ViT, the entire accuracy comparison depends on a representation the reader cannot reproduce. The authors themselves concede in Section IV-G that \"the static representation of event streams may limit the effectiveness of temporal information.\" Please provide the exact conversion procedure (with formulas or code), and ideally compare at least one alternative event representation (e.g., voxel grids or time surfaces) to show that the reported results are not an artifact of this particular encoding.","section":"Section III-C, Eqs. (2)-(3)"},{"comment":"The hyperparameter selection procedure undermines the statistical strength of the state-of-the-art claim. Tables IV, V, and VI vary the number of input frames, the LoRA rank, and the LoRA insertion location, and the configuration with the best test accuracy (rank 4, 8 frames, MLP-only LoRA) is then used for the headline results in Tables I and II. The paper does not describe a held-out validation split; the PokerEvent description gives only training and test sizes, and no validation set is mentioned. With margins as small as 0.36 points over SAFE on PokerEvent and 0.01 points over C3D on HARDVS, run-to-run variation could easily change the ranking. Please report means and standard deviations over multiple seeds, or use a proper validation protocol for model selection, and state the number of runs used for the final numbers.","section":"Section IV-D, Tables III-VI"},{"comment":"The efficiency claims are internally inconsistent with the reported table. Table VIII shows that VELoRA has higher FLOPs than full fine-tuning (41.35 GB vs 30.98 GB), while the text states that \"our proposed method offers a quicker runtime and lower FLOPS.\" The training time is only slightly lower (4240 s vs 4536 s), and the memory reduction is 2054 MB, not the 2200 MB stated in the abstract (19574 - 17520 = 2054). The trainable-parameter reduction from 1719 MB to 7.02 MB is impressive and well supported, but the broader efficiency narrative should be corrected to match the data, with units and measurement conditions clarified.","section":"Section IV-E, Table VIII"},{"comment":"The formal description of the proposed architecture is too schematic to be reproducible. Eq. (4) writes a Transformer block update as F(l+1) = F(l) + (W0 + ΔW)F(l), which omits the multi-head attention, normalization layers, and residual structure that define a ViT block; it also does not clarify which linear projections receive the LoRA decomposition beyond the later statement that only MLP layers are updated. Eqs. (5)-(6) similarly leave unclear whether the \"modality-shared LoRA\" is applied to the attention output, the MLP, or both, and how the concatenated features are split after fusion. Please provide a precise layer-by-layer forward pass or pseudocode, or release the exact code and refer to it in the paper; currently the architecture cannot be reimplemented from the text alone.","section":"Section III-C, Eqs. (4)-(6)"}],"minor_comments":[{"comment":"There is a typo in \"Ff useusing LoRA\" near Eq. (5); it should read \"F_fuse using LoRA.\"","section":"Section III-C"},{"comment":"The method name \"R2PlusID\" should be \"R2Plus1D\" to match the cited reference and standard nomenclature.","section":"Table II"},{"comment":"Please clarify the units of \"Training Time (s)\": if this is the total training time for 50 epochs, that is about 85-90 s per epoch; if it is per-epoch time, the comparison should be labeled consistently.","section":"Section IV-B and Table VIII"},{"comment":"In Table III, rows 1 and 2 list only one component (F.D. or Reconst.) while the results are 55.78 and 55.94; the text says \"when the model was trained with only the frame difference module or the reconstruction step, the highest accuracy achieved was 55.94,\" which is consistent, but the table would benefit from also showing a no-component baseline to quantify the contribution of each module.","section":"Section IV-D, Table III"},{"comment":"Please specify whether the PokerEvent and HARDVS train/test splits follow the official protocols of the dataset papers (e.g., [12], [41]) and whether the same splits were used for all compared methods; this information is essential for a fair comparison but is not currently stated.","section":"Section IV-A"},{"comment":"The reconstruction loss equations use M for both the number of patch tokens and the number of samples; please disambiguate the notation and clarify over which dimension the summation is taken.","section":"Equations (8)-(9)"}],"recommendation":"major_revision","confidential_remarks":"The core idea—modality-specific and modality-shared LoRA for RGB-event fusion—is reasonable and timely, and the parameter-count reduction is genuinely impressive. My main concern is evidential: the small accuracy margins, the apparent test-set-based hyperparameter selection, and the missing event-encoding specification jointly make the state-of-the-art claim fragile. These are fixable with additional experiments and a clearer protocol, so I do not recommend rejection. I would also encourage the editor to ask the authors to reconcile Table VIII's FLOPs/training-time numbers with the efficiency narrative before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: VELoRA is a plausible first PEFT method for RGB-event recognition, but the reported gains are too thin and the event-input encoding too under-specified to carry the SOTA claims as written.\n\nWhat's genuinely new: applying LoRA to cross-modal RGB-event recognition with a split into modality-specific low-level and shared high-level adaptations, plus a frame-difference branch and cross-modal reconstruction loss. The combination is not in the cited literature. The efficiency numbers are real: about 7MB trainable parameters vs 1.7GB for full fine-tuning, and roughly 2GB less training memory. If those hold, that's useful for practical deployment.\n\nThe architecture is sensible. Freezing a CLIP ViT and inserting LoRA in low-level blocks per modality, then fusing with a shared LoRA block, is a natural way to adapt a foundation model to a new modality pair. The ablations show each added component helps, and the paper is honest enough to include a limitation section admitting the static event representation may limit temporal information.\n\nThe soft spots are mostly in the evidence. On PokerEvent the margin over SAFE is 0.36 points; on HARDVS it's 0.01 over C3D. With no error bars and no multiple seeds, those are not distinguishable from noise. The hyperparameters (rank, frame count, LoRA placement) were chosen by ablations on the test set, so the headline numbers are partly selected. The event-image construction in Section III-C is never specified: no binning, normalization, or definition of Td for the frame-difference branch. That is a reproducibility blocker, and the stress-test note is right that a different event representation could move accuracy by more than the observed gains. There is also a direct internal contradiction: the text claims lower FLOPs while Table VIII reports 41.35 GB for VELoRA vs 30.98 GB for full fine-tuning, and the claimed 2200MB memory saving is actually 2054MB. Finally, no same-backbone full fine-tuning baseline is reported, only a passing mention that LoRA beats it.\n\nWho should read it: anyone working on event-based recognition or PEFT for multimodal tasks. It deserves a serious referee, because the idea is timely and the efficiency claim is testable. But acceptance should require a complete event-representation description, code release, variance estimates, a corrected efficiency table, and a real full fine-tuning comparison. I'd recommend sending it to review rather than desk rejecting, with the expectation of major revision.","headline":"Plausible PEFT recipe for RGB-event fusion, but the accuracy margins are too small and the event-input encoding too under-specified to support the SOTA claims as written.","tokens_in":17371,"tokens_out":7271,"would_cite":true,"duration_ms":67299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VELoRA adapts a frozen CLIP ViT for RGB-event recognition by tuning only low-rank matrices, reporting 57.99% on PokerEvent and 50.89% on HARDVS while cutting trainable parameters to 7.02 MB.","keywords":["parameter-efficient fine-tuning","LoRA","event camera","RGB-Event fusion","multi-modal fusion","vision transformer","action recognition"],"falsifier":"Keep VELoRA's LoRA tuning identical but replace the event-image input with a voxel-grid or raw-event representation; if accuracy on HARDVS or PokerEvent falls below the reported 50.89% or 57.99% (or below SAFE's 50.17% and 57.63%), the gains depend on the specific static event rendering rather than on the LoRA fusion design.","tokens_in":16323,"feed_emoji":"🎥","tokens_out":7797,"duration_ms":66981,"temperature":0.7,"pith_summary":"The paper tries to establish that parameter-efficient fine-tuning can be applied to RGB-event recognition by keeping a pre-trained vision transformer frozen and updating only small low-rank matrices. On the PokerEvent benchmark it reports 57.99% top-1 accuracy, edging past the previous best SAFE at 57.63%, and on HARDVS it reports 50.89%, above ESTF, SAFE, and C3D. It also reports that trainable parameters drop from 1719 MB to 7.02 MB and runtime memory falls by about 2200 MB. If these results hold, VELoRA would be the first PEFT method for RGB-event recognition and a template for adapting frozen foundation models to other multi-modal tasks.","feed_headline":"7 MB of LoRA weights beats full fine-tuning in RGB-event recognition","feed_subtitle":"Tuning only low-rank matrices on a frozen CLIP ViT gives state-of-the-art accuracy with far fewer trainable parameters.","key_machinery":"The load-bearing object is Low-Rank Adaptation (LoRA), which adds a trainable low-rank update $\\Delta W = BA$ to a frozen weight matrix $W_0$, so the forward pass becomes $h = W_0 x + BA x$. VELoRA distributes LoRA in a two-stage design: modality-specific LoRA blocks in the low-level Transformer layers of the RGB, event, and frame-difference branches, followed by a modality-shared LoRA block in the final high-level Transformer layer that fuses the concatenated features. The third branch takes the mean of differences between consecutive frames (eight frames in the reported setting) and encodes it with the same frozen backbone, giving an explicit motion channel. An MSE reconstruction loss between RGB and event features, $L_{RTE}$ and $L_{ETR}$, is added to the cross-entropy classification loss, and the total loss is their sum.","core_discovery":"VELoRA's central claim is that a frozen CLIP ViT-B/16 can be adapted to RGB-event classification by inserting LoRA blocks at two levels. Low-level Transformer blocks receive modality-specific LoRA tuning in three separate branches: RGB frames, event images, and the mean of consecutive-frame differences, which supplies motion cues. A reconstruction step with MSE loss lets the RGB and event branches predict each other's features, encouraging cross-modal alignment. The last Transformer block then fuses the concatenated features with modality-shared LoRA tuning before a classification head. The paper reports that this scheme beats full fine-tuning and specialized LoRA variants on both benchmarks while updating only the low-rank matrices.","pith_inferences":["The static event-image representation, which the paper itself flags as a limitation, means the temporal information in event streams is only captured indirectly through the frame-difference branch; pairing VELoRA with a learnable event representation could close the gap to full temporal models.","The same two-stage LoRA recipe (modality-specific encoders plus a shared fusion LoRA) could apply to other paired-modality tasks such as RGB-depth or RGB-thermal recognition, where frozen foundation models are available for one modality.","The reported 2200 MB memory saving and parameter reduction come with higher FLOPs in the paper's table (41.35 GB vs 30.98 GB), so the efficiency claim is best read as memory- and parameter-centric rather than compute-centric."],"forward_implications":["If the reported results hold, RGB-event recognition no longer requires full fine-tuning of a large backbone; a frozen CLIP ViT with LoRA blocks reaches state-of-the-art accuracy.","The component ablations support that each piece contributes: adding the frame-difference branch, reconstruction loss, modality-specific LoRA, and modality-shared LoRA raises accuracy from 55.78% to 57.99% on PokerEvent.","The method transfers to at least two different recognition targets, fine-grained poker-character classification and human action recognition, suggesting the PEFT recipe is task-agnostic within RGB-event inputs.","Because trainable parameters fall to 7.02 MB, deployment becomes cheaper, since each new task only needs to store small LoRA matrices per modality."],"supporting_citations":[{"why":"Defines low-rank adaptation (LoRA), the mechanism on which all VELoRA components build.","marker":"[15]"},{"why":"Supplies the frozen ViT-B/16 CLIP vision encoder used to encode RGB, event, and frame-difference inputs.","marker":"[46]"},{"why":"SAFE is the state-of-the-art RGB-event baseline that VELoRA must beat and shares the same ViT-B/16 backbone.","marker":"[11]"},{"why":"Introduces the PokerEvent dataset and the SSTformer baseline used for evaluation.","marker":"[41]"},{"why":"Introduces the HARDVS dataset and the ESTF baseline; the training and test splits used here come from it.","marker":"[12]"},{"why":"DyLoRA is one of the LoRA variants compared in the ablation study.","marker":"[42]"},{"why":"LoRA-FA is another LoRA variant compared, and its freeze-A strategy is contrasted with VELoRA.","marker":"[43]"},{"why":"MoE-LoRA represents the mixture-of-experts LoRA baseline against which the multi-modal LoRA design is compared.","marker":"[61]"}],"fun_headline_variants":["LoRA tuning on frozen CLIP ViT beats full fine-tuning for RGB-event","VELoRA: Low-rank adaptation rivals full fine-tuning in RGB-event recognition","Frozen CLIP + LoRA: 7 MB beats full fine-tuning for RGB-event classification","Low-rank LoRA on vision transformer matches full fine-tuning on RGB-event","Efficient fine-tuning: LoRA on frozen CLIP outperforms full tuning in RGB-event"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that static, frame-aligned event images encoded by a frozen CLIP ViT preserve enough information for recognition, with the frame-difference branch as the only explicit motion cue—a choice the paper itself concedes may limit temporal effectiveness.","fun_headline_variants_meta":{"raw":{"variants":["LoRA tuning on frozen CLIP ViT beats full fine-tuning for RGB-event","VELoRA: Low-rank adaptation rivals full fine-tuning in RGB-event recognition","Frozen CLIP + LoRA: 7 MB beats full fine-tuning for RGB-event classification","Low-rank LoRA on vision transformer matches full fine-tuning on RGB-event","Efficient fine-tuning: LoRA on frozen CLIP outperforms full tuning in RGB-event"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3759,"prompt_tokens":939,"completion_tokens":2820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2712}},"tokens_in":555,"tokens_out":2820,"duration_ms":18827,"temperature":1.0,"reasoning_tokens":2712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:35:46.710251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Keep VELoRA's LoRA tuning identical but replace the event-image input with a voxel-grid or raw-event representation; if accuracy on HARDVS or PokerEvent falls below the reported 50.89% or 57.99% (or below SAFE's 50.17% and 57.63%), the gains depend on the specific static event rendering rather than on the LoRA fusion design.","supporting_citations":[{"cited_title":"Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models","cited_arxiv_id":"2311.18592","evidence_quote":"SAFE is the state-of-the-art RGB-event baseline that VELoRA must beat and shares the same ViT-B/16 backbone."},{"cited_title":"Hardvs: Revisiting human activity recognition with dynamic vision sensors,","cited_arxiv_id":null,"evidence_quote":"Introduces the HARDVS dataset and the ESTF baseline; the training and test splits used here come from it."}],"review_version":1}