{"id":"11e40783-9565-463f-8f6e-607f2970dfa8","arxiv_id":"2412.03093","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A CLIP-based encoder for event cameras, trained with contrastive, consistency, and KL losses, improves zero-shot and few-shot object recognition and extends to video anomaly detection and cross-modal retrieval.","lead":"This paper trains an event-camera encoder so it can use CLIP's language and image understanding, scoring higher on zero-shot object recognition than prior event CLIP models. It also shows the encoder can spot anomalies in surveillance video and retrieve related sounds and depth maps, without extra training on those tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'state-of-the-art' claim is internally contradicted by Table 2: on N-MNIST zero-shot the method (46.95) trails EventBind (56.81) and EventCLIP (48.72), and in 1-shot it trails EventBind (69.43 vs 74.64); the claimed +15.16% zero-shot gain is dataset- and baseline-specific.","rationale":"I read the paper in good faith: the method is straightforward and plausible, the experiments span multiple tasks, and the code is released. However, the central claim—state-of-the-art object recognition with the quoted gains—is contradicted by the authors' own Table 2. The reader's verdict is CONDITIONAL, and my concern reinforces that condition: the paper needs a major revision to align its claims with the evidence. I do not identify a fatal flaw in the training method itself; rather, the stated headline is overbroad. I chose this concern over the reader's weakest_assumption (the single-frame event representation) because, for the object-recognition datasets used, summing events over time and polarity is a well-established representation (e.g., event histograms) and likely sufficient for static-object classification; the internal inconsistency in the SOTA claim is more directly tied to the central contribution and more easily testable. I also note the reader's other red flags—Proposition 1's unsupported momentum update and the lack of error bars in few-shot results—as secondary issues that further support the need for revision. The concrete test I propose would settle whether the SOTA claim can be defended under a fair, consistent protocol; if it fails, the abstract and Section 1 must be rewritten to report dataset-wise results rather than a blanket 'state of the art.'","tokens_in":13741,"tokens_out":7367,"duration_ms":69330,"concrete_test":"Reproduce Table 2 under a strictly controlled protocol: use the same backbone (ViT-B/32) for all methods, the same prompt template ('a photo of {class}'), the same 80/20 class split for N-ImageNet, and at least 5 random seeds for few-shot sampling. Report zero-shot and 1-shot accuracy on N-ImageNet, N-Caltech101, and N-MNIST. If the proposed method does not achieve the highest accuracy on all three datasets, the Abstract and Contribution section must replace 'state-of-the-art' with dataset-specific claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the Abstract and Section 1 states that the proposed encoder 'achieves state-of-the-art performance in object recognition,' with specific gains over the state of the art (+15.16% zero-shot, +18.91% 1-shot, +7.35% fine-tuning). This claim is not supported by the paper's own Table 2. On N-ImageNet, the zero-shot result of 37.9 exceeds EventCLIP (22.74) and EventBind (11.52), yielding the +15.16% figure only when compared against EventCLIP. On N-Caltech101, Ours (66.34) is +4.54 over EventBind (61.8). However, on N-MNIST, Ours (46.95) is below both EventBind (56.81) and EventCLIP (48.72) in zero-shot, and in 1-shot Ours (69.43) is below both EventBind (74.64) and EventCLIP (74.62). The paper itself concedes in Section 4.6 that N-MNIST shows 'relatively low zero-shot performance.' Thus the unqualified phrase 'state-of-the-art performance in object recognition' is false as stated; the improvement exists on two of three datasets and relies on choosing EventCLIP as the baseline on N-ImageNet and EventBind on N-Caltech. The fine-tuning gain of +7.35% is also only on N-ImageNet with ViT-L/14; on N-MNIST, fine-tuned accuracy ties EventBind at 99.45 rather than exceeding it. Because the headline contribution is precisely this SOTA transfer, the overstated claim is load-bearing: if the claim is revised to per-dataset, the paper's novelty and impact are substantially weaker. This is not an issue of disagreement with consensus; it is an internal inconsistency between the paper's assertions and its own reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an event encoder for CLIP that transfers CLIP's image-text alignment and zero-shot capabilities to event data. Events are aggregated over time and polarity into a single gray-scale frame (Eq. 1) and fed to an event encoder initialized from CLIP's image encoder, while the image and text encoders remain frozen. Training combines an InfoNCE contrastive loss between event and image embeddings (Eq. 2), a zero-shot consistency loss adopted from ZSCL (Eq. 3), and a KL-divergence loss between event and image embedding distributions (Eq. 5). The authors report object-recognition results on N-ImageNet, N-Caltech101, and N-MNIST in zero-shot, few-shot, and fine-tuning settings, claim state-of-the-art performance with gains of +15.16% zero-shot, +18.91% 1-shot, and +7.35% fine-tuning, and additionally demonstrate event-based video anomaly detection and zero-shot cross-modal retrieval after integrating the encoder with ImageBind.","tokens_in":14183,"tokens_out":3870,"duration_ms":37366,"significance":"If the empirical results hold, the paper makes a useful practical contribution: it shows that a simple single-frame event representation, combined with a carefully chosen set of alignment losses, can transfer a substantial portion of CLIP's zero-shot and text-alignment capability to event data, and that the resulting encoder can be plugged into a cross-modal framework such as ImageBind. The release of code and the use of standard, publicly available datasets are positive features. However, the headline claim of state-of-the-art object recognition is contradicted by the paper's own Table 2 on N-MNIST, and the supporting theoretical proposition (Proposition 1) is not grounded in the training procedure actually used. The object-recognition results on N-ImageNet and N-Caltech101 are still interesting, and the cross-modal retrieval and VAD demonstrations add breadth, but the paper currently overstates its achievements and needs substantial revision to make its claims precise and reproducible.","major_comments":[{"comment":"The unqualified claim of state-of-the-art object recognition with gains of +15.16% zero-shot, +18.91% 1-shot, and +7.35% fine-tuning is not supported by the paper's own results. On N-MNIST in Table 2, the proposed method achieves 46.95 zero-shot, below EventBind (56.81) and EventCLIP (48.72), and 69.43 in 1-shot, below both baselines (74.64 and 74.62). The +15.16% figure refers only to N-ImageNet zero-shot against EventCLIP, and even Section 4.6 concedes that N-MNIST shows 'relatively low zero-shot performance.' The claims should be recast per dataset and per baseline, or the state-of-the-art claim should be withdrawn.","section":"Abstract, Section 1, Table 2, Section 4.6"},{"comment":"The momentum-like update rule θ_E ← m·θ_E + (1-m)·(θ_target - η·∇θ_E L_q) is not derived from the objective in Eq. (3), and θ_target is never defined. The gradient expression in Eq. (4) is the gradient of only the first softmax term in Lzs (the image-text term is constant with respect to θ_E), and the proposition does not explain how this leads to the proposed update. Moreover, Section 4.1 describes standard training with a learning rate of 1e-6 for 200 epochs and does not mention using this update rule in any experiment. As written, Proposition 1 is an unsupported assertion rather than a proposition, and it should be removed or replaced with a precise statement of what is actually computed.","section":"Section 3.3, Proposition 1, Eq. (4)"},{"comment":"No error bars or multiple-seed results are reported for any experiment. This is especially problematic for few-shot results where the gap between methods is small; for example, N-MNIST 5-shot accuracy is 94.23 for the proposed method versus 94.44 for EventBind, a difference of 0.21 that could easily reverse with a different random sample. The zero-shot and fine-tuning comparisons in Tables 1 and 2 also lack variance estimates. At minimum, the authors should report mean and standard deviation over at least three seeds for the few-shot settings, or clearly state that all numbers are single runs and temper the corresponding conclusions.","section":"Tables 1-4, Section 4.1"},{"comment":"The VAD experiments compare only against zero-shot CLIP on three datasets, which is a weak baseline for anomaly detection. To support the claim that the event encoder 'expands the applicability' of the event modality to VAD, the paper should compare with existing weakly supervised or zero-shot VAD methods, or at least discuss why CLIP is the appropriate reference and acknowledge that current supervised VAD systems achieve much higher AUC. In addition, the event-extraction hyperparameters (pixel threshold 25, 16-frame stacks) are not studied or justified; a small sensitivity analysis would strengthen the section.","section":"Section 4.3, Table 3"}],"minor_comments":[{"comment":"The phrases '+17.16% and +4.54%' and '+18.89% and +5.47%' do not state which baseline each gain refers to; the gains appear to use EventCLIP for N-ImageNet and EventBind for N-Caltech101. This should be stated explicitly to avoid cherry-picking ambiguity.","section":"Section 4.2, paragraph beginning 'Zero & Few-Shot'"},{"comment":"The caption says the approach demonstrates 'superior performance' on N-MNIST, but the ViT-L/14 result of 99.45 ties with EventBind (99.45); 'competitive' or 'comparable' would be more accurate.","section":"Table 1 caption"},{"comment":"There is a typo: 'image model‘s capabilities' uses a curly apostrophe erroneously; also 'V AD' in Section 4.1 has an extra space. The text should be proofread.","section":"Section 2, first paragraph"},{"comment":"The claim that event retrieval is performed 'without additional training' is confusing because an adapter layer is introduced: 'we incorporate an adapter layer, designed as a single-layer module.' Please clarify whether the adapter is trained and, if so, on which data, or rephrase the claim.","section":"Section 4.4, first paragraph"},{"comment":"The ablation figure shows accuracy curves over training for different loss configurations but provides no numerical summary or error bars; adding a table with final values would make the ablation quantitative and easier to interpret.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable candidate for a workshop or a conference with a broad empirical scope, but the current manuscript overstates its results. The strongest genuine contribution is the empirical demonstration on N-ImageNet and N-Caltech101; the N-MNIST results and the unsupported Proposition 1 need careful handling. The VAD and retrieval sections are more preliminary. I would not reject the paper outright, but the authors must substantially revise the claims and add statistical rigor before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth a serious look, but the headline SOTA claim does not survive contact with the paper's own Table 2. The method itself is a sensible incremental step in the EventBind/EventCLIP line: freeze CLIP's image and text encoders, train an event encoder initialized from the image encoder with contrastive plus zero-shot consistency plus KL losses, using a single-frame event representation. Code is released, evaluation is on held-out classes, and video anomaly detection from extracted events is genuinely new. The ablation showing all three losses contribute is the strongest evidence in the paper.\n\nWhere it gets soft: the abstract and intro claim \"state-of-the-art performance in object recognition\" with gains of +15.16%, +18.91%, and +7.35% is only true for selected baselines on two of three datasets. Their own Table 2 shows N-MNIST zero-shot at 46.95, below EventBind's 56.81 and EventCLIP's 48.72, and 1-shot also below both. The fine-tuned +7.35% is only N-ImageNet with ViT-L/14; on N-MNIST they tie EventBind at 99.45 rather than beat it. That does not kill the method, but it changes the contribution from \"state-of-the-art\" to \"strong on N-ImageNet and N-Caltech, weak on N-MNIST.\" The numbers in Section 4.2 (+17.16, +18.89) also do not match the abstract percentages, which is sloppy.\n\nProposition 1 is the clearest defect. The gradient formula is just the standard softmax contrastive gradient; it does not imply the momentum update rule θE ← m·θE + (1−m)·(θtarget − η∇Lq). That rule is asserted, not derived, and no such momentum mechanism appears in the experimental setup. It should be removed or replaced with a statement that the loss encourages alignment with the frozen encoders.\n\nMinor but real: there are no error bars or seeds anywhere; the VAD baseline is only CLIP with no adapters, so the +27.44 on XD-Violence needs context; retrieval numbers have no baselines; and the simple gray-scale aggregation discards polarity and timing, though the paper does acknowledge this in Section 4.6.\n\nVerdict: this deserves referee time. The empirical core — that this aggregation plus the loss combination beats EventCLIP and EventBind on N-ImageNet and N-Caltech — is plausible and checkable from the released code. I would send it out with a request for major revision: fix the SOTA claim, correct or delete Proposition 1, and add variance and baseline detail.","headline":"Solid incremental event-CLIP encoder with a new VAD application, but the SOTA claim is contradicted by its own Table 2 and Proposition 1 is unsupported.","tokens_in":14745,"tokens_out":2455,"would_cite":true,"duration_ms":24010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CLIP-based event encoder, trained with contrastive, zero-shot consistency, and KL-divergence losses, transfers CLIP's zero-shot and text-aligned recognition to event data and outperforms prior event-CLIP alignment methods.","keywords":["event-based vision","CLIP","zero-shot learning","contrastive learning","cross-modal alignment","catastrophic forgetting","object recognition","video anomaly detection"],"falsifier":"Train the same three-loss recipe with an event representation that preserves temporal structure, such as splitting the stream into several short time bins or separate polarity channels, then compare zero-shot accuracy on N-ImageNet and a motion-sensitive dataset; if the single-frame aggregation is load-bearing, preserving temporal detail should not improve accuracy, but if the assumption is too strong, the temporally structured representation would outperform the reported single-frame numbers.","tokens_in":13524,"feed_emoji":"⚡","tokens_out":6552,"duration_ms":53668,"temperature":0.7,"pith_summary":"The paper aims to give event-based cameras—sensors that record only pixel-level brightness changes—the same zero-shot and text-aligned recognition abilities as CLIP's image model. It does so by training a trainable event encoder, initialized from CLIP's frozen image encoder, to map a simple single-frame event representation into CLIP's shared embedding space. The training objective combines an InfoNCE contrastive loss between event and image embeddings, a zero-shot consistency loss that keeps both anchored to text embeddings, and a KL-divergence loss between the two embedding distributions. The authors report state-of-the-art object recognition on N-ImageNet, N-Caltech, and N-MNIST, and show the encoder transfers to video-extracted events for anomaly detection and to cross-modal retrieval across image, text, sound, and depth. The payoff, if correct, is that sparse event data can inherit the semantic understanding of large image-text models without collecting massive event datasets.","feed_headline":"CLIP's zero-shot power transfers to event cameras","feed_subtitle":"One trainable event encoder, guided by frozen image and text encoders, beats prior event-CLIP alignment models.","key_machinery":"The argument rests on three pieces. First, a single-frame event representation: the event stream $E(x,y,t,p)$ is summed over time $t$ and polarity $p$ and normalized, $\\bar{E} = E(x,y)/(\\max(E(x,y)) + 1)$, producing a one-channel gray-scale frame that the frozen CLIP image encoder can ingest. Second, a trainable event encoder $f_E$ initialized from CLIP's image encoder $f_I$, with $f_I$ and the text encoder $f_T$ frozen throughout training, so that only $f_E$ is updated. Third, the composite objective $L = L_{ct} + \\alpha L_{zs} + L_{kl}$: $L_{ct}$ is the InfoNCE contrastive loss treating event embeddings as queries and image embeddings as keys; $L_{zs}$ is the zero-shot consistency loss that forms prediction logits from both event and image embeddings against text embeddings, keeping the event encoder from drifting away from CLIP's text-aligned semantics; and $L_{kl}$ is the KL divergence between event and image embedding distributions, which sharpens how well the event encoder reproduces the image encoder's feature distribution.","core_discovery":"On the paper's own terms, the central discovery is that CLIP's zero-shot capability can be transferred to event data by training only an event encoder, while freezing the image and text encoders, using a combination of three losses: contrastive alignment of event embeddings with image embeddings (InfoNCE), a zero-shot consistency loss that aligns both event and image embeddings with text embeddings to prevent catastrophic forgetting, and a KL-divergence loss that matches the event embedding distribution to the image embedding distribution. This yields an event-image-text aligned model that outperforms prior event-CLIP alignment methods, with reported gains of +15.16% in zero-shot, +18.91% in 1-shot, and +7.35% in fine-tuning on N-ImageNet over the state-of-the-art using the same alignment method. The same encoder, without additional training, generalizes to events generated from ordinary video for anomaly detection and can be plugged into a cross-modal framework (ImageBind) to enable zero-shot retrieval between events and image, text, sound, and depth.","pith_inferences":["If the single-frame aggregation is indeed sufficient, this suggests CLIP's image prior dominates the semantic content of events for recognition; a natural next test is whether motion-sensitive tasks (gesture, direction of movement) would require restoring temporal structure.","The zero-shot consistency loss $L_{zs}$ is likely the linchpin of the reported stability; an ablation that replaces $L_{zs}$ with a simple linear-probe regularization would isolate whether text-anchoring or generic regularization prevents forgetting.","The method's reliance on a frozen CLIP image encoder means the ceiling is set by CLIP's own representation; extending the recipe to larger or newer vision-language backbones could push the reported numbers further.","Because the paper reports anomaly detection on video-extracted events without fine-tuning, a natural follow-up is to combine the event encoder with a weakly supervised temporal localization head to see whether event-only signals can match or exceed frame-based weakly supervised anomaly detection."],"forward_implications":["Event-based cameras could be used for zero-shot classification on classes the model has never seen, without any event-specific training data.","Events extracted from ordinary video, not just native event cameras, can support tasks such as anomaly detection with better accuracy than using full frames with CLIP alone.","A single trained event encoder can serve as a drop-in module for cross-modal retrieval systems, linking event data to image, text, sound, and depth.","The three-loss recipe may transfer to other sparse or low-information modalities where a pre-trained vision-language encoder needs to be adapted without forgetting.","Few-shot event recognition becomes substantially more practical, with reported gains of roughly 19 points at one shot over the prior alignment method."],"supporting_citations":[{"why":"Supplies the frozen image and text encoders whose shared embedding space is the target of transfer.","marker":"[39]"},{"why":"Provides the Lzs zero-shot consistency loss that ties event and image embeddings to text embeddings.","marker":"[63]"},{"why":"Provides the Lkl KL-divergence loss aligning event and image embedding distributions.","marker":"[54]"},{"why":"Defines the InfoNCE contrastive objective used as Lct between event and image embeddings.","marker":"[33]"},{"why":"EventBind is the main state-of-the-art baseline the method claims to beat in zero-shot, few-shot, and fine-tuning.","marker":"[66]"},{"why":"EventCLIP is the prior event-CLIP adaptation method used as a baseline in the same evaluations.","marker":"[53]"},{"why":"N-ImageNet is the principal benchmark and pre-training dataset for the event encoder.","marker":"[25]"},{"why":"ImageBind is the cross-modal hub into which the event encoder is plugged for event-sound and event-depth retrieval.","marker":"[15]"}],"fun_headline_variants":["Frozen CLIP guides event encoder to zero-shot wins","Event encoder learns, CLIP stays frozen for zero-shot","Event CLIP: +15% zero-shot over prior alignment methods","One event encoder aligns with image, text, sound, depth","CLIP's zero-shot jumps to event data with frozen guides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single-frame representation that sums all events across time and polarity into one gray-scale image is assumed to preserve enough information for CLIP's image encoder to extract the same semantic content as it would from a real image.","fun_headline_variants_meta":{"raw":{"variants":["Frozen CLIP guides event encoder to zero-shot wins","Event encoder learns, CLIP stays frozen for zero-shot","Event CLIP: +15% zero-shot over prior alignment methods","One event encoder aligns with image, text, sound, depth","CLIP's zero-shot jumps to event data with frozen guides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2939,"prompt_tokens":921,"completion_tokens":2018,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1933}},"tokens_in":537,"tokens_out":2018,"duration_ms":12782,"temperature":1.0,"reasoning_tokens":1933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:46:49.304842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three-loss recipe with an event representation that preserves temporal structure, such as splitting the stream into several short time bins or separate polarity channels, then compare zero-shot accuracy on N-ImageNet and a motion-sensitive dataset; if the single-frame aggregation is load-bearing, preserving temporal detail should not improve accuracy, but if the assumption is too strong, the temporally structured representation would outperform the reported single-frame numbers.","supporting_citations":[{"cited_title":"Preventing zero-shot transfer degradation in continual learning of vision-language mod- els","cited_arxiv_id":null,"evidence_quote":"Provides the Lzs zero-shot consistency loss that ties event and image embeddings to text embeddings."},{"cited_title":"Event camera data pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the Lkl KL-divergence loss aligning event and image embedding distributions."},{"cited_title":"Eventbind: Learning a unified representation to bind them all for event-based open-world understanding","cited_arxiv_id":null,"evidence_quote":"EventBind is the main state-of-the-art baseline the method claims to beat in zero-shot, few-shot, and fine-tuning."},{"cited_title":"N-imagenet: Towards robust, fine-grained object recognition with event cameras","cited_arxiv_id":null,"evidence_quote":"N-ImageNet is the principal benchmark and pre-training dataset for the event encoder."}],"review_version":1}