{"id":"e219d812-383a-4f49-b721-952b5fa87331","arxiv_id":"2504.12515","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A deep-feature distance, EQS, ranks event-camera simulators by realism, and in a three-simulator DSEC test higher EQS aligns with higher real-world detection mAP.","lead":"This paper introduces EQS, a metric that scores how closely a simulated event-camera stream matches a real one by comparing deep-network features. In tests on three simulators over a driving dataset, higher EQS coincided with better real-world object detection after training on simulated data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Three-point correlation and shared RVT architecture do not yet establish the claimed monotonic link between EQS and sim-to-real generalization.","rationale":"The paper's proposal is coherent: a differentiable, task-agnostic distance between event streams computed from a pretrained recurrent vision transformer's early convolutional activations, with code released. The experiments are internally consistent: across the three simulators, the EQS ordering (ESIM > V2E > Pix2NVS) matches the real-DSEC mAP ordering (18.6 > 14.3 > 5.4), and the qualitative examples align. The central claim, however, is an implication ('higher EQS implies improved generalization'), and the support is three data points from a single architecture family and a single real dataset. With n=3, the probability of a perfect rank match under random ordering is 1/6; the paper reports no error bars, no per-simulator mAP variance, and no statistical test. Per-sequence EQS values in Table 2 overlap across simulators, so the separability of the averages is not established. The potential confound that the feature extractor and downstream detector are both RVT-based strengthens the need for expanded validation; a simulator could be close in RVT activation space simply because the pretrained RVT is sensitive to event-count or noise statistics, without that implying better generalization for other detectors or tasks. This does not invalidate the metric as a heuristic, but it does mean the abstract's universal phrasing outruns the evidence. The reader's conditional verdict is the right one: the paper should be accepted only with the claim scoped to the tested setting, or with additional validation. Hence verdict_should_be remains UNCHANGED relative to the reader's CONDITIONAL.","tokens_in":10747,"tokens_out":5337,"duration_ms":57151,"concrete_test":"Run the same DSEC protocol on at least three additional simulators from Table 1 (e.g., EventGAN, DVS-Voltmeter, ICNS, or ESIM with varied noise parameters), computing per-sequence EQS distributions and training RVT-small on each simulated set to obtain real-DSEC mAP. If the rank correlation between simulator-average EQS and real mAP is not monotone across the expanded set, or if per-sequence EQS intervals overlap so heavily that the simulator-level averages are not statistically distinct, the central claim is unsupported. As a complementary check, replace the downstream detector with a non-RVT event detector while keeping the EQS feature extractor fixed; the EQS ranking should still predict mAP if the metric captures a general sim-to-real gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that 'a higher EQS implies improved generalization to real-world data after training on simulated events.' The evidence is Table 2 versus Table 3: three simulators, one downstream architecture (RVT-small), one real dataset (DSEC). The three observed points are monotone (ESIM EQS 0.828 -> real mAP 18.6; V2E 0.758 -> 14.3; Pix2NVS 0.599 -> 5.4), but with n=3 a random permutation would match with probability 1/6, and no error bars, confidence intervals, or significance tests are reported. Per-sequence EQS values in Table 2 overlap substantially across simulators (e.g., V2E 0.66-0.83, ESIM 0.72-0.91), so the simulator-level averages may not be statistically separable. Additionally, the metric is computed with a pretrained RVT while the downstream model is RVT-small; the correlation may reflect RVT-specific sensitivity to event statistics such as density, noise, or thresholding rather than a generic sim-to-real gap. Section 4.1 states the hypothesis explicitly, but the validation does not rule out a three-point coincidence or an architecture-family confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Event Quality Score (EQS), a differentiable metric that compares two raw event streams by computing cosine distances between activations of the first three convolutional blocks of a pretrained recurrent vision transformer (RVT). The authors hypothesize that the sim-to-real gap in event-camera simulation can be captured in this latent feature space. They compute EQS between event streams generated by three simulators (ESIM, V2E, PIX2NVS) and real DSEC data, and compare the resulting scores with the real-world mAP of an RVT-small detector trained on each simulator's output. The observed ordering (ESIM > V2E > PIX2NVS in both EQS and mAP) is presented as evidence that higher EQS implies improved sim-to-real generalization.","tokens_in":11023,"tokens_out":5612,"duration_ms":53455,"significance":"The idea of transferring perceptual metrics such as LPIPS to raw event streams is novel and timely. If the correlation were rigorously established, EQS would be a valuable tool for simulator development, providing a task-independent and differentiable objective. The paper is clearly written, the metric is simple to implement, and the code is released at a public repository. However, the current evidence for the central claim is limited to three simulators, a single downstream architecture, and a single real dataset, without statistical analysis; the significance of the claimed correlation is therefore not yet established at a level appropriate for a journal.","major_comments":[{"comment":"The central claim that 'a higher EQS implies improved generalization to real-world data' is supported only by three simulator-level data points. No error bars, confidence intervals, correlation coefficient, or significance test are reported. The per-sequence EQS values in Table 2 overlap substantially across simulators (e.g., V2E ranges 0.66–0.83 and ESIM 0.72–0.91), so the average differences may not be statistically reliable. With only three points, the probability that a random ordering would match is 1/6. The authors should report variance across sequences, compute a rank or Pearson correlation with uncertainty, and preferably include additional simulators (e.g., Vid2E, EventGAN, DVS-Voltmeter) to strengthen the claim.","section":"§5.1, Tables 2 and 3"},{"comment":"The metric uses RVT activations, and the downstream generalization test uses RVT-small. This shared architecture is a confound: the observed correlation might reflect RVT-specific sensitivity to event statistics such as density or noise rather than a generic property of simulation fidelity. To support the abstract's general claim, the authors should evaluate EQS with an independent downstream architecture (for example, a non-recurrent event detector) or otherwise demonstrate that the correlation is not an artifact of using the same model family.","section":"§4.1 and §5.1"},{"comment":"The metric's definition leaves key hyperparameters unspecified or unjustified: the number of temporal bins T and the time interval δt are not reported, and the choice of the first three convolutional blocks and the 3×3 patch size is stated but not ablated. The paper says the patch size is 'discussed in Sec. 5,' but Sec. 5 only describes the chosen value. Since EQS values and hence the claimed correlation may depend on these choices, a sensitivity analysis is needed to ensure the metric is robust and the reported values are reproducible.","section":"§4 and §5"}],"minor_comments":[{"comment":"The definition of V_ei is unclear: the text says 'such that ||V_ei|| is the number of channels', which is inconsistent with treating V_ei as a vector. Please clarify what is averaged over the patch and what the norm sign refers to.","section":"§4.1, equation for V_ei"},{"comment":"The equation for CS_pi is missing a closing norm bar: it should be ||V_e2||.","section":"§4.1, equation for CS_pi"},{"comment":"The value of T (number of temporal bins) and δt used for the event tensor representation are not given; please specify them for reproducibility.","section":"§5"},{"comment":"'5 randomly chosen sequences' without a seed makes the selection non-reproducible; specify the seed or the sequence identifiers.","section":"§5.1, Table 2"},{"comment":"The conclusion that EQS 'can potentially be used as a loss function' is speculative; no experiment optimizes a generator against EQS. If this claim is retained, a proof-of-concept would be needed.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a workshop contribution, and the core idea is worth pursuing. For a journal venue, however, the validation is too thin: the central correlation claim needs more simulators, statistical grounding, and an ablation of the architectural confound. I would encourage the authors to extend the evaluation rather than resubmit with minor edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read for you. The paper proposes EQS, a perceptual-quality metric for raw event streams: convert events to tensors, run them through a pretrained RVT, and average cosine distances between conv activations. It is an LPIPS-style transfer to event data, and as far as I can tell it is the first metric that operates directly on raw events without an intermediate frame representation. That is a genuinely useful contribution, and the authors have released code.\n\nThe good news: the idea is simple, differentiable, and cheap. The paper's internal sanity check looks fine — ESIM gets the highest EQS (0.828) and the best real-world mAP (18.6), followed by V2E and PIX2NVS. So the metric ranks the three simulators in the same order as the downstream detection result. That is a reasonable first demonstration.\n\nThe soft spots are real and mostly about evidence, not about the idea. The central claim — 'a higher EQS implies improved generalization to real-world data' — is supported by exactly three simulators. No error bars, no correlation coefficient, no significance test. With n=3, any monotone ordering has a 1/6 chance of matching by chance alone, and the per-sequence EQS values overlap between simulators (V2E ranges 0.66–0.83, ESIM 0.72–0.91). So the average differences may not be statistically meaningful.\n\nThere is also an architecture confound: the metric uses a pretrained RVT, and the downstream detector is RVT-small. You could be measuring RVT-specific sensitivity to event statistics (noise level, polarity balance, event density) rather than a generic sim-to-real gap. The paper does not test a different feature extractor or a different downstream model. The choice of the first three conv blocks and 3×3 patches is not justified against alternatives, and the authors freely admit that the distance metric choice and a comprehensive analysis are out of scope.\n\nNone of this kills the paper. It is a workshop-quality proposal with an honest statement of its hypothesis and limits. But the abstract's claim overshoots the data. My recommendation: send it to serious peer review, but the reviewers should ask for either a much larger simulator panel (and a regression or rank statistic with confidence), or a softened claim. The idea is worth engaging with; the evidence is not yet conclusive.\n\nRegards.","headline":"EQS is a neat, useful idea and a credible first cut, but the central correlation claim is supported by only three simulators with a shared RVT architecture, so treat it as suggestive, not proven.","tokens_in":11531,"tokens_out":2917,"would_cite":true,"duration_ms":30848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces EQS, a differentiable metric for raw event streams, and shows that higher EQS predicts better object-detection accuracy after training on simulated data.","keywords":["event quality score","event camera simulation","sim-to-real gap","recurrent vision transformer","latent feature distance","object detection","event streams","DSEC"],"falsifier":"Compute EQS and real-data mAP for a fourth, untested simulator: if its EQS is higher than ESIM's but its real mAP is lower, the claimed monotonic link is false. A cheaper version of the same test is to train the detector on two simulators with nearly equal EQS; if their real mAP differs sharply, EQS is not resolving the gap.","tokens_in":10583,"feed_emoji":"⚡","tokens_out":6481,"duration_ms":62947,"temperature":0.7,"pith_summary":"Event cameras produce labeled data slowly and expensively, so many research pipelines train on simulated event streams; the paper argues that the field has lacked a quantitative way to tell whether a simulator produces events that resemble real sensor output. It introduces EQS, a score computed by passing a real stream and a simulated stream through a pretrained recurrent vision transformer and measuring the average cosine distance between convolutional activations. On the DSEC driving dataset, the paper shows that EQS ranks three simulators in the same order as the real-world object-detection accuracy of a model trained on each simulator's output. If the ranking is predictive in general, EQS gives simulator developers a differentiable, task-independent target for closing the sim-to-real gap.","feed_headline":"Higher EQS scores predict sim-to-real transfer in event cameras","feed_subtitle":"A single number from RVT activations ranks ESIM, V2E, and PIX2NVS in the same order as their real-data detection mAP.","key_machinery":"The machinery is a latent feature similarity block built on the recurrent vision transformer (RVT), a network that mixes spatial transformer layers with recurrent temporal layers and is pretrained for event-camera object detection. For each of the first three RVT scales, the block averages activations over non-overlapping 3x3 spatial patches to form per-channel vectors, computes the cosine distance between vectors from the real and simulated streams, and averages these distances across patches and scales. This makes EQS a fully differentiable distance between two raw event streams, computed without converting events to an image-like frame first.","core_discovery":"The central claim is that a higher EQS implies improved generalization to real-world data after training on simulated events. Concretely, for the ESIM, V2E, and PIX2NVS simulators, EQS averaged over the DSEC dataset gives 0.866, 0.773, and 0.501 respectively, while an RVT-small detector trained on each simulator's output reaches 18.6, 14.3, and 5.4 mAP on the real DSEC test set. The paper interprets this as evidence that the distance between simulated and real activations in the representation space of a detection-pretrained recurrent vision transformer captures the simulation gap, and that EQS can therefore rank simulators and guide their improvement.","pith_inferences":["If the correlation is causal, minimizing EQS as a training signal should improve transfer for detectors other than RVT-small; that is an extension the paper motivates but does not test.","Because the feature extractor is pretrained for object detection, EQS may be biased toward object-centric realism; simulators that score high could still be poor for motion estimation or depth tasks, so a task-specific variant may be needed.","An unpaired version of EQS, comparing marginal activation statistics over many scenes rather than paired same-scene streams, would test whether EQS measures general realism rather than the simulator's ability to reproduce one scene's events.","EQS could be used as a screening tool to pick which simulated sequences to train on, or as a regularization term during detector training to keep features close to the real-data manifold."],"forward_implications":["Simulator developers gain a single number that predicts whether models trained on their output will transfer to real sensors, replacing expensive train-and-evaluate cycles with a cheap feature-space comparison.","Because EQS is differentiable, it can be used as a loss when training a simulator or an event-generation network, turning 'make the stream more realistic' into an objective with a gradient.","The metric works on raw event streams of arbitrary length, so it can be applied to new simulators and new sensor resolutions without training a task model.","The ranking produced by EQS (ESIM above V2E above PIX2NVS) is consistent with the measured sim-to-real mAP ordering, supporting the claim that higher EQS implies better generalization."],"supporting_citations":[{"why":"Supplies the pretrained recurrent vision transformer whose convolutional activations define the EQS feature space.","marker":"[11]"},{"why":"Establishes the deep-feature perceptual-distance approach that EQS adapts from images to raw event streams.","marker":"[47]"},{"why":"Provides the real-world DSEC driving event streams used for EQS comparison and for measuring sim-to-real mAP.","marker":"[12]"},{"why":"PIX2NVS is one of the three simulators compared; it gets the lowest EQS and the lowest real mAP.","marker":"[3]"},{"why":"V2E is one of the three simulators compared; it gets an intermediate EQS and real mAP.","marker":"[16]"},{"why":"ESIM is one of the three simulators compared; it gets the highest EQS and the best real mAP.","marker":"[38]"}],"fun_headline_variants":["EQS metric ranks event camera simulators by real-world transfer","Latent-space score predicts sim-to-real success for event data","Event quality score gauges the realism of simulated event streams","One score predicts how well simulated events transfer to real cameras","RVT-based EQS measures the simulation gap in event cameras"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the sim-to-real gap is visible in the average cosine distance between RVT activations of simulated and real streams, so that a higher EQS reliably means better downstream transfer; the paper checks this on one detector architecture and three simulators.","fun_headline_variants_meta":{"raw":{"variants":["EQS metric ranks event camera simulators by real-world transfer","Latent-space score predicts sim-to-real success for event data","Event quality score gauges the realism of simulated event streams","One score predicts how well simulated events transfer to real cameras","RVT-based EQS measures the simulation gap in event cameras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2799,"prompt_tokens":905,"completion_tokens":1894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1810}},"tokens_in":521,"tokens_out":1894,"duration_ms":13159,"temperature":1.0,"reasoning_tokens":1810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:29:18.625453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute EQS and real-data mAP for a fourth, untested simulator: if its EQS is higher than ESIM's but its real mAP is lower, the claimed monotonic link is false. A cheaper version of the same test is to train the detector on two simulators with nearly equal EQS; if their real mAP differs sharply, EQS is not resolving the gap.","supporting_citations":[{"cited_title":"Recurrent vision transformers for object detection with event cameras, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained recurrent vision transformer whose convolutional activations define the EQS feature space."},{"cited_title":"Efros, Eli Shecht- man, and Oliver Wang","cited_arxiv_id":null,"evidence_quote":"Establishes the deep-feature perceptual-distance approach that EQS adapts from images to raw event streams."},{"cited_title":"Dsec: A stereo event camera dataset for driving scenarios, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the real-world DSEC driving event streams used for EQS comparison and for measuring sim-to-real mAP."},{"cited_title":"Pix2nvs: Parameterized conversion of pixel-domain video frames to neuromorphic vision streams","cited_arxiv_id":null,"evidence_quote":"PIX2NVS is one of the three simulators compared; it gets the lowest EQS and the lowest real mAP."},{"cited_title":"v2e: From video frames to realistic dvs events, 2021","cited_arxiv_id":null,"evidence_quote":"V2E is one of the three simulators compared; it gets an intermediate EQS and real mAP."},{"cited_title":"Esim: an open event camera simulator","cited_arxiv_id":null,"evidence_quote":"ESIM is one of the three simulators compared; it gets the highest EQS and the best real mAP."}],"review_version":1}