{"id":"0b57a02d-58a9-4e20-b0af-cf22c0e78143","arxiv_id":"2412.19646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Chimera uses zero-shot NAS proxies and a diversity index to search heterogeneous recurrent backbones for event cameras, reaching PEDRo mAP 64.2 with 4.9M parameters.","lead":"This paper introduces Chimera, a two-stage neural architecture search framework that mixes convolutional, transformer, state-space, and MLP blocks to build backbones for event-based object detection. On the PEDRo dataset the searched models match or approach state-of-the-art detectors while using fewer parameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection on the same PEDRo test set for benchmark calibration, hyperparameter choice, and final mAP may inflate the headline results.","rationale":"The reader's weakest_assumption precisely identified the same issue: the PEDRo test set appears to be reused for calibration, hyperparameter selection (alpha), and final architecture selection, which would invalidate the headline mAP as an unbiased estimate. My independent reading of Sections 3.5, 4.1, 4.1.2, and Algorithm 1 confirms that no validation/test split is ever mentioned, and the text repeatedly says 'mAP for the test set' is used for benchmark correlation, for choosing SHIST, for choosing W, for choosing alpha, and for selecting the final individual. This is the single most load-bearing concern because the paper's central contribution is a zero-shot NAS framework whose value depends on the discovered architectures being genuinely good without test-set peeking. If the test set was used for selection, the contribution reduces to 'we trained a bunch of models and picked the best on the test set,' which is not a valid evaluation of NAS. The concern is not about internal inconsistency in the math; it is about the evaluation protocol. The GEN1 results in the Supplementary Material provide some independent support because the architectures were not optimized on GEN1, and Chimera-3M/5M perform competitively there. That is genuine evidence that the discovered architectures are not catastrophically overfit to PEDRo. However, GEN1 was not part of the search or calibration, so it cannot validate the PEDRo-specific headline numbers. A clean re-run with a proper validation/test split, as described in the concrete test, would settle the question and is feasible given the paper's stated compute budget (1.32h search + 17.4h training per alpha setting). I therefore keep the verdict CONDITIONAL rather than moving to REJECT, because the framework itself is plausible and the GEN1 transfer is encouraging, but the PEDRo headline cannot be accepted without the split being clarified and the selection bias check performed.","tokens_in":19779,"tokens_out":1994,"duration_ms":279272,"concrete_test":"Ask the authors to provide the exact PEDRo split used for the 250-model benchmark, for Table 2 weight selection, for the alpha sweep, for Stage 2's top-5 training, and for the final reported mAP. Then re-run the full Chimera-NAS pipeline (benchmark-calibrated SHIST/weights/alpha, top-5 training, final selection) with selection decisions made only on a validation split, and evaluate the final architectures once on a test split that was never touched during search, calibration, or selection. If the final test mAP drops by more than ~2 points for Chimera-5M or Chimera-3M, the headline claim of state-of-the-art-comparable performance with a 1.6x parameter reduction is not supported for zero-shot NAS.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Chimera reaches state-of-the-art-comparable PEDRo mAP with 1.6x fewer parameters. This requires the reported mAP to be a clean, held-out evaluation. The paper does not document any such split. Section 3.5 says every benchmark model's mAP 'for the test set' was recorded after 50-epoch training on PEDRo. Section 4.1 uses those exact mAP values to (a) compare proxies and pick SHIST as the encoding, (b) select Zen/MACs weights in Table 2, and (c) calibrate alpha in Section 4.1.2. Algorithm 1's Stage 2 then trains the top-5 architectures and selects the one 'with the highest mAP' from the same PEDRo test set. If every one of these selection steps used the same test labels, the final Chimera-5M/3M numbers are the result of selection on the test set, not an unbiased estimate of a zero-shot NAS policy. The magnitude of the claimed advantage (Chimera-5M 64.2 vs ReYOLOv8s 64.5 with 1.7x fewer parameters) could be partly or wholly a selection artifact. The paper never states that a validation split exists or that the final architectures were evaluated on a held-out portion not used for any selection decision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Chimera is a two-stage, block-based neural architecture search (NAS) framework for event-based object detection. Stage 1 searches over a design space of four macro-block types (C2f, MaxViT, Mamba, WaveMLP), four event encodings (VTEI, MDES, SHIST, TAF), and architectural hyperparameters, using a fitness function that combines zero-shot proxies (Zen-Score, MACs, and initially NTK) with a newly proposed Diversity Index. Stage 2 trains the top-five ranked architectures for 100 epochs and selects the one with the highest mean average precision (mAP). The method is evaluated on the PEDRo dataset, with 250 randomly generated heterogeneous backbone benchmarks used to calibrate the encoding, proxy weights, and diversity weight. The authors report Chimera-3M at 59.5 mAP / 3.0M parameters, Chimera-5M at 64.2 mAP / 4.9M parameters, and Chimera-10M at 64.9 mAP / 7.2M parameters, comparing favorably with ReYOLOv8n and ReYOLOv8s while claiming an average 1.6x parameter reduction. Additional results on the GEN1 dataset are provided in the Supplementary Material.","tokens_in":20088,"tokens_out":7440,"duration_ms":64378,"significance":"If the results hold, Chimera would be the first NAS framework specifically for event-based object detection, and it would demonstrate that zero-shot proxy search can automatically discover compact hybrid recurrent backbones that match hand-designed event detectors. The GEN1 evaluation in the Supplementary Material is a genuine strength: architectures optimized only on PEDRo are transferred to a different dataset without any adaptation and achieve competitive mAP (e.g., Chimera-3M 44.6 vs RVT-T 44.1; Chimera-10M 47.7 vs ReYOLOv8s 48.3), which supports the broad claim that the search produces transferable architectures. However, the headline PEDRo numbers are compromised by an evaluation protocol in which the same PEDRo test labels appear to be used for proxy calibration, hyperparameter selection, and final architecture selection, with no documented validation split. This selection bias could inflate the reported margins, and the GEN1 results—while promising—are less favorable than the PEDRo numbers and are currently relegated to supplementary material.","major_comments":[{"comment":"The paper does not document any validation/test split for PEDRo, yet the same PEDRo \"test set\" appears to be used to (i) select the SHIST encoding in §4.1 (Fig. 2), (ii) calibrate the proxy weight vector W in Table 2, (iii) tune the diversity weight α in §4.1.2, and (iv) select the final architecture in Stage 2 of Algorithm 1 (\"the individual I with the highest mAP\"). Consequently, the reported PEDRo mAP values in Table 12 (e.g., Chimera-5M 64.2 vs ReYOLOv8s 64.5 at 1.7x fewer parameters) are the result of selection on the test set rather than an unbiased estimate of a zero-shot NAS policy. Please introduce a proper validation split for all calibration and model-selection steps, or explicitly state the existing split and demonstrate that none of the selection steps used the test labels. The GEN1 results in Supplementary Table 13 are the only currently unbiased held-out evidence and should be moved to the main text or otherwise clearly labeled as the external validation.","section":"§3.5, §4.1, §4.1.2, Algorithm 1"},{"comment":"The Diversity Index D(f) in Eq. (7) is not fully specified: S is described only as \"related to the maximum value that the summation term can attain,\" but the claimed endpoints (D=0 for a homogeneous architecture and D=1 for a fully diverse one) hold only for a precise choice of S, such as the maximum possible pairwise-difference sum over the block-count vector for the given design space. Because D(f) is one of the two terms in the fitness function (Eq. (8)) and because α is tuned on the basis of this index, the definition of S must be made explicit for the search objective to be reproducible.","section":"Eq. (7), §3.3.1"},{"comment":"The calibration of α in §4.1.2 selects the value that gives the \"best mAP\" after training the top-five candidates for 100 epochs on PEDRo, and the claim that α=0.05 \"surpasses the other configurations by approximately +2.0\" is made on the same test labels that are later reported as the headline results. This is a hyperparameter choice made on the test set, not a neutral comparison of search policies. Confirming the α=0.05 choice on a validation split, or reporting the sensitivity of the final architectures to α on a held-out set, is necessary before the benefit of the diversity term can be interpreted.","section":"§4.1.2, §9 (Supplementary)"}],"minor_comments":[{"comment":"There is a typo: \"ZN-NAS\" should be \"ZS-NAS\".","section":"§2.3"},{"comment":"The text says \"250 randomly generated heterogeneous models\" and then that each model was trained with VTEI, MDES, TAF, and SHIST; the Supplementary clarifies that this yields about 1,000 trained combinations. Please state this explicitly in Section 3.5 to avoid ambiguity.","section":"§3.5"},{"comment":"The reported superiority of α=0.05 is given only as \"approximately +2.0\"; please report the actual mean mAP values (and, ideally, variances) for each α value.","section":"§4.1.2"},{"comment":"The \"average parameter reduction of 1.6 times\" is computed against different baselines for different Chimera sizes (ReYOLOv8n for Chimera-3M, ReYOLOv8s for Chimera-5M, and so on); please state this comparison protocol explicitly rather than implying a single baseline.","section":"Abstract and §4.3"},{"comment":"Tables 7 and 8 appear to describe the same runtime data with near-identical titles; please merge them or clarify the distinction.","section":"Supplementary §10"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem—NAS for event-based vision—and the proposed search space, diversity index, and two-stage search protocol are of interest to the community. My main concern is the evaluation protocol: as written, the PEDRo test set is used for nearly every decision in the pipeline, which risks optimistically biased headline numbers. The GEN1 external evaluation in the Supplementary is the strongest evidence for the approach's generality and should be given a more prominent role, either as a companion experiment in the main text or as the basis for reframing the central claim. I would also encourage the authors to release code for reproducibility, given that zero-shot NAS work is often judged on the exactness of the proxy computations and the design-space implementation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the first NAS framework aimed at event-based object detection, and the core machinery is imported from RGB zero-shot NAS. The result is a credible demonstration that a hybrid recurrent backbone can be found cheaply and match hand-designed detectors on PEDRo, with GEN1 in the supplementary giving a genuine out-of-distribution check. Credit where due: the paper actually trains hundreds of benchmark models, reports baselines, and publishes detailed architecture tables. That is more than many NAS papers do.\n\nThe soft spot is the evaluation protocol. Section 3.5 records mAP on the PEDRo test set for 250 benchmark models; Section 4.1 uses those numbers to choose SHIST, the proxy weights, and alpha; Algorithm 1 Stage 2 then trains the top-5 and picks the one with the highest mAP. No validation split is documented anywhere. If the same test labels drive calibration, hyperparameter selection, and final architecture selection, the headline margins over ReYOLOv8s may be partly selection artifact. That is a real flaw, not a nitpick. The fix is straightforward: split PEDRo into train/val/test, do all selection on val, report once on test, and release the split.\n\nTwo smaller issues. First, at alpha=0.05 the fitness function in Eq. 8 is 95% diversity and 5% proxy score. The paper frames this as zero-shot NAS, but the search is effectively a diversity-maximizing random search with proxies as a tie-break. That is fine as a finding, but it undercuts the claim that Zen-Score/MACs are what drive the results. Second, no error bars or multiple seeds anywhere; given the small dataset and the selection procedure, that matters.\n\nThe GEN1 results are the strongest evidence the approach generalizes, and they are honestly placed in the supplementary. But they do not rescue the PEDRo mAP from selection bias.\n\nWho this is for: event-vision researchers who want a cheap way to explore hybrid backbones, and NAS people interested in a new application domain. It deserves a serious referee, but the referee should demand a clean validation split, seeded runs, and code before accepting the performance claims.","headline":"Useful first NAS for event-based detection, but the PEDRo evaluation is compromised by selecting on the same test set used for calibration.","tokens_in":20592,"tokens_out":2248,"would_cite":false,"duration_ms":29221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chimera claims a two-stage, zero-shot neural architecture search can automatically discover compact hybrid recurrent backbones for event-based object detection, matching state-of-the-art detectors on the PEDRo dataset with an average 1.6x…","keywords":["neural architecture search","event-based vision","object detection","zero-shot NAS","hybrid architectures","recurrent backbone","PEDRo dataset","diversity index"],"falsifier":"Train an equal number of randomly sampled architectures from Chimera's design space under the same 100-epoch budget on PEDRo; if the random architectures match Chimera-5M's 64.2 mAP, the zero-shot search adds nothing beyond the design space itself. Alternatively, re-running the search with the PEDRo test set withheld until after Stage 2 selection should reproduce the reported margin over ReYOLOv8s; if the margin shrinks to noise, the headline result is selection bias.","tokens_in":19591,"feed_emoji":"👁️","tokens_out":4964,"duration_ms":41628,"temperature":0.7,"pith_summary":"The paper claims that a neural architecture search framework built on zero-shot proxies can automatically assemble compact hybrid backbones for event-based object detection, replacing hand-designed architectures. It introduces Chimera, whose design space mixes four processing paradigms—convolutions, transformers, MLP-mixers, and state-space models—over four backbone layers plus a recurrent memory cell, and searches roughly 20,000 combinations. On the PEDRo person-detection dataset, the architectures found by the search match the accuracy of ReYOLOv8 models while using about 1.6x fewer parameters on average, and the discovered layouts transfer to the GEN1 automotive dataset without re-tuning. The intended contribution is a systematic, low-cost route from RGB-era detector designs to event-camera hardware.","feed_headline":"Search shrinks event-camera detectors 1.6x at same accuracy","feed_subtitle":"Chimera's hybrid backbones match hand-designed YOLO detectors on PEDRo with fewer parameters.","key_machinery":"The load-bearing mechanism is the two-stage Chimera-NAS algorithm. Stage 1 uses an evolutionary search over a design space of four event encodings, a STEM channel count, and four Chimera layers each choosing among C2f, MaxViT, Mamba, and WaveMLP blocks, scored by a fitness function $F = \\alpha W \\cdot Z(f) + (1-\\alpha) D(f)$ that combines Zen-Score and MACs proxies with a diversity index $D(f)$ under a parameter budget. Stage 2 trains the five highest-scoring candidates for 100 epochs and selects the one with the highest mean Average Precision. The search is carried by the zero-shot proxies: the Zen-Score, which estimates expressive capacity from a finite-difference gradient surrogate, combined 0.6/0.4 with MACs after correlation analysis on a 250-model benchmark, while the diversity index prevents the proxy bias (which favors WaveMLP) from collapsing the population into a single block type.","core_discovery":"Chimera's central claim is that a two-stage search—stage one ranking candidates with zero-shot proxies (Zen-Score, MACs, and a diversity index), stage two training the top five for 100 epochs—can find heterogeneous recurrent backbones that perform comparably to state-of-the-art event-based detectors at lower parameter counts. The strongest reported result is Chimera-5M reaching 64.2 mAP with 4.9M parameters, close to ReYOLOv8s at 64.5 mAP with 8.4M parameters, and Chimera-3M at 59.5 mAP with 3.0M parameters, slightly above ReYOLOv8n at 59.0 mAP with 4.7M parameters. The authors also show the found architectures generalize: trained from scratch on GEN1, Chimera-3M/5M/10M are competitive with similarly scaled models even though all search decisions were made on PEDRo.","pith_inferences":["If the selection-bias caveat is controlled, the same two-stage recipe could be applied to other sensor domains where labeled data is scarce, such as radar or lidar detection, treating any dense encoding plus a hybrid backbone library as the search space.","The diversity index could serve as a general regularizer in zero-shot NAS to counter proxy bias; the paper itself observes that without it (α ≥ 0.4) the population collapses to all-WaveMLP architectures.","The reported 1.6x average parameter reduction is driven by the 3M and 5M models; at 10M the advantage over ReYOLOv8s essentially disappears (64.9 vs 64.5 mAP, 7.2M vs 8.4M params), so the efficiency claim is scale-dependent.","A stronger demonstration would be to run the search directly on GEN1 and compare the found architectures against the PEDRo-found ones, which would isolate whether the proxy ranking transfers or merely the architecture family transfers."],"forward_implications":["Architectures found by Chimera match hand-designed recurrent YOLO detectors on PEDRo with roughly 1.6x fewer parameters on average (59.5 mAP at 3.0M params vs 59.0 at 4.7M; 64.2 at 4.9M vs 64.5 at 8.4M).","The discovered layouts consistently place MaxViT early for global context, C2f for local extraction, Mamba for spatio-temporal dependencies, and WaveMLP late, suggesting a reusable ordering rule for hybrid event backbones.","The same searched architectures, trained from scratch on GEN1 without any search re-tuning, stay competitive with similarly sized models, showing that the PEDRo-calibrated search transfers to another event dataset.","The proxy calibration step identifies Zen-Score + MACs (0.6/0.4 weights) as the best cheap ranking signal on the 250-model benchmark, and NTK as too costly and negatively correlated, so future searches can drop NTK.","The whole first-stage search of about 1,050 models runs in roughly 1.3 hours on one GPU, plus 17.4 hours to fine-tune the top five candidates for 100 epochs each."],"supporting_citations":[{"why":"Supplies the ReYOLOv8 recurrent backbone structure, training hyperparameters, detection head design, and the direct baselines (ReYOLOv8n/s/m) that Chimera is measured against.","marker":"[10]"},{"why":"Supplies the Zen-Score zero-shot proxy that dominates the Chimera fitness function with weight 0.6.","marker":"[47]"},{"why":"Provides the PEDRo dataset used for the benchmark, the search calibration, and the final evaluation.","marker":"[59]"},{"why":"Supplies the MaxViT block configuration used in the Chimera library and the RVT transformer+RNN baselines.","marker":"[11]"},{"why":"Supplies the YOLOv8 detection head, PANET fusion, SPPF block, and training framework that Chimera inherits.","marker":"[48]"},{"why":"Provides the NTK condition-number proxy that was evaluated and then excluded because of negative correlation and high cost.","marker":"[58]"},{"why":"Supplies the Mamba state-space block included in the Chimera design space.","marker":"[38]"},{"why":"Supplies the WaveMLP block included in the Chimera design space and noted to be favored by the Zen-Score proxy.","marker":"[54]"},{"why":"Provides the zero-shot NAS proxy framework and the Kendall/Spearman correlation methodology used to select the search metrics.","marker":"[57]"}],"fun_headline_variants":["NAS finds event-camera detectors 1.6x smaller, same accuracy","Chimera shrinks event detectors 1.6x while keeping accuracy","Event-camera detection: NAS finds 1.6x leaner models, same mAP","Chimera: NAS shrinks event detectors 1.6x, accuracy intact","Automated search yields event detectors with 1.6x fewer params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the PEDRo test-set mAP used to calibrate the search proxies and to select the final architecture is a genuinely held-out signal, not reuse of the same labels that guide the search.","fun_headline_variants_meta":{"raw":{"variants":["NAS finds event-camera detectors 1.6x smaller, same accuracy","Chimera shrinks event detectors 1.6x while keeping accuracy","Event-camera detection: NAS finds 1.6x leaner models, same mAP","Chimera: NAS shrinks event detectors 1.6x, accuracy intact","Automated search yields event detectors with 1.6x fewer params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3356,"prompt_tokens":888,"completion_tokens":2468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":2362}},"tokens_in":504,"tokens_out":2468,"duration_ms":23051,"temperature":1.0,"reasoning_tokens":2362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:01:46.030759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an equal number of randomly sampled architectures from Chimera's design space under the same 100-epoch budget on PEDRo; if the random architectures match Chimera-5M's 64.2 mAP, the zero-shot search adds nothing beyond the design space itself. Alternatively, re-running the search with the PEDRo test set withheld until after Stage 2 selection should reproduce the reported margin over ReYOLOv8s; if the margin shrinks to noise, the headline result is selection bias.","supporting_citations":[{"cited_title":"A Recurrent YOLOv8-based framework for Event-Based Object Detection","cited_arxiv_id":"2408.05321","evidence_quote":"Supplies the ReYOLOv8 recurrent backbone structure, training hyperparameters, detection head design, and the direct baselines (ReYOLOv8n/s/m) that Chimera is measured against."},{"cited_title":"Zen-nas: A zero-shot nas for high-performance image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the Zen-Score zero-shot proxy that dominates the Chimera fitness function with weight 0.6."},{"cited_title":"Pedro: an event-based dataset for person detection in robotics","cited_arxiv_id":null,"evidence_quote":"Provides the PEDRo dataset used for the benchmark, the search calibration, and the final evaluation."},{"cited_title":"Recurrent vision transformers for object detection with event cameras","cited_arxiv_id":null,"evidence_quote":"Supplies the MaxViT block configuration used in the Chimera library and the RVT transformer+RNN baselines."},{"cited_title":"An image patch is a wave: Phase-aware vision mlp","cited_arxiv_id":null,"evidence_quote":"Supplies the WaveMLP block included in the Chimera design space and noted to be favored by the Zen-Score proxy."},{"cited_title":"Zero-shot neural architecture search: Challenges, solutions, and opportunities","cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot NAS proxy framework and the Kendall/Spearman correlation methodology used to select the search metrics."}],"review_version":1}