{"id":"341b47fe-c128-495a-b72a-ca25ff2731ba","arxiv_id":"2411.10504","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"USP-Gaussian jointly optimizes spike-to-image reconstruction, camera poses, and 3D Gaussian Splatting, reducing cascaded errors and improving 3D reconstruction quality on synthetic and real-world spike-camera data.","lead":"This paper combines three steps needed to turn spike-camera streams into 3D scenes into one jointly trained system: image reconstruction, camera pose correction, and Gaussian splatting. On synthetic data it reports better reconstruction quality than previous cascaded methods, and it keeps much of that advantage when initial camera poses are noisy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linear SE(3) trajectory model in Eq. (10) is the load-bearing assumption: with only start/end poses per spike interval, non-constant-velocity motion breaks temporal alignment in the joint loss and can corrupt both branches.","rationale":"The reader's weakest_assumption points to the same Eq. (10) trajectory model, and I agree that it is the most load-bearing element: the joint optimization claim requires that the rendered and reconstructed sequences correspond to the same camera poses at the same times. I would sharpen the concern: it is not only about real-world hand-shake being non-linear, but also about the internal mechanism of Ljoint. With only start/end poses per interval, any non-constant-velocity motion cannot be represented, so the joint loss will align the two branches at geometrically inconsistent poses. The flip-min operation only fixes temporal order reversal, not non-linear velocity. The paper provides no real-world pose-error metrics and no ablation against a higher-order trajectory model, so the pose-correction contribution is not yet validated in the regime it targets. This does not overturn the central claim, because the synthetic experiments and ablations (Tabs. 1, 2, 4) are consistent with the proposed framework and show meaningful improvements. However, it does reinforce the need for the conditions already stated in the reader's verdict: released code/data and additional validation on real or realistically non-linear trajectories before full acceptance. Thus the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":12736,"tokens_out":9938,"duration_ms":109664,"concrete_test":"Run a controlled synthetic experiment with known ground-truth non-linear trajectories: for each spike interval, define T(t)=T_start*exp(f(t)*log(T_start^{-1} T_end)) with f(t)=(t/T)^2 (accelerated motion) or a sinusoidal perturbation, simulate spike streams, and run USP-Gaussian. Compare the optimized start/end poses and interpolated poses against ground truth using the Tab. 3 MAE metric, and compare reconstruction PSNR against (a) the linear model in Eq. (10) and (b) a variant with per-frame poses plus a smoothness prior. A substantial degradation in pose MAE or PSNR would show the linear trajectory assumption is load-bearing for the pose-correction and joint-alignment claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that jointly optimizing Recon-Net, poses, and 3DGS mitigates cascading errors—depends on the temporal alignment of the two image sequences in the joint loss (Eq. 11). Alignment is provided by the camera trajectory model of Eq. (10), which represents the pose at any time inside a spike interval by linear interpolation in the Lie algebra between T_start and T_end. This is a constant-velocity geodesic model. It has only two degrees of freedom per interval and cannot represent accelerated or otherwise non-linear hand-shake motion. The flip-and-min operation in Eq. (12) only resolves the binary forward/reverse ambiguity; it does not correct for non-linear velocity profiles. If the true trajectory within T is not a geodesic, the rendered 3DGS frames Igs(tm) are evaluated at the wrong camera poses, so Ljoint forces the Recon-Net outputs toward those wrong poses (and the 3DGS toward the Recon-Net's possibly compensating errors) rather than toward the true scene. The real-world dataset in Sec. 5.1 is exactly a fast hand-shaken capture, yet the paper reports only qualitative real-world comparisons and no pose-error or trajectory-error metric there; Tab. 3 pose errors are only for synthetic perturbed poses. The limitation statement in Sec. 6 mentions only training time and memory, not this model risk. Therefore the claim that pose correction transfers to arbitrary real-world motions is not currently supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes USP-Gaussian, an end-to-end framework that jointly optimizes a spike-to-image reconstruction network (Recon-Net), camera poses, and 3D Gaussian Splatting for 3D reconstruction from spike-camera data. The authors argue that the standard three-stage cascade (reconstructing images from spikes, estimating poses, then fitting a 3D representation) suffers from error accumulation, and they replace it with a unified loss combining a multi-reblur reconstruction loss, a 3DGS reblur loss supervised by the physically derived long-exposure signal E(T), and a joint alignment loss between the rendered and reconstructed image sequences. Experiments on a synthetic dataset report improved PSNR/SSIM/LPIPS over cascaded baselines and over SpikeGS, and a synthetic pose-perturbation experiment shows reduced translation and rotation errors. Real-world captures are compared only qualitatively.","tokens_in":12989,"tokens_out":6992,"duration_ms":63833,"significance":"If the experimental claims hold up, the paper makes a useful contribution: it shows that spike-to-image reconstruction and 3D scene representation can be trained jointly without ground-truth sharp images, and that the multi-view consistency of 3DGS can regularize a self-supervised deblurring network. The physically derived E(T) anchors the loss externally and is not circular, and the ablation study is informative. However, the significance is currently bounded by an asymmetric comparison with SpikeGS and by the lack of quantitative real-world evidence for the pose-correction claim; these issues directly affect the strength of the central conclusion.","major_comments":[{"comment":"The camera trajectory model represents the pose at time tm by linear interpolation between start and end poses in the SE(3) Lie algebra, which corresponds to constant-velocity geodesic motion. The real-world capture in Sec. 5.1 is described as a fast hand-shaken sequence; for such motion, non-constant velocity or non-geodesic rotations will make the rendered Igs(tm) in Eq. (11) correspond to incorrect camera poses. The flip-and-min operation in Eq. (12) only resolves the forward/reverse ambiguity and does not correct for a nonlinear velocity profile. Table 3 reports pose errors only for synthetic perturbed poses, and Sec. 6 lists only training time and memory as limitations. Please either justify the constant-velocity assumption for the target capture motions (e.g., with a velocity-profile analysis of the real spike data) or adopt a more expressive trajectory model, and provide quantitative pose or trajectory error evaluation on the real-world data.","section":"Sec. 4.3, Eq. (10)"},{"comment":"The SpikeGS baseline is run without its exposure loss, because that loss requires several poses per spike stream, while USP-Gaussian retains long-exposure supervision through Lgs and Lrec. Removing a central component of SpikeGS likely weakens the baseline, so the comparison is not apples-to-apples and the reported gaps (e.g., about 0.7 dB PSNR over SpikeGS in Table 1) may overstate the improvement. Please run SpikeGS with its exposure loss (using COLMAP poses or the optimized poses where needed), or quantify how much of the gap comes from the removed loss by also ablating the corresponding supervision from USP-Gaussian.","section":"Sec. 5.2, Tables 1 and 2"},{"comment":"All quantitative results are reported as single-run metrics without standard deviations or multiple seeds. Because 3DGS optimization involves stochastic initialization and adaptive density control, and the pose optimization is coupled to both branches, it is unclear whether the improvements over baselines are statistically significant. At minimum, the main comparisons in Tables 1 and 2 and the pose-error results in Table 3 should include mean and standard deviation over several runs, or an error-bar analysis.","section":"Tables 1-4 and Table 3"},{"comment":"The real-world evaluation is only qualitative (Fig. 5); there are no quantitative reconstruction metrics, no pose or trajectory errors, and no comparison with methods on the real captures. Given that the paper's contribution is specifically motivated by inaccurate real-world poses, this is a major evidence gap. Please add quantitative real-world evaluation (e.g., reconstruction metrics against a reference, or relative pose accuracy if ground truth is unavailable), or explicitly restrict the pose-correction claim to the synthetic setting.","section":"Sec. 5.1 and Sec. 5.2, real-world dataset"}],"minor_comments":[{"comment":"The abstract contains the typo 'casecaded' and the introduction uses 'casecading'; both should be 'cascaded' and 'cascading'.","section":"Abstract and Sec. 1"},{"comment":"The text says 'The multi-reblur loss Lrec in Eq. (7) is designed to substitute the single-reblur loss Lrec in Eq. (8)', but Eq. (7) defines the single-reblur loss and Eq. (8) defines the multi-reblur loss; the equation references are reversed.","section":"Sec. 5.3, paragraph II"},{"comment":"The notation Lrec is used for both the single-reblur loss and the multi-reblur loss; please use distinct symbols (e.g., Lrec^single and Lrec^multi) to avoid the confusion visible in Table 4 and the Sec. 5.3 discussion.","section":"Sec. 4.1 and Fig. 3"},{"comment":"For reproducibility, please report the number of viewpoints, the spike-window length, and the temporal sampling details for both the synthetic and real-world datasets; the current text only states that 97 spike frames are extracted per view.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the proposed joint-optimization idea is timely, but the evaluation currently favors the proposed method through an asymmetric SpikeGS baseline and lacks real-world quantitative validation. The trajectory-model concern is fixable in revision and should not by itself cause rejection. I see no novelty or attribution problem: the multi-reblur loss is clearly tied to previous work, and the pose interpolation follows BAD-NeRF/BAD-Gaussian; the novelty lies in combining these with a spike reconstruction network."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen et al. put together a one-stage pipeline for spike-camera 3D reconstruction, and on synthetic scenes it actually works: the numbers in Table 1 beat the cascaded baselines, and the ablations in Table 4 show the joint loss is doing something. The flip-and-minimum alignment to fix temporal-order ambiguity is a small but real trick. The rest—multi-reblur self-supervision, SE(3) interpolation, long-exposure supervision—comes from earlier work, including the authors' own, and they say so.\n\nThe main soft spot is the real-world story. The comparison with SpikeGS on real data is not apples-to-apples: SpikeGS runs without its exposure loss while USP-Gaussian keeps long-exposure supervision. That's a real handicap for the baseline. Also, every metric in the paper is from a single run—no error bars, so the reported gains could be noise. And the linear SE(3) trajectory model (Eq. 10) is a genuine assumption. The real-world dataset is exactly a fast shake, and the paper only shows qualitative results there, with no pose-error or trajectory-error numbers. So the claim that pose correction carries over to arbitrary hand motion is simply not demonstrated. The limitation section doesn't mention this, only training time and memory.\n\nThat said, the synthetic results and ablations are credible evidence that the framework does what it claims within its assumptions. The paper is a solid engineering contribution to neuromorphic vision; it's not a conceptual leap. The promised code and data would help, but they're not out yet.\n\nI'd send it to peer review, but the reviewers should push for code/data, repeated trials, a fair SpikeGS baseline, and at least one real-world sequence with quantitative pose error.","headline":"A credible one-stage integration of spike-to-image reconstruction, pose correction, and 3DGS that beats cascaded baselines on synthetic data, but the real-world pose-correction claim is under-supported.","tokens_in":13590,"tokens_out":4461,"would_cite":true,"duration_ms":43404,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Joint spike, pose, and 3D Gaussians beat three-stage pipelines.","keywords":["spike camera","3D Gaussian splatting","joint optimization","camera pose correction","novel view synthesis","neuromorphic vision","self-supervised image reconstruction","motion deblurring"],"falsifier":"Run USP-Gaussian on a sequence captured with a motion-capture-verified trajectory that includes a sharp acceleration or a pause within a spike interval, and compare optimized poses to ground truth; if translation or rotation errors stay near the initial values or diverge from the ground-truth trajectory, the linear-SE(3) motion model is the failing load-bearing assumption.","tokens_in":12495,"feed_emoji":"📸","tokens_out":5255,"duration_ms":43123,"temperature":0.7,"pith_summary":"Spike cameras record 40-kHz binary streams, and previous methods turn those streams into images first, then estimate poses, then build 3D scenes. That cascade lets image-reconstruction errors corrupt pose estimation and ultimately the 3D model. This paper claims that optimizing the spike-to-image network, the camera poses, and a 3D Gaussian Splatting scene in a single stage removes that error amplification. It reports higher PSNR/SSIM/LPIPS than cascaded baselines on synthetic scenes and stable quality under pose perturbation, while also correcting noisy initial poses.","feed_headline":"Joint spike, pose, and 3D Gaussians beat three-stage pipelines","feed_subtitle":"A unified framework reconstructs images, corrects poses, and trains 3DGS together, cutting cumulative error.","key_machinery":"The load-bearing mechanism is the joint loss $L_{\\text{joint}}$ that aligns the $M$ reconstructed frames $I_{\\text{rec}}(t_m)$ from Recon-Net with the $M$ frames $I_{\\text{gs}}(t_m)$ rendered by 3DGS at the same interpolated poses, minimized with a flip-and-minimum operation to handle reversed temporal order. Around it, the multi-reblur loss $L_{\\text{rec}}$ (and its 3DGS counterpart $L_{\\text{gs}}$) compare the average of $M$ sharp frames against the long-exposure image $E(T) = C \\cdot N / T$ computed from spike counts, which prevents the identity-mapping failure mode and lets both branches be trained without pre-reconstructed clean images. The camera-trajectory model in Eq. (10), linear interpolation in the Lie algebra of SE(3) between an optimized $T_{\\text{start}}$ and $T_{\\text{end}}$, supplies the poses at each sampled time and is what pose correction actually adjusts.","core_discovery":"USP-Gaussian proposes a one-stage framework that takes spike streams from multiple viewpoints with imprecise initial poses and jointly optimizes three things: a convolution-based Recon-Net that maps a long-plus-short spike window to a sharp image at time t; the start and end SE(3) poses of each spike interval, with intermediate poses by linear Lie-algebra interpolation; and a set of 3D Gaussian primitives. The training signal is a sum of three losses: a multi-reblur loss for Recon-Net, a reblur loss for 3DGS, both comparing averaged sharp frames against the long-exposure image computed from spike counts, and a joint loss that aligns the Recon-Net image sequence with the 3DGS-rendered sequence, with a flip-and-minimum operation to fix temporal ordering. The central claim is that this joint optimization makes the two branches mutually corrective, so that multi-view consistency from 3DGS regularizes reconstruction and spike-reconstruction details sharpen the 3D scene, avoiding the cumulative errors of cascaded pipelines.","pith_inferences":["Replacing the linear SE(3) interpolation with a spline or per-timestamp learnable poses would test whether the constant-velocity model is the bottleneck for arbitrary hand motion.","The L2 alignment between 3DGS and Recon-Net outputs could be reweighted per scene or learned, since relative scale between the two branches may affect optimization stability.","The same unifying recipe could transfer to event cameras and RGB blur cameras, where reconstruction, pose, and radiance-field optimization are also cascaded.","The flip-and-minimum temporal-alignment trick is a cheap regularizer that other multi-frame joint-optimization pipelines could adopt to avoid trajectory reversals."],"forward_implications":["End-to-end joint training should replace the image-reconstruction-then-3DGS recipe for spike cameras, since it removes the dominant source of cumulative error.","Pose optimization makes the pipeline usable when COLMAP-style initialization is poor, such as fast hand-shaken capture, without a separate bundle-adjustment stage.","The mutual supervision between Recon-Net and 3DGS yields better spike-to-image reconstruction than training Recon-Net alone, as the ablation ID-V versus ID-III shows.","Multi-reblur and long-short spike input are necessary to avoid a trivial identity mapping and to suppress salt-and-pepper noise, respectively.","Real-world gains transfer only if the linear SE(3) trajectory model fits the actual motion during each spike interval."],"supporting_citations":[{"why":"Supplies the 3DGS rasterization backbone and the L1+D-SSIM loss used throughout.","marker":"[13]"},{"why":"Primary cascaded baseline; motivates the joint optimization by inheriting cumulative errors.","marker":"[37]"},{"why":"Baseline spike-based NeRF that requires noisy spike images as input.","marker":"[50]"},{"why":"Source of the motion-reblur supervision and the multi-reblur loss design.","marker":"[3]"},{"why":"Provides the learnable pose-interpolation-in-exposure model adopted in Eq. (10).","marker":"[30]"},{"why":"Extends the same pose modeling to Gaussian splatting.","marker":"[42]"},{"why":"Defines TFP, used to compute the long-exposure image E(T) from spike counts.","marker":"[48]"},{"why":"Supervised reconstruction baseline and dataset for Spk2ImgNet comparison.","marker":"[41]"},{"why":"Blind-spot denoiser that SpikeGS relies on; its degradation motivates joint training.","marker":"[15]"},{"why":"COLMAP estimates initial pose sequences for the real-world dataset.","marker":"[24]"}],"fun_headline_variants":["Unified spike-to-3D pipeline cuts cumulative errors","Joint spike image, pose, and Gaussians beat cascades","End-to-end spike camera 3D reconstruction with pose fix","One-stage spike, pose, and splatting optimization","Mutual correction: spike recon meets 3D Gaussians"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes each spike interval is captured during constant-velocity motion, because the pose at any time is obtained by linear interpolation in SE(3) between a start and an end pose; real hand-shaken trajectories with sudden accelerations break this assumption, and then the pose-correction gains in the real-world experiment may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Unified spike-to-3D pipeline cuts cumulative errors","Joint spike image, pose, and Gaussians beat cascades","End-to-end spike camera 3D reconstruction with pose fix","One-stage spike, pose, and splatting optimization","Mutual correction: spike recon meets 3D Gaussians"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1520,"prompt_tokens":1051,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":667,"tokens_out":469,"duration_ms":4619,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:50:17.875821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run USP-Gaussian on a sequence captured with a motion-capture-verified trajectory that includes a sharp acceleration or a pause within a spike interval, and compare optimized poses to ground truth; if translation or rotation errors stay near the initial values or diverge from the ground-truth trajectory, the linear-SE(3) motion model is the failing load-bearing assumption.","supporting_citations":[{"cited_title":"SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion","cited_arxiv_id":"2407.10062","evidence_quote":"Primary cascaded baseline; motivates the joint optimization by inheriting cumulative errors."},{"cited_title":"Spikenerf: Learning neural radi- ance fields from continuous spike stream","cited_arxiv_id":null,"evidence_quote":"Baseline spike-based NeRF that requires noisy spike images as input."},{"cited_title":"Spikereveal: Unlocking temporal sequences from real blurry inputs with spike streams","cited_arxiv_id":null,"evidence_quote":"Source of the motion-reblur supervision and the multi-reblur loss design."},{"cited_title":"Bad-nerf: Bundle adjusted deblur neural radiance fields","cited_arxiv_id":null,"evidence_quote":"Provides the learnable pose-interpolation-in-exposure model adopted in Eq. (10)."},{"cited_title":"A retina-inspired sampling method for visual texture recon- struction","cited_arxiv_id":null,"evidence_quote":"Defines TFP, used to compute the long-exposure image E(T) from spike counts."},{"cited_title":"Spk2imgnet: Learning to reconstruct dy- namic scene from continuous spike stream","cited_arxiv_id":null,"evidence_quote":"Supervised reconstruction baseline and dataset for Spk2ImgNet comparison."},{"cited_title":"High-quality self-supervised deep image denoising","cited_arxiv_id":null,"evidence_quote":"Blind-spot denoiser that SpikeGS relies on; its degradation motivates joint training."},{"cited_title":"Structure- from-motion revisited","cited_arxiv_id":null,"evidence_quote":"COLMAP estimates initial pose sequences for the real-world dataset."}],"review_version":1}