{"id":"0be9ee3e-e0c3-4e6c-b30f-8c9439959c4b","arxiv_id":"2412.03102","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper reports a 26M-parameter MPI network that converts planar video to stereo at 2K in about 18 ms per frame with perceptual quality close to the much slower TMPI.","lead":"A lightweight neural network converts ordinary 2D video into 3D stereo in real time at 2K resolution by combining a small depth-sensing branch with simplified multi-plane image rendering. For companies building 3D displays, VR, or real-time video pipelines, the paper's 40x speed-up over the previous state of the art is the headline result to verify.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low-resolution MPI rendering (Eq. 9) is validated only by BIQA on 10 clips; a GT-based comparison on the 3D movie test set is needed to support the comparable-quality and real-time claims.","rationale":"The reader's weakest assumption identifies exactly the load-bearing step: Eq. 9's low-resolution MPI prediction and bilinear upsampling is what enables 18 ms inference at 2K. The paper validates this only with no-reference metrics on ten Youku2K clips and a 15-observer MOS that does not compare rendering variants directly. The 3D movie test set has ground truth but is used only for the full pipeline, leaving the core rendering assumption untested on content with large disocclusions. I agree with the CONDITIONAL verdict: the claim is plausible but depends on this unverified assumption, and a direct GT-based ablation would settle it. No separate concern requires changing the verdict.","tokens_in":10799,"tokens_out":6954,"duration_ms":74524,"concrete_test":"Run the proposed low-resolution rendering (Ours) and the full-resolution rendering variant listed in Table 4 on the 3D movie test set, comparing PSNR, SSIM, and LPIPS against the existing ground-truth right views. If the full-resolution variant outperforms Ours by a material margin (for example, LPIPS difference greater than 0.01 or PSNR difference greater than 0.5 dB), the 'no visible degradation' assumption in Eq. 9 is not supported for general content; if the gap is negligible, the concern is resolved. Report per-clip breakdowns, especially for clips containing large disoccluded regions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central speed/quality tradeoff rests on Eq. 9: MPI planes are predicted at low resolution and only blended with the full-resolution input where visible, under the claim that upsampling artifacts are not prominent for 'small and smooth occlusion regions.' This is an empirical assumption, and the only evidence offered is Table 4's BIQA scores on ten Youku2K clips; the 3D movie test set, the only benchmark with ground truth, is never used to compare the low-resolution rendering against the full-resolution variant. If the assumption fails on videos with large or textured disocclusions, the method would need full-resolution MPI (155 ms per Table 4, about 6 fps), eliminating both the real-time claim and the 40x speedup versus TMPI. Table 2's MOS (3.42 vs 3.47) is based on 15 observers and does not isolate this rendering hypothesis, so it cannot validate Eq. 9.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LMPIN, a lightweight MPI-based network for planar-to-stereo video conversion. The network contains a detail branch and a lightweight depth-semantic branch; a training-only auxiliary branch supervises depth perception through a pretrained monocular depth model, allowing inference without explicit depth input. The main acceleration comes from computing the MPI at low resolution and upsampling it with bilinear interpolation while blending with the original high-resolution image (Eq. 9). The authors evaluate on a 3D movie test set with ground truth and on ten 2K YouTube-style videos with no-reference metrics. They report LPIPS 0.036, PSNR 33.037, and SSIM 0.913 on the 3D movie set (TMPI: 0.034, 33.630, 0.924), and 18 ms per frame at 2K versus 721 ms for TMPI, claiming over 40x acceleration with similar subjective quality.","tokens_in":11043,"tokens_out":10510,"duration_ms":89666,"significance":"If the results hold, the paper offers a practical real-time 2D-to-3D conversion solution at 2K resolution, a meaningful step for glasses-free 3D displays and VR. The architecture is compact (26M parameters) and effectively combines a lightweight depth-semantic branch with a training-only heavy supervision branch. The experiments include several baselines, ablations of the network components, and an ablation of the low-resolution rendering. However, the central real-time/quality claim rests on an empirical assumption in Eq. (9) that is only validated with no-reference metrics on a small set of clips, and the paper omits the downsampling factor and hardware details needed to reproduce the runtime. These issues must be addressed before the claims can be fully accepted.","major_comments":[{"comment":"The central real-time speed-up relies on the assumption in Eq. (9) that computing MPI planes at low resolution and upsampling them does not visibly degrade the synthesized view. The only evidence provided is Table 4, which uses no-reference quality metrics (MUSIQ, HIQA, NIQE) on ten Youku2K clips; the 3D movie test set, which has ground truth, is never used to compare the low-resolution rendering against the full-resolution variant. Since the proposed method is already 0.6 dB below TMPI in PSNR on the 3D movie test set (Table 1), the paper should report PSNR/SSIM/LPIPS for the full-resolution variant on that set to show whether Eq. (9) contributes to the gap. This experiment is load-bearing: if Eq. (9) degrades quality on videos with large or textured disocclusions, the real-time claim would require full-resolution MPI at 155 ms per frame (Table 4), eliminating the headline speed advantage.","section":"Accelerate Rendering with Low-Resolution MPI, Eq. (9); Table 4"},{"comment":"The downsampling factor for the low-resolution MPI is never specified. The text states only that the MPI is computed 'in low-resolution space,' but the resolution of the planes (e.g., 480×270 for 2K input) is required to reproduce the 18 ms runtime and to judge the trade-off. Please state the factor and, ideally, ablate over it on a ground-truth benchmark, since this free parameter directly controls the speed/quality balance.","section":"Method, Accelerate Rendering with Low-Resolution MPI"},{"comment":"The runtime comparison in Table 2 is reported without specifying the GPU model or measurement protocol. Since real-time inference is a headline contribution, the hardware (e.g., NVIDIA V100 vs RTX 3090) and the measurement method (e.g., average over frames after warm-up) must be stated. Without this, the 'over 40x acceleration' number is not reproducible and the comparison across methods may be affected by different implementation efficiencies.","section":"Experiments, Datasets and Implementation Details"}],"minor_comments":[{"comment":"There are several typos, e.g., 'Experimential Results' in the Experiments section and 'Full resotion' in Table 4; these should be corrected.","section":"General / typos"},{"comment":"In the definition of w_n, the product over j>n uses α_n_s instead of α_j_s; the subscript should be j to match the index of the product.","section":"Eq. (8)"},{"comment":"The 'Bilinear interpolation' row is not clearly defined; specify whether it upscales the final rendered image or the MPI planes, and describe how it differs from the proposed low-resolution rendering.","section":"Table 4"},{"comment":"The MOS results are based on 15 observers with no reported variance or significance test; please state the number of evaluated clips, the display device, and provide confidence intervals, since the 0.05 difference between the proposed method and TMPI is within likely inter-observer noise.","section":"Table 2 / MOS"},{"comment":"The detail branch is described only as 'a commonly used autoencoder (Zhang et al. 2023)'; the exact architecture or a precise reference should be given for reproducibility.","section":"Method, Detail Branch"},{"comment":"The resolution of the 3D movie test set is not stated in Table 1; if it is different from the 2K Youku test set, this should be explicitly mentioned so the reader can interpret the two evaluations separately.","section":"Experiments, Test set"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the topic is timely. The main concern is that the most novel component (low-resolution MPI rendering) is the least rigorously validated; the authors should be asked to supply a ground-truth comparison on the 3D movie test set between full-resolution and low-resolution rendering, as well as the missing implementation details. The use of the authors' own TMPI as the SOTA baseline is acceptable, especially because other baselines are included, but the fairness of the runtime comparison should be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about if you work on 2D-to-3D conversion. The headline result—40× faster than TMPI at similar perceptual quality—is plausible but rests on one under-validated assumption.\n\nWhat is actually new is the combination: a training-only DPT teacher distills depth knowledge into a lightweight branch, and the MPI is computed at low resolution and upsampled only for blending with the full-res input (Eq. 9). The low-res rendering buys the real-time claim, and it is the paper's softest spot. The assumption—that upsampling artifacts are not visible for small, smooth occlusion regions—is tested only on ten Youku2K clips with no-reference metrics (MUSIQ, HIQA, NIQE), which are not designed for stereo artifacts. The 3D movie test set has ground truth and is used elsewhere in the paper, but it is never used to compare low-res versus full-res rendering. That is a straightforward missing experiment. If the assumption fails on videos with large or textured disocclusions, the method drops to 155 ms (about 6 fps), which kills the real-time 2K claim and the 40× headline. However, even then the architecture is still 4.6× faster than TMPI at full resolution, so the work does not collapse entirely.\n\nOther soft spots are minor: MOS comes from 15 observers with no variance or significance test, the GPU model for runtime is not reported, and no code or data are provided. These are fixable in revision. The comparison to TMPI is fair as a system-level benchmark, though TMPI is the authors' own prior work and uses temporal information; the paper does not dwell on that asymmetry.\n\nOverall, the paper is honest, the method is sensible, and the central claim is credible but depends on one assumption that needs stronger evidence. It deserves a serious referee. I would send it to review, and ask the authors for the low-res vs full-res comparison on the 3D movie test set plus more transparency on hardware and MOS.","headline":"Solid lightweight MPI stereo conversion; the 40x speedup is plausible, but the low-res rendering assumption needs a GT-based check before the real-time claim fully lands.","tokens_in":11576,"tokens_out":5189,"would_cite":true,"duration_ms":46084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 26M-parameter multiplane-image network converts planar video to stereo at 2K resolution in 18 ms per frame, over 40 times faster than a state-of-the-art temporal-MPI method while matching its perceptual quality.","keywords":["planar-to-stereo conversion","multiplane images","real-time 3D video conversion","lightweight network","depth-aware features","2K video","view synthesis"],"falsifier":"Test the method on a suite of 2K videos with large foreground objects and textured backgrounds, such as a person walking in front of foliage; if the low-resolution render produces visibly blurred or jagged edges in the disoccluded regions, or if its MOS drops noticeably below the full-resolution variant, the load-bearing assumption in Eq. (9) is falsified.","tokens_in":10618,"feed_emoji":"🎥","tokens_out":5740,"duration_ms":51336,"temperature":0.7,"pith_summary":"The paper tries to establish that a small multiplane-image network with 26 million parameters can convert ordinary planar video into stereoscopic video at 2K resolution in real time, without needing an explicit depth map at inference. It argues that a lightweight depth-semantic branch can learn depth-aware features under supervision from a heavy pretrained monocular depth estimator during training, and that the expensive MPI rendering can be computed at low resolution and then upsampled. If correct, this would make MPI-based 3D conversion practical for glasses-free displays and VR, where current high-quality methods take hundreds of milliseconds per frame. The reported result is 18 ms per 2K frame versus 721 ms for the temporal MPI baseline, with nearly equal perceptual scores.","feed_headline":"18 ms per 2K frame: 2D-to-3D conversion goes real-time","feed_subtitle":"A 26M-parameter network matches the quality of a much larger temporal-MPI model while running over 40x faster.","key_machinery":"The central object is the multiplane image (MPI): a scene represented as N fronto-parallel RGBA planes at fixed depths, rendered into a novel view by horizontal pixel warping followed by back-to-front alpha compositing (the 'over' operation). The paper's mechanism is to generate these planes with two encoders: a detail branch that encodes the full-resolution texture once per frame, and a lightweight depth-semantic branch with a large receptive field that produces depth-aware features; mask heads slice those features into per-plane masks, and a small decoder produces the color-plus-density planes. A training-only auxiliary branch, consisting of two depth heads and a pretrained monocular depth reference, forces the depth-semantic features to align with real depth, and the low-resolution rendering shortcut in Eq. (9) replaces full-resolution plane generation with bilinear upsampling.","core_discovery":"The central claim is that a 26M-parameter MPI network with no explicit depth input at inference can match the perceptual quality of a 37M-parameter temporal MPI model plus a 123M-parameter depth estimator while running over 40 times faster (18 ms versus 721 ms per 2K frame). The architecture couples a detail branch that extracts features from the input frame with a depth-semantic branch that encodes coarse depth cues; during training a coarse-to-fine auxiliary branch with two depth heads is supervised by a large pretrained monocular depth network, but that branch is discarded at inference. Rendering is accelerated by computing the 16 MPI planes at reduced resolution and bilinearly upsampling the blend weights and colors before compositing, on the claim that the artifacts of MPI upsampling are not prominent in small, smooth occlusion regions. The paper reports LPIPS 0.036 versus 0.034 for TMPI and comparable blind-quality and MOS scores on its 2K test set.","pith_inferences":["An unstated extension is that the low-resolution rendering shortcut should transfer to 4K and 8K input with similar speed scaling, since the MPI computation itself is resolution-independent; the 'small and smooth occlusion' assumption is where that transfer would likely strain.","The same depth-semantic-branch-plus-training-auxiliary recipe could be applied to free-viewpoint video, not just fixed-baseline stereo, by rendering from a different target viewpoint per frame.","Because the depth reference network is needed only during training, the final model could be trained with synthetic depth from graphics engines, decoupling its quality from the reference network's failure modes without affecting inference cost."],"forward_implications":["Real-time 2K stereoscopic conversion becomes feasible on a single modern GPU, since the reported 18 ms per frame is well under the typical interactive budget for glasses-free 3D displays.","MPI methods no longer need an external monocular depth estimator at inference, removing a large parameter dependency and its associated latency.","The training-only depth auxiliary branch means the model can be supervised with any pretrained depth estimator without adding any inference cost.","The reduction from hundreds of millions of parameters to 26M makes deployment on embedded or mobile devices more plausible."],"supporting_citations":[{"why":"Provides the TMPI baseline whose runtime and quality numbers anchor the over-40x acceleration claim.","marker":"Diao et al. 2024"},{"why":"Supplies the adaptive MPI baseline (ADAMPI) and the mask-loss inspiration for depth-guided supervision.","marker":"Han, Wang, and Yang 2022"},{"why":"Supplies MINE, a single-image MPI baseline that infers depth implicitly, serving as a comparison point for the depth-free design.","marker":"Li et al. 2021"},{"why":"Provides the pretrained monocular depth estimator used to generate reference depth maps for training supervision and for depth-input baselines.","marker":"Ranftl, Bochkovskiy, and Koltun 2021"},{"why":"Defines the alpha-compositing 'over' operation used in MPI rendering.","marker":"Porter and Duff 1984"},{"why":"Supplies the data preparation method for generating synthetic stereo training pairs from single images.","marker":"Watson et al. 2020"},{"why":"Provides the Youku2K planar-video dataset used to test high-resolution performance and inference speed.","marker":"Youku 2019"}],"fun_headline_variants":["40x faster 2D-to-3D conversion with matching quality","Real-time 2D-to-3D: 26M-parameter net does 18 ms per 2K","TMPI quality at 40x speed: 18 ms 2K conversion","No depth input at test: 40x faster 2D-to-3D conversion","Real-time stereo conversion: 18 ms per 2K, 40x faster than TMPI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-time and quality claims rest on the assumption that computing the MPI planes at low resolution and bilinearly upsampling them does not visibly harm the rendered stereo view, which the paper justifies by saying that occluded regions are small and smooth and supports with ten Youku2K clips.","fun_headline_variants_meta":{"raw":{"variants":["40x faster 2D-to-3D conversion with matching quality","Real-time 2D-to-3D: 26M-parameter net does 18 ms per 2K","TMPI quality at 40x speed: 18 ms 2K conversion","No depth input at test: 40x faster 2D-to-3D conversion","Real-time stereo conversion: 18 ms per 2K, 40x faster than TMPI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3748,"prompt_tokens":977,"completion_tokens":2771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2651}},"tokens_in":593,"tokens_out":2771,"duration_ms":19873,"temperature":1.0,"reasoning_tokens":2651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:45:57.914806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Test the method on a suite of 2K videos with large foreground objects and textured backgrounds, such as a person walking in front of foliage; if the low-resolution render produces visibly blurred or jagged edges in the disoccluded regions, or if its MOS drops noticeably below the full-resolution variant, the load-bearing assumption in Eq. (9) is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TMPI baseline whose runtime and quality numbers anchor the over-40x acceleration claim."},{"cited_title":"M.; Turmukhambetov, D.; Brostow, G","cited_arxiv_id":null,"evidence_quote":"Supplies the data preparation method for generating synthetic stereo training pairs from single images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Youku2K planar-video dataset used to test high-resolution performance and inference speed."}],"review_version":1}