{"id":"1d119ac4-c4a3-4b77-8698-1155dc05c3fe","arxiv_id":"2507.00006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MVGBench evaluates multi-view generators through self-consistency of 3D reconstructions and uses this protocol to rank 12 models and build a better one.","lead":"MVGBench, a new benchmark for multi-view image generation models, scores 3D consistency by fitting two 3D models to disjoint subsets of generated views and measuring their disagreement, avoiding any need for ground-truth geometry. It ranks 12 existing models on synthetic and real data, exposes a consistency-versus-quality trade-off, and introduces a model that balances the two.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The self-consistency metric is validated only on perfectly consistent inputs; its rank validity on inconsistent generated views is untested, so the 'outperforms all' claim may partly reflect 3DGS fitting fragility.","rationale":"The reader's weakest assumption identifies precisely the load-bearing point: the 3D consistency metric is the benchmark's core novelty, and its validity for inconsistent generated views is not established. My stress-test agrees and sharpens the concern into a testable monotonicity/calibration requirement. The manuscript's Tab. 1 validation is necessary but not sufficient: it demonstrates invariance of the fitting pipeline on perfectly consistent inputs, not discriminative validity on the actual evaluation distribution. The proposed controlled-corruption experiment would settle whether the metric tracks known inconsistency levels and whether fitting artifacts are small relative to the method ranking gaps. Since the reader's CONDITIONAL verdict already flags this and the other issues (ICP sensitivity, normalization dependence on method set, code release), my pass does not shift the verdict; the concern strengthens the conditionality but does not invalidate the work. I recommend keeping CONDITIONAL, with the corruption-sweep experiment as a concrete acceptance requirement.","tokens_in":23132,"tokens_out":3355,"duration_ms":40055,"concrete_test":"Construct a controlled inconsistency sweep on GSO30: take ground-truth multi-views and randomly replace a fraction p of views, for p in {0, 0.1, 0.2, 0.5, 1.0}, with views of a different object (or with geometrically jittered cameras), then run the full MVGBench 3D consistency pipeline on each corrupted set. If the consistency metric is not monotonically decreasing in p, or if small corruption levels (p around 0.1) already produce score changes larger than the ranking gaps between methods in Tab. 2, the metric cannot be trusted to rank real MVGs. This directly tests whether fitting artifacts, rather than true inconsistency, dominate the reported scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MVGBench's 3D self-consistency metric fairly measures generator inconsistency rests on the assumption in Sec. 3.1 that the discrepancy between two independently fitted 3DGS models is caused by multi-view inconsistency rather than by 3DGS fitting artifacts. Sec. 4.1/Tab. 1 tests this only by fitting GT multi-views—perfectly consistent images—under different view counts and camera settings and observing less than 8% relative deviation. This shows the fitting pipeline is stable on ideal inputs, but it does not establish that the metric is neutral when inputs are inconsistent. For real generated outputs, 3DGS fitting can fail in method-specific ways: blurry or detail-poor images may produce smoother, easier-to-fit Gaussians and artificially low Chamfer/depth errors, while high-frequency inconsistent images may cause divergent local optima across view subsets, inflating errors. Because Tab. 2 ranks methods exactly under this regime, the headline result that 'ours' outperforms all methods on 3D consistency may partly reflect that the proposed model's outputs (video-based with ConvNextV2 encoder) are more 3DGS-friendly, rather than more genuinely 3D-consistent. No independent check—human ranking, known-inconsistency injection, or cross-metric agreement—is provided to rule out this confound. The paper's Limitations section only excludes methods generating fewer than 10 views and does not address this more general threat to metric validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MVGBench proposes a benchmark for multi-view image generation (MVG) models that evaluates ten metrics across three dimensions: 3D geometric/texture consistency, image quality, and semantic consistency. Its central novelty is a ground-truth-free 3D self-consistency metric: generated multi-views are split into two disjoint subsets, two 3D Gaussian Splatting (3DGS) models are fitted independently, and the discrepancy between them (Chamfer distance, depth error, cPSNR/cSSIM/cLPIPS) is used as a measure of 3D consistency. The benchmark covers 12 MVG methods on four datasets, with best-setup, real-image generalization, and robustness evaluations. Using the benchmark's design-choice analysis, the authors propose ViFiGen (SV3D plus camera positional encoding plus a ConvNextV2 encoder) and claim it outperforms all evaluated MVGs on 3D consistency. The metric is validated on ground-truth views (Table 1), and the VLM-based quality/semantic metrics and oFID are validated against user studies.","tokens_in":23414,"tokens_out":4333,"duration_ms":48993,"significance":"If the 3D consistency metric is valid, MVGBench fills a real gap: existing evaluations compare generated views against method-specific ground truth, which is inappropriate for generative tasks. The paper's strengths include a genuinely GT-free consistency protocol, a broad systematic comparison of 12 state-of-the-art methods on four datasets, externally validated VLM metrics (Pearson 0.69-0.92), and a user study showing oFID aligns with human rankings better than dataset-level FID. The design-choice analysis (camera embedding, input encoder, feature interaction, training data amount) is informative and the code/model release will be valuable. However, the central metric's validity on inconsistent generated inputs is the load-bearing assumption, and it is not yet established; the benchmark's usefulness and the headline 'outperforms all' claim depend on resolving this concern. The paper is a solid candidate for publication after substantial additional validation.","major_comments":[{"comment":"The design choices for ViFiGen were selected using the MVGBench analysis, and the same benchmark is then used to claim that ViFiGen outperforms all evaluated methods. This creates a mild self-referential loop: the benchmark is used both to discover best practices and to evaluate the method built from them. The benchmark is partly externally grounded (Table 1, VLM user studies), so this is not disqualifying, but the 'outperforms all' claim currently has no independent check. Please provide an external validation of this claim, such as a human study on 3D consistency or an evaluation on a held-out dataset not involved in the design-choice analysis; alternatively, explicitly frame the claim as 'in-benchmark' rather than universal.","section":"Section 4.3 / Section 5 / Table 3"},{"comment":"The normalization for CD, depth, and cLPIPS in Eq. (5) uses max_e_mvg, the maximum error over the evaluated methods, making the normalized robustness scores dependent on the composition of the evaluated method set. If the set of methods changes, the normalized values and possibly the method rankings can change. The paper should state this relativity explicitly and report raw scores or use a fixed reference (e.g., ground-truth-based upper bound) as the normalization anchor.","section":"Section 6.1 / Equation (5)"}],"minor_comments":[{"comment":"The relative standard deviation for depth is 0.082 (8.2%), while the text states 'the deviation is less than 8% of the average score across all variants'; please reconcile the number and the claim.","section":"Section 4.1 / Table 1"},{"comment":"The sentence 'our method cannot evaluate methods that generate very few views (<10)' refers to the benchmark, not to the proposed ViFiGen model; please rephrase to avoid ambiguity.","section":"Section 8"},{"comment":"Several references contain stray page numbers or malformed entries (e.g., references [2], [8], [9], and [54] end with extra page-like numbers); please clean the reference list.","section":"References"},{"comment":"The claim that VLM metrics strongly correlate with human perception is supported numerically only in Section 5's summary; please report the per-metric Pearson coefficients in Section 4.1 as well.","section":"Section 4.1 / Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central metric-validity concern is the main gatekeeper. If the authors can add controlled-inconsistency tests, human 3D-consistency ranking, and a reconstruction-backbone robustness check, the paper would be a strong benchmark contribution. The self-referential design-choice evaluation is acceptable but should be explicitly acknowledged. The paper fits the journal scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a serious benchmark paper that deserves a proper review, and the central idea — measuring 3D consistency by comparing two 3DGS reconstructions fit from disjoint subsets of generated views — is genuinely useful. But the headline \"outperforms all\" for their ViFiGen model should be read with caution, because the metric's behavior on genuinely inconsistent generated views is not directly validated.\n\nThe new stuff is real. The self-consistency metric avoids ground-truth comparison, which is the right call for a generative task with multiple plausible outputs. The evaluation of 12 methods on 4 datasets across best-setup, generalization, and robustness performance is the most complete comparison I've seen in this subfield. And the design-choice study (camera embedding, input encoder, feature interaction, training data size) is a substantial contribution on its own — it gives the community concrete guidance instead of another \"ours is better\" paper. The validation work is also real: Table 1 shows the metric is stable across view counts and camera settings when fit from GT views (relative std under 8%), and the VLM metrics correlate with human judgments (Pearson 0.69–0.92). The observation that the autoencoder alone degrades 3D consistency is a nice diagnostic.\n\nThe soft spot is the one the stress-test flags, and I think it's legitimate. The metric assumes that the disagreement between two 3DGS fits is driven by multi-view inconsistency rather than fitting artifacts, but that assumption is only tested on perfectly consistent GT views. The paper's own limitations section admits that for methods generating fewer than 10 views, fitting error dominates the consistency score — which is exactly the confound at stake, just in a different regime. For real generated outputs, a method that produces blurry, detail-poor images may be easier to fit and could earn artificially good consistency scores, while a sharp but slightly inconsistent generator gets penalized. I don't think this invalidates the benchmark; the rankings broadly match qualitative examples and user perception. But the paper would be much stronger with a known-inconsistency injection test (perturb some GT views and verify the error rises monotonically) and a sensitivity analysis of the ICP alignment used on real data.\n\nMinor issues: the normalization in Eq. 5 uses the max error across the evaluated method set, which makes the normalized robustness scores meaningful only relative to that set; that should be frozen and published if the benchmark is to be extended. Also, \"outperforms all\" is slightly imprecise: on GSO, SyncDreamer actually has a lower Chamfer distance (2.99 vs 3.15), and the win is mostly on depth and texture consistency. It's a fair aggregate claim, but the wording is loose. Code and benchmark release are promised but not yet available; for a benchmark paper, that's a practical concern.\n\nWho this is for: anyone building or evaluating multi-view generators, or using them for single-image 3D. It addresses a real evaluation bottleneck in a field with more than twenty papers a year. My recommendation: send it to peer review. The metric validity question is fixable and should be the main revision request. I would engage as a referee.","headline":"A genuinely useful ground-truth-free benchmark for multi-view generation whose 3D consistency metric is clever and mostly well validated, but the 'outperforms all' claim for ViFiGen rests on a metric whose behavior on inconsistent outputs is only indirectly tested.","tokens_in":24021,"tokens_out":3743,"would_cite":true,"duration_ms":35047,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MVGBench argues that multi-view generators should be judged by whether their outputs fuse into one coherent 3D object, and proposes a ground-truth-free 3D self-consistency metric to do it.","keywords":["multi-view generation","3D consistency","benchmark","3D Gaussian Splatting","image-to-3D","generative evaluation","vision-language model","ViFiGen"],"falsifier":"Take ground-truth multi-view images, inject a controlled amount of geometric or texture inconsistency into one subset of views, and verify that the metric rises monotonically with the injected amount across view counts and elevations; any plateau or reversal would show that fitting artifacts, not generation inconsistency, drive the score.","tokens_in":22891,"feed_emoji":"🧊","tokens_out":6744,"duration_ms":66876,"temperature":0.7,"pith_summary":"MVGBench argues that multi-view image generation models should be judged by whether their outputs cohere into a single 3D object, not by how closely they match one ground-truth view. Its central proposal is a ground-truth-free 3D self-consistency metric: the generated views are split into two subsets, each subset is used to fit a 3D Gaussian Splatting, and the disagreement between the two reconstructions is treated as the measure of inconsistency. The benchmark adds image-quality and semantic metrics, evaluates twelve existing models on four curated real and synthetic datasets, and finds a persistent trade-off between 3D consistency and image quality as well as a large synthetic-to-real gap. Combining the design choices it identifies as best, the paper introduces ViFiGen, a video-based generator reported to outperform all evaluated models on 3D consistency.","feed_headline":"New metric scores 3D consistency without ground truth","feed_subtitle":"Fits two 3D models to disjoint views; their disagreement reveals which generators truly agree.","key_machinery":"Split-then-reconstruct self-consistency comparison. Generated multi-views are divided into two subsets, two independent 3D Gaussian Splatting reconstructions (a point-based radiance-field representation of anisotropic Gaussians) are optimized, and their disagreement is computed as geometry errors (Chamfer distance on points resampled from the Gaussians, plus rendered depth error) and texture errors (PSNR, SSIM, and LPIPS between renderings from shared test views). Because all methods are compared through the same test views, with ICP alignment when no ground-truth scale exists, methods trained with different cameras, focal lengths, and elevation conventions can be ranked on the same scale.","core_discovery":"Fitting two 3D Gaussian Splattings to disjoint subsets of a generator's output views and measuring their geometric and texture disagreement gives a valid, fair measure of 3D consistency for multi-view generators. The paper validates this metric against perfectly consistent ground-truth views, showing it is nearly invariant to the number of views and the camera rendering settings, and it can be applied to real images where no 3D ground truth exists. Classic pair-wise metrics can rank an inconsistent generator above a consistent one; the self-consistency metric reverses such rankings. On this measure, video-diffusion models balance consistency and image quality better than image-diffusion models, and the paper's ViFiGen, a video-based model with a fine-grained convolution input encoder and a stronger camera embedding, achieves the best 3D consistency among the twelve methods evaluated.","pith_inferences":["The split-then-reconstruct protocol is a general recipe that could be carried over to text-to-3D or multi-view video generators with little change, since it never needs a reference 3D model.","If 3D Gaussian Splatting fitting is fragile for very few views or extreme elevations, the score could partly measure reconstruction stability rather than generation consistency; a learned few-view reconstructor as the fitting stage would separate the two.","The large synthetic-to-real gap points to training data diversity as the next major lever, independently of architecture choices.","At roughly 2.7 hours per 100 objects, the metric is cheap enough to serve as an automated regression test for new releases of multi-view generators."],"forward_implications":["Multi-view generators can be ranked on real photographs, because the metric requires no 3D ground truth and no method-specific target views.","A model can look strong under pair-wise PSNR/SSIM comparison yet fail the self-consistency check, so the two rankings are not interchangeable.","Video-diffusion backbones have a measurable advantage in the consistency-quality trade-off, giving future work a concrete starting point.","Replacing a category-level image encoder with a fine-grained convolution encoder, and using ray- or position-based camera embeddings, is a transferable recipe for better 3D consistency.","High-frequency detail loss in the latent autoencoder degrades consistency even before generation, identifying the autoencoder as a bottleneck for fine structures."],"supporting_citations":[{"why":"Supplies the 3D Gaussian Splatting representation that is fitted to each view subset; the entire self-consistency metric rests on these reconstructions.","marker":"[27]"},{"why":"Provides the video-diffusion base model and its 21-view camera setup, which the paper fine-tunes for design-choice ablations and uses as the main comparison point for ViFiGen.","marker":"[54]"},{"why":"Establishes the single-image multi-view generation task and is one of the strongest baselines whose 3D inconsistency the benchmark exposes.","marker":"[2]"},{"why":"Contributes the synchronized 3D feature interaction design choice that the paper tests and finds largely redundant on top of video attention.","marker":"[38]"},{"why":"Supplies the ray-conditioned camera embedding alternative that outperforms simple MLP embeddings in the ablations.","marker":"[80]"},{"why":"Contributes the camera positional embedding and the fine-grained convolution input encoder that ViFiGen combines.","marker":"[28]"}],"fun_headline_variants":["Self-consistency metric benchmarks multi-view generators","MVGBench: 3D consistency score without ground truth","New metric measures 3D consistency via disjoint views","Multi-view generation benchmark uses 3D reconstruction metric"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric assumes that the disagreement between the two fitted 3D models comes from the generator's inconsistency, not from the reconstruction algorithm failing differently on the two view subsets.","fun_headline_variants_meta":{"raw":{"variants":["Self-consistency metric benchmarks multi-view generators","MVGBench: 3D consistency score without ground truth","New metric measures 3D consistency via disjoint views","Multi-view generation benchmark uses 3D reconstruction metric"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1436,"prompt_tokens":956,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":572,"tokens_out":480,"duration_ms":5580,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:46:52.939217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take ground-truth multi-view images, inject a controlled amount of geometric or texture inconsistency into one subset of views, and verify that the metric rises monotonically with the injected amount across view counts and elevations; any plateau or reversal would show that fitting artifacts, not generation inconsistency, drive the score.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 42 (4), 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D Gaussian Splatting representation that is fitted to each view subset; the entire self-consistency metric rests on these reconstructions."},{"cited_title":"SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using La- tent Video Diffusion, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the video-diffusion base model and its 21-view camera setup, which the paper fine-tunes for design-choice ablations and uses as the main comparison point for ViFiGen."},{"cited_title":"Free3D: Consis- tent Novel View Synthesis Without 3D Representation","cited_arxiv_id":null,"evidence_quote":"Supplies the ray-conditioned camera embedding alternative that outperforms simple MLP embeddings in the ablations."},{"cited_title":"EscherNet: A Generative Model for Scalable View Synthesis","cited_arxiv_id":null,"evidence_quote":"Contributes the camera positional embedding and the fine-grained convolution input encoder that ViFiGen combines."}],"review_version":1}