{"id":"90c3b867-9a63-4717-89a9-a9a2d9c28b67","arxiv_id":"2507.17388","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EndoGen generates condition-guided endoscopic videos by arranging frames into a grid and applying variance-based token masking inside an autoregressive model, improving video quality and downstream polyp segmentation.","lead":"EndoGen is a system that generates short endoscopic videos from a condition such as 'polyp' or 'ulcer'. It is designed to create realistic synthetic footage for medical training and for training AI systems, which is a step beyond generating single still images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-class FVD comparisons lack conditional controls for VideoGPT/VDM/SimDA; SOTA claim is not established","rationale":"The reader's weakest assumption focuses on the frozen ImageNet-pretrained VQGAN potentially capping endoscopy-specific texture fidelity. That is a plausible concern, but it does not attack the central comparison directly: even if the VQGAN is imperfect, a relative ranking between methods could still be meaningful if all methods were equally conditioned. However, the paper never specifies how the three baselines were adapted to the conditional setting. VideoGPT is famously unconditional; SimDA is a text-to-video adapter; VDM is video diffusion but the paper does not report class-conditioned training details. Without this information, every per-class FVD number for the baselines in Table 1 and every downstream comparison built on the generation quality is suspect. The 'first conditional endoscopic video generation' claim and the headline SOTA performance rest entirely on this comparison. I therefore disagree with the reader's choice of weakest assumption. My recommendation is UNVERDICTED rather than CONDITIONAL because the missing control is not a minor add-on; it is a prerequisite for evaluating the paper's main claim. The proposed test is cheap and decisive: condition the baselines in the same way and see if the gap persists. This does not accuse the authors of any misconduct; it simply notes that the experimental protocol as written cannot rule out the trivial explanation that only EndoGen received a label.","tokens_in":7566,"tokens_out":5097,"duration_ms":59917,"concrete_test":"Run a class-conditioned VideoGPT variant: add a learned pathology-class embedding to the initial token position (and optionally to each frame's tokens), train on the same 16-frame, 128x128 HyperKvasir clips for 300 epochs, and recompute per-class FVD. Apply analogous class-conditioning to VDM and SimDA. Use at least 3 seeds and report mean±std. If conditioned VideoGPT attains FVD near EndoGen's 507.2, the SOTA claim is unsupported; if it remains near 1548.6, the claim survives and the concern is resolved.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Table 1 reports per-class FVD on HyperKvasir as evidence for state-of-the-art conditional generation. The only conditioning mechanism described in §3.1 is EndoGen's: a learnable condition token with ImageNet-pretrained class-conditional AR weights. For the baselines, the paper does not state whether VideoGPT—an intrinsically unconditional video model—was modified to accept pathology labels, nor how SimDA and VDM were conditioned to the eight findings. If VideoGPT was used in its standard unconditional form, its per-class FVD values (e.g., 1433.1 for Barrett's) come from unlabeled sampling and are not metrics of conditional generation; comparing them with EndoGen's conditioned outputs tests presence of condition information, not video quality. The claimed 'significant margins' thus conflates an architectural advantage with a missing experimental control. No seeds or confidence intervals are reported either, so the 507.2 vs 1548.6 average gap cannot be separated from training noise. This is more load-bearing than the frozen VQGAN concern: even a perfect VQGAN would not rescue an invalid baseline comparison.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EndoGen proposes a conditional autoregressive video generation framework for endoscopy. Videos are converted into grid-frame images via the proposed Spatiotemporal Grid-Frame Patterning (SGP), tokenized by a frozen VQGAN, and modeled autoregressively from a learnable condition token; a variance-based Semantic-Aware Token Masking (SAT) mechanism is used during training. The paper reports FVD, CD-FVD, FID, and LPIPS comparisons against SimDA, VDM, and VideoGPT on HyperKvasir and SurgVisdom, ablations of SGP/SAT and the masking threshold pmax, and a downstream semi-supervised polyp segmentation experiment in which EndoGen-generated synthetic frames are used as unlabeled data.","tokens_in":7794,"tokens_out":5051,"duration_ms":55258,"significance":"If the reported results are reproducible, EndoGen would be a useful step toward controllable endoscopic video synthesis and a practical source of synthetic training data for downstream tasks. The method is clearly described, the code release is announced, and the ablation study in Table 4 isolates the main components. However, the evidence for the state-of-the-art claim is currently incomplete: the conditioning of the baselines is not specified, no uncertainty estimates are given, and the frozen VQGAN's fidelity on endoscopic imagery is unexamined. These gaps are fixable with additional experiments and reporting, so the manuscript would benefit from a major revision rather than being rejected outright.","major_comments":[{"comment":"The manuscript does not state how the three baselines were conditioned on the eight pathological findings or the three surgical tasks. VideoGPT is originally an unconditional video model, while SimDA and VDM are diffusion models whose conditioning mechanisms (e.g., class embeddings, cross-attention, classifier-free guidance) are not described. If VideoGPT was run in its standard unconditional form, its per-class FVD values in Table 1 are not conditional-generation metrics; they are FVDs of unlabeled samples against class-specific reference sets, and the comparison conflates the ability to use the condition with video quality. Please specify the exact conditioning setup for every baseline, and ideally include an unconditional variant of EndoGen as a control; without this, the state-of-the-art claim is not established.","section":"Sec. 3.2, Tables 1-3"},{"comment":"No error bars, standard deviations, or repeated-run statistics are reported for any metric. The FVD gap between EndoGen (507.2) and VideoGPT (1548.6) in Table 1 is large, but the ablation gap between pmax=0.2 (514.8) and pmax=0.3 (507.2) in Table 4 is small; without variance estimates the reader cannot judge whether either difference is reliable. Please report at least three random seeds with mean and standard deviation, or equivalent interval estimates, for the main comparisons and ablations.","section":"Sec. 3.2, Tables 1-4"},{"comment":"All generation quality is capped by a frozen VQGAN pretrained on ImageNet, yet the paper provides no evaluation of how faithfully this encoder-decoder reconstructs endoscopic grid-frame images. This concern is amplified because SGP may produce large grid images (e.g., 16 frames at 128x128 arranged in a grid) that differ in resolution and content from the VQGAN's training distribution. Please report reconstruction FID/LPIPS on endoscopic grids, codebook usage statistics, and either finetune the tokenizer on endoscopic data or justify the frozen choice with concrete evidence.","section":"Sec. 3.1, implementation details and Sec. 2.1"},{"comment":"The HyperKvasir split for polyp segmentation is described as 1,000 images with an 8:2 train-test ratio, but no information is given about whether the videos used to train EndoGen overlap with the segmentation test images. If the generator saw the test-distribution frames, the synthetic unlabeled data could leak information and inflate the segmentation gains. In addition, only EndoGen synthetic frames are tested; to support the claim that the generated data are useful, include synthetic data from at least one baseline generator (e.g., VDM or VideoGPT) under the same semi-supervised protocol.","section":"Sec. 3.3, downstream polyp segmentation"}],"minor_comments":[{"comment":"The notation max(sigma^2_i) is ambiguous: specify whether the maximum is taken over segments in the batch, over the dataset, or over channels, and define sigma^2_i consistently.","section":"Sec. 2.2, Eq. (1)"},{"comment":"The abbreviations Bar., Ecta., and Perf. are not expanded in the caption or the text; please define them in the caption or use the full condition names.","section":"Table 1"},{"comment":"The sentence 'SurgVisdom [31] contains surgical videos on porcine model' contains a grammatical error; also consider giving the official challenge name and a brief description of the video content.","section":"Sec. 3.1"},{"comment":"Figure 1 is referenced but not analyzed in the text; please add a sentence describing what the reader should observe in the generated frames at different resolutions and frame counts.","section":"Fig. 1"},{"comment":"The 'w/o SGP' ablation is described only as 'a simple 2D reshaping of the video sequence'; please specify the exact reshaping operation (e.g., concatenating frames along width or height) so the ablation is reproducible.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision or medical imaging venue, but the strength of the claims ('first', 'state-of-the-art', 'significant margins') currently outruns the experimental support. The main missing pieces are conditioned baseline controls, uncertainty estimates, and a demonstration that the frozen VQGAN is adequate for endoscopic grid inputs. These are substantial but fixable within the manuscript's scope, hence my major-revision recommendation rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real step forward—nobody else has conditioned endoscopic video generation on pathology labels—and the SGP grid-frame trick is a clean way to reuse an autoregressive image model for video. The downstream segmentation experiment is also the right kind of evidence. But the experimental section has a hole the authors need to fix before the headline claim can stand: the per-class FVD comparison doesn't tell us how any of the baselines were conditioned, and VideoGPT is unconditional in its original form. If VideoGPT and the others were not given class labels, the comparison is between conditional generation and unconditional generation, and the reported margin mostly reflects that difference. That's not a fatal flaw in the method, but it is a fatal flaw in the evidence as presented.\n\nWhat's genuinely new: SGP maps a video to a grid image and runs a LlamaGen-style autoregressive model over the tokens, which is simple and effective; the ablation shows it is doing heavy lifting. SAT—variance-based masking—is a minor embellishment but seems to help modestly. The paper is honest about the frozen VQGAN and reports code release. The citation pattern looks fine; self-citations are to relevant prior work.\n\nSoft spots, in order of severity. First, no error bars or seeds anywhere; FVD differences of 500–1000 points could be noise with small per-class test sets. Second, the downstream evaluation uses synthetic unlabeled frames from a model trained on the same HyperKvasir data the segmentation test split is drawn from, so distribution overlap could inflate the gains. Fix: split the generator's training subjects from the test subjects. Third, the frozen ImageNet VQGAN is a plausible quality cap; a quick codebook analysis on endoscopic frames would settle whether it captures endoscopy-specific textures.\n\nBottom line: the method is promising, the writing is clear, and the limitations are addressable. The paper deserves a serious referee, but the referee should demand controlled baseline conditioning, repeated-run statistics, and a leakage-free downstream setup.","headline":"First credible conditional endoscopic video generator, but the baseline-conditioning gap undermines the SOTA claim until fixed.","tokens_in":8256,"tokens_out":2938,"would_cite":true,"duration_ms":33926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An autoregressive transformer that packs video frames into a grid and masks low-information tokens can generate condition-specific endoscopic videos, beating prior baselines on all eight HyperKvasir findings and improving semi-supervised…","keywords":["conditional video generation","endoscopy","autoregressive models","spatiotemporal grid-frame patterning","semantic-aware token masking","semi-supervised learning","polyp segmentation","Fréchet Video Distance"],"falsifier":"Encode real HyperKvasir frames through the same frozen VQGAN grid-patterning path EndoGen uses, decode immediately, and compare the reconstruction with the original frames using FID or LPIPS; if that reconstruction gap is as large as the quality gap between EndoGen and VideoGPT, then the tokenizer, not the autoregressive model, sets the quality ceiling.","tokens_in":7396,"feed_emoji":"🩺","tokens_out":13230,"duration_ms":119854,"temperature":0.7,"pith_summary":"The paper introduces EndoGen as the first conditional endoscopic video generation framework: instead of producing arbitrary or static endoscopic content, it takes a condition label for a pathological finding or surgical task and generates a temporally coherent video clip for that condition. The central claim is that an autoregressive transformer can do this well if the video frames are rearranged into a single grid image, so the model attends over space and time in one pass, and if training stochastically masks low-information tokens so the model concentrates on diagnostically meaningful content. On HyperKvasir, the authors report the best Fréchet Video Distance on all eight pathological findings, with an average of 507.2 versus 1548.6 for the best prior baseline, and on SurgVisdom they report the best results on all three surgical tasks. They further show that synthetic polyp frames generated by EndoGen can replace real unlabeled data in semi-supervised polyp segmentation and improve it further when combined with real data. If these results hold, the framework offers a practical route to on-demand, pathology-specific endoscopic video for training simulators, diagnostic-system development, and data augmentation in low-annotation regimes.","feed_headline":"Autoregressive model beats diffusion on endoscopy videos","feed_subtitle":"Conditional generation produces sharper pathology-specific clips that also improve semi-supervised polyp segmentation.","key_machinery":"The load-bearing machinery has two parts. Spatiotemporal Grid-Frame Patterning (SGP) converts a video clip into one composite image by arranging its frames row by row, then treats that grid as an image for the VQGAN encoder and the autoregressive transformer; the transformer's attention can therefore model spatial detail and temporal continuity simultaneously, and the decoder re-splits the generated grid into a video. Semantic-Aware Token Masking (SAT) partitions the tokenized features into segments, computes each segment's variance across the channel dimension, and applies a Bernoulli mask with ratio $p_i = \\mathrm{Clamp}\\big((1 - \\sigma_i^2/\\max_i \\sigma_i^2)\\,p_{\\max},\\,0,\\,p_{\\max}\\big)$, where $\\sigma_i^2$ is the segment variance, so low-variance, information-poor tokens are masked during training. A learnable condition embedding is prepended as the starting token, and training uses a standard cross-entropy next-token loss. The ablation study shows SGP carries most of the gain: replacing the grid layout with a plain 2D reshaping of the video raises FVD from 507.2 to 2617.5, while removing SAT raises it to 562.0.","core_discovery":"The discovery the paper argues for is that the long-range conditional modeling of autoregressive transformers transfers from images to endoscopic video without specialized temporal modules, provided the video is re-represented as a grid. EndoGen stitches the frames of a clip into a grid image, encodes it with a frozen VQGAN, prepends a learnable condition token that encodes the desired class, and generates the remaining grid tokens autoregressively before decoding and re-splitting them into frames. Semantic-Aware Token Masking modifies the training objective so that segments with low channel variance are masked with an adaptive probability, steering capacity toward tokens that carry clinically relevant structure such as lesions or instruments. The paper reports that this system outperforms diffusion-based and autoregressive baselines on every condition tested: all eight pathological findings of HyperKvasir, with an average FVD of 507.2 compared with 1548.6 for VideoGPT, and all three surgical tasks of SurgVisdom; it also reports lower FID and LPIPS. In the downstream study, generated polyp frames serve as unlabeled data for semi-supervised segmentation, matching or improving on real unlabeled data, with 70.96% versus 70.80% Dice for FixMatch when synthetic data replaces real data.","pith_inferences":["The grid ordering in SGP is fixed as row-by-row, which determines which frames are spatial neighbors for the transformer; varying or learning that ordering is a plausible way to improve temporal coherence further, but the paper does not explore it.","Because the VQGAN is frozen and pretrained on general images, its codebook is the one component not adapted to endoscopy; if it is the bottleneck, an endoscopy-tuned tokenizer should lower FVD further, which would also clarify how much of the reported gain is due to SGP and SAT rather than tokenizer capacity.","SAT uses channel variance as a proxy for semantic richness; masking guided by segmentation maps or pathology annotations could push generation toward clinically critical regions even more directly, at the cost of needing labels during training.","The downstream evaluation covers only polyp segmentation, so the same synthetic-data substitution logic remains untested for other findings, for detection, and for classification; those tests would show whether the benefit generalizes."],"forward_implications":["Supplying only a condition token lets EndoGen generate video for a specified finding or surgical task, so clinicians or training systems can request a particular pathology instead of searching through unconditionally generated footage.","Synthetic frames can substitute for real unlabeled data in semi-supervised polyp segmentation: with FixMatch, synthetic-only unlabeled data reaches 70.96% Dice versus 70.80% with real unlabeled data, and combining both reaches 71.03%.","Because the pipeline reuses a frozen VQGAN and an ImageNet-pretrained autoregressive image generator, the framework inherits the scalability of transformer image generation and is reported to extend to 64-frame clips and 256×256 resolution in the supplementary material.","The reported margins over diffusion baselines on FVD, CD-FVD, FID, and LPIPS suggest that, for endoscopic video, an autoregressive generator with a grid-frame representation is a competitive alternative to video diffusion models."],"supporting_citations":[{"why":"Supplies the HyperKvasir dataset with eight pathological findings used for the main conditional generation evaluation.","marker":"[1]"},{"why":"Frozen VQGAN tokenizer that encodes the grid-frame arrangement into discrete tokens and decodes generated tokens back into video frames.","marker":"[5]"},{"why":"Video diffusion model baseline that EndoGen compares against and outperforms in FVD on both datasets.","marker":"[8]"},{"why":"PolypMixNet, the semi-supervised polyp segmentation method used to test whether EndoGen synthetic frames improve downstream performance.","marker":"[9]"},{"why":"FixMatch, the semi-supervised learning baseline used to evaluate synthetic frames as substitutes for real unlabeled data.","marker":"[20]"},{"why":"Autoregressive image generator that supplies the transformer architecture, the learnable condition-token scheme, and the ImageNet initialization used by EndoGen.","marker":"[21]"},{"why":"Defines Fréchet Video Distance, the primary metric used to report conditional video generation quality.","marker":"[24]"},{"why":"SimDA diffusion adapter baseline that EndoGen beats in the head-to-head conditional generation comparisons.","marker":"[26]"},{"why":"VideoGPT, the autoregressive video generation baseline whose approach EndoGen extends to conditional endoscopic video.","marker":"[28]"},{"why":"Supplies the SurgVisdom surgical video dataset with three tasks used for the second evaluation setting.","marker":"[31]"}],"fun_headline_variants":["Autoregressive model beats diffusion on endoscopy videos","EndoGen: Autoregressive videos beat diffusion baselines","Grid-based autoregression yields sharper endoscopic video generation","Conditional autoregressive endoscopy videos improve polyp segmentation","Autoregressive generation tops diffusion for endoscopic clips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"If the frozen image tokenizer trained on ordinary photographs distorts endoscopy textures once frames are packed into a grid, then every generated video is capped at that distortion, and the paper does not measure the tokenizer's reconstruction fidelity on real endoscopic frames.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive model beats diffusion on endoscopy videos","EndoGen: Autoregressive videos beat diffusion baselines","Grid-based autoregression yields sharper endoscopic video generation","Conditional autoregressive endoscopy videos improve polyp segmentation","Autoregressive generation tops diffusion for endoscopic clips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3101,"prompt_tokens":983,"completion_tokens":2118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2042}},"tokens_in":599,"tokens_out":2118,"duration_ms":17049,"temperature":1.0,"reasoning_tokens":2042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:48:01.680361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Encode real HyperKvasir frames through the same frozen VQGAN grid-patterning path EndoGen uses, decode immediately, and compare the reconstruction with the original frames using FID or LPIPS; if that reconstruction gap is as large as the quality gap between EndoGen and VideoGPT, then the tokenizer, not the autoregressive model, sets the quality ceiling.","supporting_citations":[{"cited_title":"Sci- entific data 7(1), 283 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the HyperKvasir dataset with eight pathological findings used for the main conditional generation evaluation."},{"cited_title":"Advances in Neural Information Processing Systems35, 8633– 8646 (2022)","cited_arxiv_id":null,"evidence_quote":"Video diffusion model baseline that EndoGen compares against and outperforms in FVD on both datasets."},{"cited_title":"Computers in Biology and Medicine170, 108006 (2024)","cited_arxiv_id":null,"evidence_quote":"PolypMixNet, the semi-supervised polyp segmentation method used to test whether EndoGen synthetic frames improve downstream performance."},{"cited_title":"Advances in neural information processing systems33 (2020)","cited_arxiv_id":null,"evidence_quote":"FixMatch, the semi-supervised learning baseline used to evaluate synthetic frames as substitutes for real unlabeled data."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"SimDA diffusion adapter baseline that EndoGen beats in the head-to-head conditional generation comparisons."}],"review_version":1}