REVIEW 4 major objections 6 minor 1 cited by
Render-FM: Feedforward Model for Real-time Photorealistic Volumetric Rendering
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that a single feedforward network can predict all 6D Gaussian Splatting parameters from a CT volume in about 2.8 seconds, producing interactive photorealistic renderings without any per-scan optimization.
desk verdict A genuinely useful feedforward 6DGS renderer for CT with a real 500x speedup, but the 'generalizes to unseen anatomies' claim outruns what the mask-limited experiments actually test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the 6D Gaussian Splatting parameter volume combined with Anatomy-Guided Priming. The network predicts, for every foreground voxel, a covariance matrix in 6D spatio-angular space, a mean direction, spherical-harmonic color terms, and an opacity offset; during rendering the 6D covariance is sliced according to the viewing direction to produce an effective 3D Gaussian whose shape and opacity depend on viewpoint, which captures the view-dependent effects seen at tissue interfaces. Anatomy-Guided Priming supplies positions from voxel world coordinates, base color and opacity from the transfer function, and a foreground filter plus class label from the segmentation mask, so the prediction task is reduced to residual offsets rather than absolute appearance.
What would settle it
Take a single CT scan, run a standard pretrained segmentation tool to get a mask, then render with Render-FM. Deliberately erase one well-segmented organ, such as a kidney, from the mask and render again; if the kidney still appears, the mask is not actually gating the Gaussians, and if a corrupted mask causes visible missing anatomy, the dependency is confirmed. A quantitative version would measure how rendering SSIM and PSNR degrade as the mask is eroded or dilated by increasing amounts, showing whether anything besides mask accuracy controls output quality.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that volumetric rendering of CT data can be treated as a parameter regression problem rather than an optimization problem. The network takes a six-channel input volume (normalized CT intensity, a segmentation mask, and transfer-function RGBA values) and outputs 37 channels per voxel that encode Gaussian direction, spherical-harmonic color coefficients, an opacity offset, and the elements of a 6x6 covariance matrix. Only voxels marked as foreground by the segmentation mask instantiate Gaussians, and a differentiable tile-based rasterizer with view-dependent covariance slicing renders the result and provides training gradients. Trained on a corpus of CT scans with physically-based renderings as targets, the model matches or exceeds the quality of scan-specific 6DGS optimization on in-domain and out-of-domain test sets while cutting preparation time from roughly 1,500 seconds to about 2.8 seconds.
Load-bearing premise
The load-bearing premise is that the input segmentation mask is complete and correct, because Gaussians are created only at voxels the mask labels as foreground and any structure it misses is absent from every rendered view.
Editorial extensions
If this is right
- A CT volume can be turned into an interactive 3D visualization in seconds instead of roughly half an hour, opening real-time use in surgical planning and diagnostic review.
- The same trained network generalizes to unseen CT scans and acquisition protocols without retraining, because it has learned priors about anatomical appearance rather than fitting one scan.
- Transfer functions can be swapped at inference by changing the RGBA input channels, so a scan can be re-styled for different clinical views without reprocessing.
- Organ composability comes for free: because each Gaussian carries a semantic label from the mask, structures such as the skeleton can be shown or hidden with zero additional preparation time.
- A brief fine-tuning step of 89 to 140 seconds improves quality beyond per-scan optimized baselines, giving users a fast refinement option when maximal fidelity is needed.
Reading between the lines
- The paper's practical ceiling is set by the segmentation mask, not the renderer: any structure the mask misses never appears, so in real clinical use an inaccurate or incomplete auto-segmentation would silently delete anatomy from the image.
- Because Gaussians are instantiated at half-resolution foreground voxels, very fine or very large structures may stress the fixed-resolution parameter volume; an adaptive multi-resolution variant would be a natural test of whether the feedforward approach scales to sub-millimeter vasculature or whole-body scans.
- The same architecture could plausibly be retrained for other modalities or for time-resolved 4D volumes by changing the input channels and the ground-truth rendering pipeline, though the paper does not demonstrate this.
- The measured quality margins over per-scan optimization are narrow in some out-of-domain conditions, so the headline advantage is speed and generalization rather than a large fidelity gain; the more meaningful clinical comparison would be against expert-tuned direct volume rendering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Render-FM, a 3D U-Net that maps a 6-channel CT volume (density, segmentation mask, and RGBA values derived from transfer functions) to a dense 37-channel tensor of 6D Gaussian Splatting parameters. Gaussian primitives are instantiated only at foreground voxels of the segmentation mask, and a differentiable 6DGS rasterizer is used for end-to-end training against PBRT-rendered ground-truth views. The model is trained on 991 TotalSegmentator scans and evaluated on 46 in-domain and 10 CT-ORG scans. The main reported result is a preparation-time reduction from 1463.9 seconds (6DGS optimization) to 2.8 seconds for a forward pass, with SSIM/PSNR at or near per-scan optimized 6DGS levels, plus optional fine-tuning that further improves quality. The paper also introduces Anatomy-Guided Priming (AGP) and demonstrates compositional organ visualization.
Significance. Render-FM addresses a real obstacle for neural rendering in medical imaging: per-scan optimization. The feed-forward formulation and the roughly 500x speedup are significant if the reported quality holds, and the use of 6DGS for view-dependent medical volume effects is well motivated. The paper is strong in using public datasets, comparing against a strong per-scan baseline (6DGS), and measuring both quality and rendering speed. The central generalization and 'surpassing' claims are, however, broader than the evidence supports. The evaluation is confined to structures present in the segmentation mask and among the 11 consolidated semantic groups, so 'unseen anatomies' is not tested for anatomy outside that label set. With the scope clarified and the identified ablations and error analyses added, the contribution would be a useful step toward practical real-time medical volume rendering.
major comments (4)
- [§3.3, §4.1] Gaussian instantiation is restricted to voxels marked foreground by the segmentation mask, and both the ground-truth and predicted renderings are formed with zero opacity outside the 11 consolidated semantic groups. The reported SSIM, PSNR, and LPIPS therefore measure fidelity only for label-covered structures; a pulmonary nodule, intervertebral disc, or lesion outside TotalSegmentator's label set is absent from both the target and the predicted image, so the metrics cannot detect a silently missing structure. The Abstract's claim that Render-FM 'generalizes to unseen anatomies' is unsupported for any anatomy outside the consolidated groups. Please either limit the claims to the covered structures or add an experiment with unlabeled/pathological anatomy and report whether such structures are rendered at all.
- [Table 1; Abstract; §4.2] In the TotalSeg in-domain seen-TF row, Render-FM's LPIPS (0.097) is worse than 6DGS (0.096), so the statement that Render-FM is 'surpassing the 6DGS baseline' with these metrics is not correct, and the Abstract's 'surpassing per-scan optimized baselines' is at least ambiguous. The differences between the methods are small (e.g., SSIM 0.919 vs 0.912, PSNR 27.30 vs 26.63), and Table 1 reports no variance or per-scan statistics across the 46 TotalSeg and 10 CT-ORG test scans. Please report confidence intervals or per-scan paired comparisons and correct the wording to 'comparable on LPIPS and better on SSIM/PSNR' unless statistical significance is established.
- [§3.2, §4.2] Anatomy-Guided Priming is presented as a key contribution, but the controlled evidence for it is limited to 6DGS vs 6DGS+AGP in Table 1. No ablation removes the mask and RGBA input channels from Render-FM or replaces the anatomically primed initialization with a generic alternative; as a result, the contribution of AGP within the feedforward model itself is not demonstrated. Please add ablations of the 6 input channels and of the priming mechanism, reporting the same metrics on at least one test set.
- [§3.5, §4.1] The inference pipeline relies on an external segmentation mask (TotalSegmentator when none is provided), and the training masks are also TotalSegmentator outputs. There is no experiment on segmentation errors, even though any voxel not in the mask is silently excluded from the scene. Since the clinical-value claim depends on not omitting relevant anatomy, please evaluate robustness to mask imperfections (e.g., erased labels, eroded masks, or a different segmenter) or explicitly state in the abstract and conclusion that the method renders only structures identified by the input mask.
minor comments (6)
- [§4.1] The text says TotalSegmentator covers 117 anatomical classes while the cited paper [30] describes 104 structures; please reconcile the count.
- [§4.1, Unseen TF] The 'Unseen TF' experiment does not explain how novel transfer functions enter the network input during inference; please specify which RGBA inputs are fed to the model for unseen transfer functions.
- [§3.2, Figure 2] The network outputs parameters at half the input resolution, but the text and figure do not clarify how this subsampling affects small structures; please explain the spatial correspondence between output voxels and full-resolution anatomy.
- [§4.2] The fine-tuning times for a fixed 300 iterations vary across settings (89.4 s, 133.4 s, 136.2 s, 140.1 s); please explain the source of variation.
- [General] No code or trained model is provided; for a foundation-model paper, releasing the model or a demo would substantially strengthen reproducibility.
- [Related Work] The paper cites the authors' prior DDGS-CT work but does not explicitly state what new architectural components distinguish Render-FM from DDGS-CT beyond the foundation-model-style pretraining; please clarify the incremental contribution.
Circularity Check
No significant circularity: Render-FM is a supervised feed-forward regressor whose rendering quality is measured against independently rendered PBR ground truth, not derived from its inputs by construction.
full rationale
The paper's central claim is empirical: a 3D U-Net regresses 6DGS parameters from a 6-channel CT volume and is trained end-to-end with an L1 plus MS-SSIM rendering loss against PBR ground-truth images. The loss compares the rendered image to an independently computed target, so the reported SSIM/PSNR/LPIPS values are not forced by construction; the network could fail to match the target. The fact that the input includes the segmentation mask and transfer-function-derived RGBA, while the ground truth is also generated from the same mask and transfer functions, is a deliberate conditioning setup for a supervised rendering task, not a self-definitional derivation: the input does not already contain the output image, and the rendering operation (covariance slicing, rasterization, alpha compositing) is a non-trivial learned mapping. The paper's citations to the authors' prior 6DGS and DDGS-CT work are used as the rendering formulation and as a per-scan optimization baseline, not as an external uniqueness theorem that forbids alternatives; the comparison is an empirical head-to-head measurement. The Limitations section explicitly acknowledges missing relighting and future clinical deployment, and the mask-filtered evaluation limits the scope of the 'unseen anatomy' generalization claim, but that is an evaluation-scope concern rather than circularity. Overall, no load-bearing step reduces to its inputs by definition or by self-citation, so the circularity score is minimal.
Assumptions & free parameters
free parameters (2)
- Loss weights lambda_L1 and lambda_SSIM =
not reported
- Semantic group consolidation and transfer function tables =
11 groups and TF points in Table S3
assumptions (3)
- domain assumption The differentiable 6DGS renderer with view-dependent covariance slicing correctly models relevant view-dependent effects for medical CT data.
- domain assumption Physically based rendering (PBRT) outputs are a valid ground truth for photorealistic clinical visualization.
- domain assumption Segmentation masks from TotalSegmentator or similar tools are accurate enough to cover all structures that should appear in the rendering.
Cite this review
Pith. "Pith review of Render-FM: Feedforward Model for Real-time Photorealistic Volumetric Rendering." pith.science (2026). https://pith.science/paper/A27AWBTZ
@misc{pith2026250517338,
author = {Pith},
title = {Pith review of: Render-FM: Feedforward Model for Real-time Photorealistic Volumetric Rendering},
year = {2026},
howpublished = {\url{https://pith.science/paper/A27AWBTZ}},
note = {Machine review of arXiv:2505.17338}
}
read the original abstract
Photorealistic volumetric rendering of CT scans greatly benefits clinical workflows, yet neural approaches such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) require prohibitive per-scan optimization (hours for NeRF, about 30 minutes for 3DGS), making them impractical in clinical settings. We propose Render-FM, a feedforward model that eliminates this bottleneck by directly regressing 6D Gaussian Splatting (6DGS) parameters from a CT volume in a single 2.8-second forward pass, a 500x speedup over per-scan optimization. To bridge the domain gap between natural scene reconstruction and medical volumetric rendering, we introduce Anatomy-Guided Priming (AGP), which incorporates segmentation masks and transfer functions as structural and appearance priors, information that existing Gaussian splatting methods overlook. Built on an nnU-Net-inspired 3D U-Net trained on diverse CT scans, Render-FM predicts per-voxel 6DGS parameters and supports immediate real-time rendering. Unlike per-scan methods, it generalizes to unseen anatomies, novel transfer functions, and enables compositional organ visualization with zero additional preparation time. Optional 89-second fine-tuning further improves quality, surpassing per-scan optimized baselines. Project page: https://gaozhongpai.github.io/renderfm/.
Figures
Forward citations
Cited by 1 Pith paper
-
Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy
A hybrid VR renderer combines foveated volumetric path tracing with a rapidly regenerated Gaussian-splatting periphery for interactive medical anatomy visualization.
Reference graph
Works this paper leans on
-
[1]
Johanna Beyer, Markus Hadwiger, Stefan Wolfsberger, and Katja Bühler. High-quality mul- timodal volume rendering for preoperative planning of neurosurgical interventions.IEEE Transactions on Visualization and Computer Graphics, 13(6):1696–1703, 2007
work page 2007
-
[2]
Hudson, Ehsan Adeli, Russ Altman, and et al
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, and et al. On the Opportunities and Risks of Foundation Models.arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[3]
Structure-Aware Sparse-View X-ray 3D Reconstruction (SAX-NeRF)
Yuanhao Cai, Jiahao Wang, Alan Yuille, Zongwei Zhou, and Angtian Wang. Structure-Aware Sparse-View X-ray 3D Reconstruction (SAX-NeRF). InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2024
work page 2024
-
[4]
M Travis Caton Jr, Walter F Wiggins, and Diego Nunez. Three-dimensional cinematic rendering to optimize visualization of cerebrovascular anatomy and disease in ct angiography.Journal of Neuroimaging, 30(3):286–296, 2020
work page 2020
-
[5]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.IEEE transactions on pattern analysis and machine intelligence, 40(4): 834–848, 2017
work page 2017
-
[6]
Evelyn Dappa, Kai Higashigaito, Jürgen Fornaro, Sebastian Leschka, Simon Wildermuth, and Hatem Alkadhi. Cinematic rendering–an alternative to volume rendering for 3d computed tomography imaging.Insights into imaging, 7:849–856, 2016
work page 2016
-
[7]
Marwen Eid, Carlo N De Cecco, John W Nance Jr, Damiano Caruso, Moritz H Albrecht, Adam J Spandorfer, Domenico De Santis, Akos Varga-Szemes, and U Joseph Schoepf. Cinematic rendering in ct: a novel, lifelike 3d visualization technique.American Journal of Roentgenology, 209(2):370–379, 2017
work page 2017
-
[8]
Moustafa Elshafei, Johannes Binder, Justus Baecker, Maximilian Brunner, Michael Uder, Georg F Weber, Robert Grützmann, and Christian Krautz. Comparison of cinematic rendering and computed tomography for speed and comprehension of surgical anatomy.JAMA surgery, 154(8):738–744, 2019
work page 2019
Show all 32 references
-
[9]
Maier-Hein
Alexandra Ertl, Shuhan Xiao, Stefan Denner, Robin Peretzke, David Zimmerer, Peter Neher, Fabian Isensee, and Klaus H. Maier-Hein. nnLandmark: A Self-Configuring Method for 3D Medical Landmark Detection.arXiv preprint arXiv:2504.06742, 2025
2025
-
[10]
Flashgs: Efficient 3d gaussian splatting for large-scale and high-resolution rendering.arXiv preprint arXiv:2408.07967, 2024
Guofeng Feng, Siyan Chen, Rong Fu, Zimu Liao, Yi Wang, Tao Liu, Zhilin Pei, Hengjie Li, Xingcheng Zhang, and Bo Dai. Flashgs: Efficient 3d gaussian splatting for large-scale and high-resolution rendering.arXiv preprint arXiv:2408.07967, 2024
2024 arXiv
-
[11]
Ddgs-ct: Direction-disentangled gaussian splatting for realistic volume rendering
Zhongpai Gao, Benjamin Planche, Meng Zheng, Xiao Chen, Terrence Chen, and Ziyan Wu. Ddgs-ct: Direction-disentangled gaussian splatting for realistic volume rendering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[12]
6DGS: Enhanced Direction-Aware Gaussian Splatting for V olumetric Rendering.arXiv preprint arXiv:2410.04974, 2024
Zhongpai Gao, Benjamin Planche, Meng Zheng, Anwesa Choudhuri, and Terrence Chen. 6DGS: Enhanced Direction-Aware Gaussian Splatting for V olumetric Rendering.arXiv preprint arXiv:2410.04974, 2024
2024 arXiv
-
[13]
Gpu-based volume rendering for medical image visualization
Yang Heng and Lixu Gu. Gpu-based volume rendering for medical image visualization. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pages 5145–5148. IEEE, 2006
2005
-
[14]
Jäger, Simon A
Fabian Isensee, Paul F. Jäger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18:203–211, 2021
2021
-
[15]
nnu-net revisited: A call for rigorous validation in 3d medical image segmentation
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. InInternational Conference on Medical Image Computing and Computer- Assist...
2024
-
[16]
nninteractive: Redefining 3d promptable segmentation.arXiv preprint arXiv:2503.08373, 2025
Fabian Isensee, Maximilian Rokuss, Lars Krämer, Stefan Dinkelacker, Ashis Ravindran, Flo- rian Stritzke, Benjamin Hamm, Tassilo Wald, Moritz Langenberg, Constantin Ulrich, et al. nninteractive: Redefining 3d promptable segmentation.arXiv preprint arXiv:2503.08373, 2025
2025 arXiv
-
[17]
Younhyun Jung, Jinman Kim, Lei Bi, Ashnil Kumar, David Dagan Feng, and Michael Fulham. A direct volume rendering visualization approach for serial pet–ct scans that preserves anatomical consistency.International Journal of Computer Assisted Radiology and Surgery, 14:733–744, 2019
2019
-
[18]
3d gaussian splatting for real-time radiance field rendering.ACM Trans
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023
2023
-
[19]
Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[20]
Segment anything in medical images.Nature Communications, 15(1):654, 2024
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images.Nature Communications, 15(1):654, 2024
2024
-
[21]
Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021
2021
-
[22]
Foundation models for generalist medical artificial intelligence.Nature, 616(7956):259–265, 2023
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence.Nature, 616(7956):259–265, 2023
2023
-
[23]
Foundation models in radiology: What, how, why, and why not.Radiology, 314(2):e240597, 2025
Magdalini Paschali, Zhihong Chen, Louis Blankemeier, Maya Varma, Alaa Youssef, Christian Bluethgen, Curtis Langlotz, Sergios Gatidis, and Akshay Chaudhari. Foundation models in radiology: What, how, why, and why not.Radiology, 314(2):e240597, 2025
2025
-
[24]
MIT Press, 2023
Matt Pharr, Wenzel Jakob, and Greg Humphreys.Physically based rendering: From theory to implementation. MIT Press, 2023
2023
-
[25]
Ct-org, a new dataset for multiple organ segmentation in computed tomography.Scientific Data, 7(1): 381, 2020
Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org, a new dataset for multiple organ segmentation in computed tomography.Scientific Data, 7(1): 381, 2020
2020
-
[26]
Structure-from-motion revisited
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016
2016
-
[27]
Lgm: Large multi-view gaussian model for high-resolution 3d content creation.arXiv preprint arXiv:2402.05054, 2024
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation.arXiv preprint arXiv:2402.05054, 2024
2024 arXiv
-
[28]
Bovik, Hamid R
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE Transactions on Image Processing, 13(4): 600–612, 2004
2004
-
[29]
Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4): 600–612, 2004
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4): 600–612, 2004
2004
-
[30]
Totalseg- mentator: robust segmentation of 104 anatomic structures in ct images.Radiology: Artificial Intelligence, 5(5):e230024, 2023
Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalseg- mentator: robust segmentation of 104 anatomic structures in ct images.Radiology: Artificial Intel...
2023
-
[31]
Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation
Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. InEuropean Conference on Computer Vision, pages 1–20. Springer, 2024
2024
-
[32]
The unrea- sonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 11 Supplementary Materi...
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.