{"id":"6e6e0bdb-fffa-4b5e-b31a-0ee13480a3ad","arxiv_id":"2411.12981","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GazeGaussian is a 3D Gaussian Splatting based gaze redirection method that separately models face and eyes and reports state-of-the-art redirection accuracy and image quality on ETH-XGaze, ColumbiaGaze, MPIIFaceGaze, and GazeCapture.","lead":"GazeGaussian uses two separate clouds of 3D Gaussian splats, one for the face and one for the eyes, to redirect a person's gaze in a photo toward a new direction while keeping their identity. The method reports sharper images, lower gaze error, and faster rendering than previous NeRF-based gaze redirection approaches, which matters for generating training data for gaze estimation and for virtual avatars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gaze accuracy gains may reflect optimizing and evaluating with the same ETH-XGaze gaze estimator, so the SOTA accuracy claim needs an independent, non-circular evaluation before it can be relied on.","rationale":"The reader's weakest assumption already identifies the estimator circularity, and my read agrees: this is the most load-bearing threat to the central claim. If the evaluation metric is not measuring true gaze direction, then the headline numbers in Tables 1 and 2, the cross-dataset generalization claims, and the downstream gaze-estimation gains in Fig. 6 all lose their evidential force. I considered the rendering-speed issue as an alternative: supplementary Sec. 10 admits a 30-second per-input fine-tuning step, which is not included in the 74 FPS figure, and the 0.2 s/image generation time is internally inconsistent with 74 FPS. This is a real secondary problem, but it mainly affects one of the three claimed advantages and is easy to repair by reporting amortized end-to-end cost. The metric circularity is more fundamental because it undermines the core accuracy contribution. I am not claiming the model is deliberately gaming the metric; the concern is structural, since Eq. (12) and the metric use the same estimator. A concrete independent evaluation would settle the question. The two-stream architecture, explicit eye-rotation field, and expression-guided renderer are well-motivated and the ablations support their individual contributions, so I would not reject the paper; the conditional verdict already appropriately requires such validation.","tokens_in":17281,"tokens_out":4824,"duration_ms":49630,"concrete_test":"Re-run the same input-target pairings as Tables 1 and 2, but compute the Gaze and Head Pose errors with an independent gaze estimator that was never used in training or loss—for example, a published estimator trained only on MPIIFaceGaze or GazeCapture, or a geometric iris/limbus fitting on the rendered eye region—and compare the margin between GazeGaussian and GazeNeRF. If GazeGaussian's advantage shrinks below noise or reverses under the independent estimator, the reported accuracy improvement is likely an artifact of optimizing Eq. (12) with the evaluation estimator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Tables 1 and 2) is that GazeGaussian achieves state-of-the-art gaze redirection accuracy. The training objective in Eq. (12) minimizes the angular error Eang between the rendered head image Ih and ground-truth Igt as judged by a pre-trained gaze estimator ψg, and Secs. 5.1 and 9 state that the same ResNet50-based estimator (trained on ETH-XGaze) is used to report all gaze and head-pose errors. This creates direct train/test metric overlap: the model is explicitly trained to reduce the exact quantity in which it is evaluated. Because 3DGS plus the neural renderer has substantial capacity, it can satisfy ψg by generating estimator-friendly eye appearances rather than by geometrically rotating the eyeball to the target gaze. The circularity is not fully mitigated by the fact that GazeNeRF also uses this loss, since the two models have different capacities to exploit the estimator. The same estimator is also applied across ColumbiaGaze, MPIIFaceGaze, and GazeCapture, so cross-dataset claims inherit the estimator's domain bias. Section 5.5's downstream experiment is likewise evaluated with the same estimator, making the improvement loop partly self-referential. The paper provides no independent geometric ground truth or a second, held-out estimator to break this loop.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GazeGaussian, a gaze redirection method built on 3D Gaussian Splatting (3DGS). It represents the face and eye regions with two separate streams of Gaussians, deforms the face branch with expression/head-pose conditioned MLPs, and models eye rotation via learned offsets and multiplications conditioned on gaze direction. An expression-guided neural renderer injects subject-specific latent codes to improve generalization. The method is evaluated on ETH-XGaze, ColumbiaGaze, MPIIFaceGaze, and GazeCapture, reporting improvements over baselines in gaze and head angular errors, image quality, identity preservation, and rendering speed (74 FPS). The paper claims to be the first 3DGS-based gaze redirection pipeline and to achieve state-of-the-art performance.","tokens_in":17552,"tokens_out":7727,"duration_ms":77536,"significance":"If the claims hold, GazeGaussian is a meaningful advance: it brings explicit, controllable 3DGS representations to gaze redirection, decouples eye and face deformations, and introduces a subject-conditional renderer that improves generalization across identities. The experimental scope is broad (four datasets), the ablations are systematic, and the code is promised. The main significance is tempered by a train/evaluation metric overlap: the gaze redirection loss and the reported gaze errors both rely on ETH-XGaze-trained estimator families, so the headline accuracy gains need independent confirmation before they can be relied upon.","major_comments":[{"comment":"The gaze redirection loss in Eq. (12) minimizes the angular error between the rendered and ground-truth images as judged by a VGG-based gaze estimator fine-tuned on ETH-XGaze (Supplementary Sec. 9). All reported gaze and head errors in Tables 1 and 2 use a ResNet50 gaze estimator trained on ETH-XGaze. The model is therefore explicitly trained to minimize the same quantity family on which it is evaluated, and both estimators share the same training-domain bias. With the high capacity of 3DGS plus a neural renderer, the model may satisfy the estimator by producing estimator-friendly eye appearances rather than by truly rotating the eyeball to the target gaze. This affects the central SOTA accuracy claim. Please add an independent evaluation that breaks this loop: for example, report gaze errors using a held-out estimator trained on a different dataset, or use synthetic/geometric ground truth for eye orientation. Without such a check, the reported accuracy numbers are not sufficient to support the paper's main claim.","section":"Sec. 4.4 (Eq. 12), Sec. 5.1, Supplementary Sec. 9"},{"comment":"The abstract and contributions describe the method as enabling 'rigid eye rotation' based on the target gaze. However, Eq. (4) computes the eye Gaussian center as µe = Ee_μ(µe_0, τ) + Ge_μ(µe_0, φ) µe_0, which is an MLP-predicted offset plus a learned affine transformation, with no orthogonality or rotation constraint on Ge_μ. Similarly, Eq. (5) applies learned scales/rotations to other attributes without enforcing rigidity. The method is better described as a learned, gaze-conditioned affine deformation of eye Gaussians, which is still an explicit spatial control but not a rigid rotation. Please revise the terminology, or change the formulation to explicitly predict and apply a rotation (e.g., a quaternion) to support the 'rigid eye rotation' claim.","section":"Abstract and Sec. 4.2.2 (Eqs. 4-5)"},{"comment":"The abstract highlights 'rendering speed' and Table 1 reports 74 FPS for GazeGaussian. However, Supplementary Sec. 10 states that 'during inference, GazeGaussian fine-tunes on a single input image, taking approximately 30 seconds for fine-tuning and 0.2 seconds per image for generation.' The reported FPS therefore measures only the forward rendering pass, not the end-to-end inference cost required for a new input image. This materially affects the comparison with baselines, which do not all require per-input fine-tuning. Please report the full per-image inference time (fine-tuning plus rendering) and either remove or qualify the 'rendering speed' claim in the abstract and Table 1.","section":"Abstract, Table 1, Supplementary Sec. 10"}],"minor_comments":[{"comment":"The phrase 'To access the generalization capability' should be 'To assess the generalization capability'.","section":"Sec. 5.3"},{"comment":"The rows in Table 3 are shown only with checkmarks, making it hard to map each row to the variant described in Sec. 5.4. Please add explicit row labels (e.g., 'Vanilla', 'w/o Eye Rep.', 'w/o Expression-Guided', 'w/o Two-stream', 'Ours') to the table.","section":"Table 3"},{"comment":"Sec. 5.1 says the metrics use 'the same ResNet50-based estimator in GazeNeRF', while Sec. 4.4 and Supplementary Sec. 9 reveal the loss uses a different VGG-based estimator fine-tuned on ETH-XGaze. The main text should clearly distinguish the loss network and the evaluation network, and state their training domains, to avoid the impression that the metric is the exact loss network.","section":"Sec. 5.1 and Sec. 4.4"},{"comment":"Eqs. (4)-(5) introduce MLPs Ee_μ and Ge_μ but do not specify their inputs' dimensionality or architecture. A sentence noting that the gaze direction φ and latent code τ are concatenated and fed into these MLPs would improve reproducibility.","section":"Sec. 4.2.2"},{"comment":"The ablation text refers to 'the first row of the table and the first column of the visualizations' without naming the variant; making the row-to-variant mapping explicit in the table and figure captions would help readers follow the ablation discussion.","section":"Sec. 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a novel and plausible 3DGS-based gaze redirection framework, but the central accuracy claim currently rests on an evaluation that shares the ETH-XGaze training distribution with the loss's gaze estimator. This is fixable with an additional independent evaluation. The per-input fine-tuning time of ~30 seconds should be prominently disclosed, as the 74 FPS headline may mislead readers. The 'rigid rotation' terminology should also be corrected. If the authors address these points with concrete experiments and text revisions, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dan,\n\nRead GazeGaussian. The core idea holds up: it is the first 3DGS-based gaze redirection pipeline, and the two-stream face/eye decomposition with a gaze-conditioned eye branch is a real contribution. The ablations show each component (two-stream, eye rep, expression-guided renderer) pulls its weight, and the cross-dataset numbers are consistently better than GazeNeRF. The expression-guided neural renderer with cross-attention is a nice idea, and the authors even show it helps the baselines in the supplementary. Credit also for shipping code and using standard benchmarks.\n\nThe soft spots are real but not fatal. The \"rigid eye rotation\" language in the abstract and Sec. 4.2.2 oversells Eq. (4-5): that is an MLP-predicted offset plus a gaze-conditioned scale, not a geometric rotation. It works, but it is a deformation, not a rotation. Second, the speed claim: Table 1 says 74 FPS, but the supplementary says inference is 0.2 s per image (5 FPS) after roughly 30 s of per-input fine-tuning. The 74 FPS number presumably comes after that initialization, so the \"rendering speed\" headline skips a large fixed cost. Third, the metric circularity: the loss uses a VGG-based gaze estimator trained on ETH-XGaze, and the evaluation uses a ResNet50 trained on ETH-XGaze. The stress-test note says \"the same estimator,\" which is not accurate—they are different architectures—but both are trained on the same dataset, so the model could be learning to satisfy that estimator family rather than actually rotating the eyeball. The qualitative results and the fact that GazeNeRF is evaluated under the same protocol mitigate this, but an independent evaluation with a held-out estimator or synthetic geometric ground truth would settle it. No error bars either, so take the small margins (e.g., 6.622 vs 6.944) with some salt.\n\nWho is this for: anyone working in gaze redirection or head avatars who wants a fast, explicit 3DGS alternative to NeRF-based methods. It is a serious piece of engineering, and the claims are mostly backed up. I would send it to reviewers—it is referee-worthy—but I would ask for a revision addressing the rotation overclaim, the inference-time caveat, and preferably an added independent evaluation.\n\nThanks.","headline":"A genuine and well-engineered 3DGS gaze redirection system whose novelty is real, but whose headline speed and SOTA accuracy claims need caveats about per-input finetuning and an ETH-XGaze-trained estimator appearing on both sides of the evaluation.","tokens_in":18106,"tokens_out":3711,"would_cite":true,"duration_ms":33579,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GazeGaussian, a two-stream 3D Gaussian splatting model that explicitly rotates eye Gaussians according to the target gaze direction, achieves state-of-the-art gaze redirection accuracy and rendering speed (74 FPS), outperforming existing…","keywords":["gaze redirection","3D Gaussian Splatting","gaze estimation","neural rendering","head avatar synthesis","face-eye disentanglement","data augmentation","expression-guided rendering"],"falsifier":"Evaluate redirected images from GazeGaussian and the baselines with a gaze estimator trained on a different dataset (e.g., MPIIFaceGaze or a synthetic rigged-eye dataset with known ground-truth gaze), and compare gaze angular errors; if GazeGaussian's advantage shrinks or disappears, the method is overfitting to the particular estimator used in training. Alternatively, compare the 3D rotation of the eye Gaussians against known ground-truth eye rotation from a calibrated multi-camera rig.","tokens_in":17077,"feed_emoji":"👁️","tokens_out":9150,"duration_ms":75222,"temperature":0.7,"pith_summary":"Gaze redirection changes the direction a person appears to look in a photo, and it is used to create training data for gaze estimators. The paper's central claim is that this can be done more accurately and far faster with 3D Gaussian splatting than with the neural radiance fields used by prior state-of-the-art methods. GazeGaussian represents the face and the eyes as two separate sets of 3D Gaussians, rotates the eye Gaussians explicitly according to the desired gaze direction, and uses an expression-guided renderer to keep each subject's identity intact. The authors report that this first 3DGS-based gaze redirection pipeline outperforms previous methods on redirection accuracy, image quality, identity preservation, and rendering speed across four datasets, reaching about 74 frames per second after a short per-image initialization. If correct, high-quality gaze redirection becomes a practical, real-time tool for improving gaze estimation on out-of-distribution data.","feed_headline":"Gaze redirection with 3D Gaussians beats NeRF on accuracy and speed","feed_subtitle":"A two-stream 3D Gaussian splatting model rotates eye geometry explicitly, reaching state-of-the-art accuracy at 74 FPS.","key_machinery":"The key mechanism is a two-stream 3D Gaussian Splatting (3DGS) model: a face-only stream deforms canonical Gaussians with MLPs conditioned on expression codes and head pose, while an eye stream introduces a Gaussian eye rotation representation that rotates eye Gaussians in canonical space according to the target gaze direction, with learned bias MLPs to absorb gaze-label noise. The transformed Gaussians from both streams are concatenated, rasterized into feature maps, and fed into an expression-guided neural renderer (EGNR) that injects subject-specific latent codes via cross-attention at the bottleneck. This explicit geometric control of the eye region is the feature that distinguishes the method from prior implicit feature-map-based approaches.","core_discovery":"The central discovery is that decoupling the head into a face-only Gaussian stream and an eye Gaussian stream, and explicitly rotating the eye Gaussians in 3D according to the target gaze vector, gives both more precise gaze redirection and more detailed face synthesis than implicit feature-map manipulation. The paper validates this on ETH-XGaze, ColumbiaGaze, MPIIFaceGaze, and GazeCapture, reporting lower gaze and head angular errors, higher SSIM, PSNR, LPIPS, FID, and identity-similarity scores than GazeNeRF, HeadNeRF, STED, and Gaussian Head Avatar, while rendering at 74 FPS. The two-stream design plus an expression-guided neural renderer is what makes both precise gaze control and cross-subject generalization possible.","pith_inferences":["A portion of the reported accuracy gain may come from the model fitting the specific gaze estimator used both in the loss function and in the evaluation; testing with a held-out estimator trained on a different dataset would reveal whether the geometric eye rotation itself is more accurate.","Because the eye rotation is explicit and parameterized by gaze direction, the learned rotation parameters could be read off as a direct 3D gaze estimate, potentially making the method serve as a gaze estimator as well as a synthesis tool.","The per-input initialization takes about 30 seconds, so a fully amortized version that avoids per-image fine-tuning would be needed for truly interactive applications; otherwise the 74 FPS figure applies only after that per-input step.","The two-stream recipe could generalize to other region-specific face edits (e.g., blinking, mouth movements) by assigning each region its own Gaussian stream and control signal."],"forward_implications":["Gaze redirection becomes fast enough (about 74 FPS after initialization) to serve as a real-time data augmentation engine for gaze estimators.","The face–eye disentanglement allows direct, controllable eye movement while preserving identity and expression, which is useful for avatar animation and content editing.","The expression-guided renderer generalizes to unseen subjects, addressing a known limitation of 3DGS head avatars that are typically person-specific.","The reported state-of-the-art results on four datasets suggest 3DGS is a practical alternative to NeRF for gaze-related synthesis and possibly other fine-grained face editing tasks."],"supporting_citations":[{"why":"Supplies the 3D Gaussian splatting representation that the two-stream model is built on.","marker":"[18]"},{"why":"GazeNeRF is the main NeRF-based baseline and provides the preprocessing and gaze redirection loss that the method adopts.","marker":"[36]"},{"why":"Gaussian Head Avatar contributes the neutral-mesh initialization and face-deformation machinery used in the face stream.","marker":"[51]"},{"why":"STED introduces the functional gaze redirection loss used in training.","marker":"[59]"},{"why":"ETH-XGaze is the primary training and within-dataset evaluation dataset with multi-view images and gaze labels.","marker":"[58]"},{"why":"MPIIFaceGaze is one of the cross-dataset generalization benchmarks.","marker":"[55]"},{"why":"ColumbiaGaze is one of the cross-dataset generalization benchmarks.","marker":"[39]"},{"why":"GazeCapture is one of the cross-dataset generalization benchmarks.","marker":"[21]"},{"why":"The face parsing model generates the eye and face masks used to supervise the separate streams.","marker":"[63]"}],"fun_headline_variants":["Two-stream 3D Gaussian gaze redirection hits 74 FPS","GazeGaussian: 3D Gaussians outrun NeRF for gaze redirection","Explicit eye rotation via 3D Gaussians boosts gaze fidelity","Gaze redirection at 74 FPS with 3D Gaussian splatting","Splatted gaze: faster, sharper redirection than NeRF"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the pre-trained gaze estimator used both as a training loss and as the accuracy metric (trained on ETH-XGaze) measures true gaze direction; if the model learns to exploit that estimator's blind spots rather than geometrically rotating the eyeball, the reported angular-error improvements would not reflect genuine redirection quality.","fun_headline_variants_meta":{"raw":{"variants":["Two-stream 3D Gaussian gaze redirection hits 74 FPS","GazeGaussian: 3D Gaussians outrun NeRF for gaze redirection","Explicit eye rotation via 3D Gaussians boosts gaze fidelity","Gaze redirection at 74 FPS with 3D Gaussian splatting","Splatted gaze: faster, sharper redirection than NeRF"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3312,"prompt_tokens":930,"completion_tokens":2382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2291}},"tokens_in":546,"tokens_out":2382,"duration_ms":15391,"temperature":1.0,"reasoning_tokens":2291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:58:17.527971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate redirected images from GazeGaussian and the baselines with a gaze estimator trained on a different dataset (e.g., MPIIFaceGaze or a synthetic rigged-eye dataset with known ground-truth gaze), and compare gaze angular errors; if GazeGaussian's advantage shrinks or disappears, the method is overfitting to the particular estimator used in training. Alternatively, compare the 3D rotation of the eye Gaussians against known ground-truth eye rotation from a calibrated multi-camera rig.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D Gaussian splatting representation that the two-stream model is built on."},{"cited_title":"Gazenerf: 3d-aware gaze redirection with neural radiance fields","cited_arxiv_id":null,"evidence_quote":"GazeNeRF is the main NeRF-based baseline and provides the preprocessing and gaze redirection loss that the method adopts."},{"cited_title":"Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians","cited_arxiv_id":null,"evidence_quote":"Gaussian Head Avatar contributes the neutral-mesh initialization and face-deformation machinery used in the face stream."},{"cited_title":"Self-learning transformations for improving gaze and head redirection","cited_arxiv_id":null,"evidence_quote":"STED introduces the functional gaze redirection loss used in training."},{"cited_title":"Eth-xgaze: A large scale dataset for gaze estimation under extreme head pose and gaze variation","cited_arxiv_id":null,"evidence_quote":"ETH-XGaze is the primary training and within-dataset evaluation dataset with multi-view images and gaze labels."},{"cited_title":"Appearance-based gaze estimation in the wild","cited_arxiv_id":null,"evidence_quote":"MPIIFaceGaze is one of the cross-dataset generalization benchmarks."},{"cited_title":"Smith, Q","cited_arxiv_id":null,"evidence_quote":"ColumbiaGaze is one of the cross-dataset generalization benchmarks."},{"cited_title":"Eye tracking for everyone","cited_arxiv_id":null,"evidence_quote":"GazeCapture is one of the cross-dataset generalization benchmarks."},{"cited_title":"Using modified bisenet for face parsing in pytorch,","cited_arxiv_id":null,"evidence_quote":"The face parsing model generates the eye and face masks used to supervise the separate streams."}],"review_version":1}