arxiv: 2604.23094 · v1 · submitted 2026-04-25 · 💻 cs.CV · cs.GR· cs.LG

Recognition: unknown

Toward Real-World Adoption of Portrait Relighting via Hybrid Domain Knowledge Fusion

Qian Huang , Mayoore Selvarasa Jaiswal , Zhen Zhong , Rochelle Pereira , Jianyuan Min

Authors on Pith no claims yet

Pith reviewed 2026-05-08 08:39 UTC · model grok-4.3

classification 💻 cs.CV cs.GRcs.LG

keywords portrait relightingknowledge distillationdomain adaptationsynthetic datasethybrid fusionlightweight modelreal-world adoptionone-light-at-a-time

0 comments

The pith

Hybrid Domain Knowledge Fusion transfers multi-domain expertise into a compact portrait relighting model for real-world use.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Portrait relighting struggles to move from labs to everyday use because of mismatches between synthetic training data and real camera captures, plus the high compute cost of accurate models. This paper proposes fusing knowledge from synthetic, one-light-at-a-time, and real-world datasets through domain-adapted prior models and then distilling that knowledge into a small student network. If successful, the resulting model runs dramatically faster while preserving high visual quality, making real-time relighting practical on consumer devices. The work also releases a large synthetic dataset with detailed ground-truth lighting information to aid further progress.

Core claim

The central claim is that specialized prior models trained on different domains can be adapted and their knowledge distilled into a lightweight student model that inherits multi-domain capabilities, achieving substantial inference speedups without sacrificing state-of-the-art quality on real inputs. This is enabled by the Hybrid Domain Knowledge Fusion paradigm and supported by a new large-scale synthetic dataset with diverse intrinsics.

What carries the argument

Hybrid Domain Knowledge Fusion, a two-stage process of domain-aware adaptation of prior models followed by augmented knowledge distillation to a compact student.

If this is right

The compact model enables real-time portrait relighting on edge devices.
Quality remains comparable to larger specialized models across domains.
The new synthetic dataset provides better training signals for intrinsic decomposition tasks.
Inference costs drop by factors of 6 to 240 times compared to prior approaches.
This fusion approach generalizes to other image synthesis tasks facing domain gaps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Applications in mobile photography apps could become feasible without cloud processing.
Future work might test the method's robustness to extreme lighting conditions not covered in the datasets.
The speedup could allow integration into video relighting pipelines for live streaming.
One might explore whether the same fusion technique reduces the need for large real-world capture setups in other vision tasks.

Load-bearing premise

The domain-aware adaptation and augmented knowledge distillation successfully pass on expertise from multiple domains to the student model with no significant drop in quality or introduction of artifacts when tested on real-world images.

What would settle it

Running the lightweight model on a held-out set of real-world portraits captured under varied camera conditions and lighting, and observing if its relit outputs match or exceed the visual fidelity of the original prior models without new artifacts.

Figures

Figures reproduced from arXiv: 2604.23094 by Jianyuan Min, Mayoore Selvarasa Jaiswal, Qian Huang, Rochelle Pereira, Zhen Zhong.

**Figure 1.** Figure 1: Our method relights portrait images (top) and videos (bottom) under arbitrary HDR environment maps in real time on consumer GPUs. Given inputs from different cameras, our model faithfully reproduces the directionality, color temperature, and complex illumination of diverse target environments (shown as insets) while preserving subject identity. Video relighting is temporally stable across frames. details. … view at source ↗

**Figure 2.** Figure 2: Overview of Hybrid Domain Knowledge Fusion. Stage 1: Three specialized prior models are trained on heterogeneous data sources—Pphys on synthetic data with unsupervised real-data adaptation via Lconsist, Pref l on synthetic and OLAT data with optical augmentations, and Preal initialized from Pref l and fine-tuned on real data. Stage 2: The frozen priors (P ∗ ) generate domain-routed pseudo-ground truth to s… view at source ↗

**Figure 3.** Figure 3: Qualitative comparison on the OLAT test set. From left to right: (a) input, (b) ground truth, (c) LUMOS [35], (d) DiffusionRenderer [17], (e) UniRelight [12], (f) IC-Light [37], (g) SwitchLight v3 [15], (h) 3D-Aware Relighting [6] (face crop only), and (i) Ours. Our results most closely match the ground truth in brightness, directionality, and skin tone across the various in-the-wild HDRs. LPIPS. Notably,… view at source ↗

**Figure 4.** Figure 4: Relighting fidelity comparison on portraits from cameras of varying quality, relit under a common target HDR environment. From left to right: input, target environment map, and results from DiffusionRenderer [17], UniRelight [12], SwitchLight v3 [15], and Ours. DiffusionRenderer introduces excessive specularity; SwitchLight v3 oversmooths skin and hair, yielding a synthetic appearance. Our method best pre… view at source ↗

**Figure 5.** Figure 5: Ablation of prior model components. Each row shows (left to right) the input, the output without the indicated component, and the full model output. Top: without physical augmentations, the model hallucinates lens glare as facial texture. Middle: without albedo consistency losses, ambient light leaks into the relit output. Bottom: without domain-aware normalization, global contrast and textural realism degrade view at source ↗

**Figure 6.** Figure 6: Ablation of distillation components. From left to right: input, teacher output, baseline distillation (without augmentation or multi-teacher fusion), distillation with augmentation, and the full pipeline. Asymmetric augmentation improves disentanglement of colored illumination; multi-teacher fusion preserves subject identity. targeted augmentations effectively decouple the relighting signal from imagingd… view at source ↗

**Figure 7.** Figure 7: Interactive Interface. The demonstration is recorded from our interactive interface running on a standard laptop with a mobile graphics card (NVIDIA RTX PRO 3000 GPU). A visualization of this interface is provided in view at source ↗

**Figure 8.** Figure 8: First row: input, target environment lighting. Second row: DiffusionRenderer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗

**Figure 9.** Figure 9: First row: input, target environment lighting. Second row: DiffusionRenderer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours. C Additional Details on the Synthetic Dataset Our dataset is rendered utilizing the NVIDIA Omniverse Replicator extension. To ensure the model learns a highly generalized representation, the dataset comprises an … view at source ↗

**Figure 10.** Figure 10: First row: input, target environment lighting. Second row: DiffusionRenderer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗

**Figure 11.** Figure 11: First row: input, target environment lighting. Second row: DiffusionRenderer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗

**Figure 12.** Figure 12: First row: input, target environment lighting. Second row: DiffusionRenderer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗

**Figure 13.** Figure 13: Examples of our synthetic dataset using Digital Human avatars. (a) and (b) are paired images under two different HDRIs, along with the corresponding (c) diffuse albedo map, and (d) camera-space normal maps view at source ↗

read the original abstract

The real-world adoption of portrait relighting is hindered by dataset domain gaps, camera sensitivity, and computational costs. We address these challenges with Hybrid Domain Knowledge Fusion, a paradigm that fuses the specialized strengths of synthetic, One-Light-at-A-Time (OLAT), and real-world datasets into a compact model. Our approach features specialized prior models hardened by domain-aware adaptation, followed by augmented knowledge distillation into a lightweight student model with multi-domain expertise. Our method demonstrates a 6x to 240x inference speedup while maintaining state-of-the-art (SOTA) visual quality in the experiments. Additionally, we construct a massive, high-fidelity synthetic dataset with diverse ground-truth intrinsics to support our training pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The hybrid fusion pipeline for portrait relighting combines domain-adapted priors with distillation to target real-world speed and data gaps, but the performance claims rest on unshown experiments.

read the letter

The main thing to know is that this paper introduces Hybrid Domain Knowledge Fusion as a way to blend synthetic, OLAT, and real portrait data into one compact relighting model. They adapt specialized prior models to each domain first, then distill the combined knowledge into a lightweight student network using an augmented process. They also release a large synthetic dataset with diverse ground-truth intrinsics to support training. This directly tackles the dataset shifts, camera issues, and compute demands that have limited portrait relighting outside labs, aiming for models that run on everyday hardware for mobile or AR use. The approach builds logically on standard adaptation and distillation steps but packages them specifically for this problem, which is a reasonable practical step. The new synthetic data contribution is also helpful for the field. The soft spots center on verification. The abstract states 6x to 240x speedups and SOTA quality, yet supplies no metrics, baselines, ablations, or failure analysis, so it is impossible to judge whether the transfer step preserves detail on real inputs or creates artifacts. The central assumption that domain-aware adaptation followed by distillation succeeds without meaningful quality loss is the one that needs checking most. There is no sign of circular reasoning or self-referential results; the method depends on external data and ordinary training procedures. This paper would interest applied computer vision groups working on deployable relighting or image synthesis tools. It deserves peer review because the targeted problem matters for adoption and the pipeline is clearly motivated, even if the results section will likely require strengthening with concrete numbers and comparisons.

Referee Report

1 major / 1 minor

Summary. The paper introduces the Hybrid Domain Knowledge Fusion paradigm for portrait relighting, which fuses specialized strengths from synthetic, OLAT, and real-world datasets. It does so via domain-aware adaptation of prior models followed by augmented knowledge distillation into a lightweight student model that acquires multi-domain expertise. The authors also construct a large-scale synthetic dataset with diverse ground-truth intrinsics. The central claim is that the resulting model achieves 6x to 240x inference speedup while preserving state-of-the-art visual quality.

Significance. If the performance claims are substantiated, the work could meaningfully advance practical deployment of portrait relighting by mitigating domain gaps and computational overhead, with potential benefits for mobile photography, AR/VR, and content creation pipelines. The new synthetic dataset with intrinsics would be a reusable resource for the community.

major comments (1)

Abstract: the assertion of 'state-of-the-art (SOTA) visual quality' and '6x to 240x inference speedup' is the load-bearing claim, yet the abstract supplies no quantitative metrics (e.g., PSNR/SSIM/LPIPS values), baseline comparisons, ablation tables, or error analysis. Without these, it is impossible to determine whether the results support the claim or reflect post-hoc evaluation choices.

minor comments (1)

The phrase 'Hybrid Domain Knowledge Fusion paradigm' is introduced without a concise formal definition or overview diagram; a schematic in §3 or §4 would clarify the flow from prior-model adaptation to student distillation.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The single major comment highlights an opportunity to strengthen the abstract, and we address it directly below with a commitment to revision.

read point-by-point responses

Referee: Abstract: the assertion of 'state-of-the-art (SOTA) visual quality' and '6x to 240x inference speedup' is the load-bearing claim, yet the abstract supplies no quantitative metrics (e.g., PSNR/SSIM/LPIPS values), baseline comparisons, ablation tables, or error analysis. Without these, it is impossible to determine whether the results support the claim or reflect post-hoc evaluation choices.

Authors: We agree that the abstract would be more informative with explicit quantitative anchors. The full manuscript reports standard metrics (PSNR, SSIM, LPIPS) and inference timings against multiple baselines in the Experiments section, with ablations and cross-dataset evaluations that follow established protocols in the portrait relighting literature. To address the concern, we will revise the abstract to include concise key results (e.g., average PSNR/LPIPS gains and the observed speedup range relative to prior methods) while preserving brevity. This change will make the central claims immediately verifiable from the abstract itself. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents a standard machine-learning pipeline: domain-aware adaptation of specialized prior models trained on synthetic/OLAT/real-world data, followed by augmented knowledge distillation into a compact student model. Claims of 6x–240x speedup and SOTA quality are positioned as empirical outcomes measured on held-out data, not as algebraic identities or fitted parameters renamed as predictions. No equations, self-referential definitions, or load-bearing self-citations appear in the provided text; the construction of an auxiliary synthetic dataset is an input step, not a circular output. The derivation chain therefore remains self-contained and externally falsifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests primarily on domain assumptions about successful cross-domain adaptation and distillation rather than new mathematical derivations, fitted parameters, or invented physical entities.

axioms (2)

domain assumption Specialized prior models trained on individual data domains can be hardened by domain-aware adaptation to contribute useful knowledge across domains.
Invoked as the first step of the fusion process.
domain assumption Augmented knowledge distillation can transfer combined multi-domain expertise into a single lightweight student model while preserving visual quality.
Central mechanism for achieving the reported inference speedup.

invented entities (1)

Hybrid Domain Knowledge Fusion paradigm no independent evidence
purpose: Framework for fusing strengths of synthetic, OLAT, and real-world datasets into a compact model
Newly introduced method name and pipeline; no independent evidence outside the paper is provided.

pith-pipeline@v0.9.0 · 5429 in / 1538 out tokens · 60598 ms · 2026-05-08T08:39:38.805482+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

38 extracted references · 6 canonical work pages · 1 internal anchor

[1]

In: CVPR

Afifi, M., Brubaker, M.A., Brown, M.S.: HistoGAN: Controlling colors of GAN- generated and real images via color histograms. In: CVPR. pp. 7941–7950 (2021)

2021
[2]

In: IEEE ICSIP

Alam, M.Z., Giuliani, N., Chen, H., Mantiuk, R.K.: Reduction of glare in images with saturated pixels. In: IEEE ICSIP. pp. 498–502 (2021)

2021
[3]

In: ICCVW (2025)

Anonymous: DNF-Avatar: Distilling neural fields for real-time animatable avatar relighting. In: ICCVW (2025)

2025
[4]

In: NeurIPS (2023)

Bashkirova, D., Ray, A., Mallick, R., Bargal, S.A., Zhang, J., Krishna, R., Saenko, K.: Lasagna: Layered score distillation for disentangled image editing. In: NeurIPS (2023)

2023
[5]

Information11(2) (2020)

Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: Fast and flexible image augmentations. Information11(2) (2020)

2020
[6]

In: CVPR

Cai, Z., Jiang, K., Chen, S.Y., Lai, Y.K., Fu, H., Shi, B., Gao, L.: Real-time 3D- aware portrait video relighting. In: CVPR. pp. 6221–6231 (2024)

2024
[7]

In: CVPR (2025)

Chaturvedi, S., Ren, M., Hold-Geoffroy, Y., Liu, J., Dorsey, J., Shu, Z.: SynthLight: Portrait relighting with diffusion model by learning to re-render synthetic faces. In: CVPR (2025)

2025
[8]

In: ACM SIGGRAPH

Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Ac- quiring the reflectance field of a human face. In: ACM SIGGRAPH. pp. 145–156 (2000)

2000
[9]

In: Smart Tools and Apps for Graphics (STAG) (2024)

Dulecha, T.G., et al.: Optimized NeuralRTI relighting through knowledge distilla- tion. In: Smart Tools and Apps for Graphics (STAG) (2024)

2024
[10]

arXiv preprint arXiv:2501.16330 (2025)

Fang, Y., Sun, Z., Zhang, S., Wu, T., Xu, Y., Zhang, P., Wang, J., Wetzstein, G., Lin, D.: RelightVid: Temporal-consistent diffusion model for video relighting. arXiv preprint arXiv:2501.16330 (2025)

work page arXiv 2025
[11]

Communications of the ACM63(11), 139–144 (2020)

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM63(11), 139–144 (2020)

2020
[12]

Unirelight: Learning joint decomposition and synthesis for video relighting.arXiv preprint arXiv:2506.15673, 2025

He, K., Liang, R., Munkberg, J., Hasselgren, J., Vijaykumar, N., Keller, A., Fidler, S., Gilitschenski, I., Gojcic, Z., Wang, Z.: UniRelight: Learning joint decomposition and synthesis for video relighting. arXiv preprint arXiv:2506.15673 (2025)

work page arXiv 2025
[13]

In: NIPS Deep Learning and Representation Learning Workshop (2015)

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop (2015)

2015
[14]

arXiv preprint arXiv:2510.23494 (2025)

Jüttner, E., Pfeifer, J., Krath, L., Korfhage, S., Dröge, H., Hullin, M.B., Stam- minger, M., Thies, J.: Yesnt: Are diffusion relighting models ready for cap- ture stage compositing? A hybrid alternative to bridge the gap. arXiv preprint arXiv:2510.23494 (2025)

work page arXiv 2025
[15]

In: CVPR (2024)

Kim, K., et al.: SwitchLight: Co-design of physics-driven architecture and pre- training framework for human portrait relighting. In: CVPR (2024)

2024
[16]

Adam: A Method for Stochastic Optimization

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

work page internal anchor Pith review arXiv 2014
[17]

Huang et al

Liang, R., Gojcic, Z., Ling, H., Munkberg, J., Hasselgren, J., Lin, Z.H., Gao, J., Keller, A., Vijaykumar, N., Fidler, S., Wang, Z.: Diffusionrenderer: Neural inverse 16 Q. Huang et al. and forward rendering with video diffusion models. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2025)

2025
[18]

arXiv preprint arXiv:2601.22135 (2026)

Liang, Z., Chen, Z., Chen, Y., Wei, T., Wang, T., Pan, X.: PI-Light: Physics- inspired diffusion for full-image relighting. arXiv preprint arXiv:2601.22135 (2026)

work page arXiv 2026
[19]

In: CVPR

Lin, H.: DreamSalon: A staged diffusion framework for preserving identity-context in editable face generation. In: CVPR. pp. 8589–8598 (2024)

2024
[20]

In: CVPR

Lin, M.H., Reddy, M., Berger, G., Sarkis, M., Porikli, F., Bi, N.: EdgeRelight360: Text-conditioned 360-degree HDR image generation for real-time on-device video portrait relighting. In: CVPR. pp. 831–840 (2024)

2024
[21]

In: CVPR

Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. In: CVPR. pp. 11976–11986 (2022)

2022
[22]

In: CVPR

Mei, Y., He, M., Ma, L., Philip, J., Xian, W., George, D.M., Yu, X., Dedic, G., Taşel, A.L., Yu, N., et al.: Lux post facto: Learning portrait performance relighting with conditional video diffusion and a hybrid dataset. In: CVPR. pp. 5510–5522 (2025)

2025
[23]

In: SIGGRAPH Advanced Computer Graphics Animation Course Notes (1984)

Miller, G.S., Hoffman, C.R.: Illumination and reflection maps: Simulated objects in simulated and real environments. In: SIGGRAPH Advanced Computer Graphics Animation Course Notes (1984)

1984
[24]

ACM TOG40(4) (2021)

Pandey, R., Orts-Escolano, S., Legendre, C., Haene, C., Bouaziz, S., Rhemann, C., Debevec, P., Fanello, S.: Total relighting: Learning to relight portraits for back- ground replacement. ACM TOG40(4) (2021)

2021
[25]

In: ACM SIGGRAPH (2024)

Rao, Y., et al.: Lite2Relight: 3D-aware single image portrait relighting. In: ACM SIGGRAPH (2024)

2024
[26]

arXiv preprint arXiv:1701.08893 (2017)

Risser, E., Wilmot, P., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893 (2017)

work page arXiv 2017
[27]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241 (2015)

2015
[28]

In: ISMAR

Song, G., Cham, T.J., Cai, J., Zheng, J.: Real-time shadow-aware portrait re- lighting in virtual backgrounds for realistic telepresence. In: ISMAR. pp. 729–738 (2022)

2022
[29]

ACM TOG26(3), 37 (2007)

Talvala, E.V., Adams, A., Horowitz, M., Levoy, M.: Veiling glare in high dynamic range imaging. ACM TOG26(3), 37 (2007)

2007
[30]

In: CVPR

Wang, T.C., Mallya, A., Liu, M.Y.: One-shot free-view neural talking-head syn- thesis for video conferencing. In: CVPR. pp. 10039–10049 (2021)

2021
[31]

IEEE TIP (2004)

Wang, Z., et al.: Image quality assessment: from error visibility to structural sim- ilarity. IEEE TIP (2004)

2004
[32]

Optical engineering19(1), 139–144 (1980)

Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Optical engineering19(1), 139–144 (1980)

1980
[33]

In: ICCV

Wu, Y., He, Q., Xue, T., Garg, R., Chen, J., Veeraraghavan, A., Barron, J.T.: How to train neural networks for flare removal. In: ICCV. pp. 2239–2247 (2021)

2021
[34]

In: CVPR (2022)

Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., Guo, B.: Advancing high-resolution video-language representation with large-scale video transcriptions. In: CVPR (2022)

2022
[35]

ACM TOG41(6), 1–15 (2022)

Yeh, Y.Y., Nagano, K., Khamis, S., Kautz, J., Liu, M.Y., Wang, T.C.: Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM TOG41(6), 1–15 (2022)

2022
[36]

In: ICCV

Zhang, L., Zhang, Q., Wu, M., Yu, J., Xu, L.: Neural video portrait relighting in real-time via consistency modeling. In: ICCV. pp. 802–812 (2021)

2021
[37]

In: ICLR

Zhang, L., Rao, A., Agrawala, M.: Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport. In: ICLR. pp. 10728–10745 (2025) Toward Real-World Adoption of Portrait Relighting 17

2025
[38]

In: CVPR (2018) 18 Q

Zhang, R., et al.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018) 18 Q. Huang et al. A Video Demonstration Wehighlyencouragereaderstoviewthesupplementaryvideodemonstration.mp4, provided as an ancillary file with this arXiv submission. This video provides comprehensive visual evidence of the relighting fidelity, tem...

2018