{"id":"c3b9d681-f61f-4a72-876c-aad65a43f3d7","arxiv_id":"2412.07260","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DFREC recovers both the source and target faces from a deepfake image using identity segmentation and an identity-aware masked autoencoder.","lead":"This paper introduces DFREC, a deep-learning system that takes a face-swapped (deepfake) image and attempts to reconstruct both the original source person's face and the original target person's face. It is a forensic tool aimed at helping investigators trace who was swapped onto whom.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline IDSim metric shares its FaceNet feature extractor with the training identity loss, so Table I may measure optimization of the metric rather than identity recovery; an independent identity metric is needed.","rationale":"The reader's weakest_assumption focuses on whether the forged image retains enough target identity information, which is a genuine architectural premise. I do not choose that as the primary concern because the paper itself acknowledges the limitation in Section V, and the cross-forgery generalization results (Table II) plus Fig. 1 empirically support the presence of recoverable target-related information. The metric-contamination concern is more load-bearing because Table I is the central quantitative evidence for 'superior recovery performance' and 'high fidelity,' and the evaluation metric is provably entangled with DFREC's training loss. This does not prove fraud or even that the method fails; it means the headline comparisons are currently not neutral. The reader's verdict is CONDITIONAL, and this concern reinforces that condition rather than overturning the paper: re-evaluation with an independent identity metric is a concrete, feasible requirement. The missing source quantitative evaluation and absent error bars are additional weaknesses, but they are secondary to the metric issue for the 'superior recovery' claim.","tokens_in":20742,"tokens_out":6187,"duration_ms":71307,"concrete_test":"Recompute the target IDSim rows of Table I on the same test sets using an identity embedding that was not used in any DFREC loss, e.g., ArcFace (InsightFace) or a FaceNet checkpoint trained on a disjoint identity partition; also report FID and, if available, source IDSim on the same images. If DFREC's margin over DFI and Delocate shrinks by more than about 0.05 or reverses on the independent metric, the reported target-recovery superiority is substantially a metric-optimization artifact. If the margins persist, this concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the primary quantitative evidence for the central claim, target identity recovery, is obtained with an evaluation metric that DFREC's training objective directly optimizes. Section IV.A.3 defines IDSim as cosine similarity between FaceNet embeddings of the original and recovered target faces, using the pretrained FaceNet of [30]. Section III.D defines the target identity loss L_tgt_id (Eq. 3) on the same FaceNet features (Mixed6a, repeat 3, block8, last_bn), included in Lid and hence in the total loss (Eq. 10). DFREC is therefore trained to maximize the exact similarity surface that Table I uses to compare it with MAT, RePaint, RECCE, Delocate, and DFI, none of which train against FaceNet target identity. The reported margins (e.g., 0.5367 vs 0.4749 in FF++-DeepFake) may partly reflect that DFREC has been fit to the metric rather than a generic improvement in identity recovery. No independent face-identity metric (ArcFace, MegaFace, human perceptual identity matching) is reported, so the 'high fidelity' and 'superior recovery' claims rest on a single metric that is not neutral between DFREC and the baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DFREC, a deepfake face-recovery framework that aims to reconstruct both the source identity and the target identity from a single face-swapped image. The method consists of an Identity Segmentation Module (ISM) that separates source- and target-related facial information, a Source Identity Recovery Module (SIRM) that reconstructs the source face and extracts latent target-identity features, and a Target Identity Recovery Module (TIRM) that uses an identity-aware Masked Autoencoder to fuse background context with the target-identity features and reconstruct the target face. The loss function combines FaceNet-based identity losses, VGG-based perceptual losses, an attribute loss, and a patch-recovery loss. Experiments are reported on FaceForensics++, CelebaMegaFS, and FFHQ-E4S across six face-swapping methods, with comparisons to inpainting and deepfake-recovery baselines using FID, IDSim, and a derived identity-based recovery accuracy Accid. The central claim is that DFREC is the only scheme that can recover both pristine source and target faces directly from a forgery image with high fidelity.","tokens_in":20961,"tokens_out":4500,"duration_ms":50060,"significance":"If fully validated, DFREC would be a useful forensic tool because it offers interpretable identity traceability rather than just a detection score. The architecture is well motivated, and the authors evaluate on several datasets and forgery algorithms, including cross-dataset generalization and robustness to post-processing. The ablation study in Table VI and Table VII provides evidence for the contribution of the ISM, the identity fusion, and the individual loss terms. However, the current evidence does not fully support the headline claims: the primary identity metric shares its feature extractor with the training objective, source-recovery performance is never quantified, and the method's stated limitation on splicing/cropping is not reflected in the abstract's unqualified claim. These are correctable with additional experiments and re-analysis, which is why I view the paper as promising but not yet acceptable in its present form.","major_comments":[{"comment":"The headline identity metric is not neutral between DFREC and the baselines. IDSim is defined in Section IV.A.3 as the cosine similarity between FaceNet embeddings of the original and recovered target faces, using the pretrained FaceNet of [30]. The target identity loss in Eq. (3) minimizes L2 distances between the same FaceNet feature layers (Mixed6a, repeat 3, block8, last_bn), and this loss enters the total objective through Eqs. (4) and (10). None of the baselines (MAT, RePaint, RECCE, Delocate, DFI) are trained to optimize FaceNet target-identity similarity, so the margins in Table I (e.g., 0.5367 vs 0.4749 on FF++-DeepFake) partly reflect that DFREC has been fit to the evaluation metric itself. I request an independent identity metric, such as ArcFace or MegaFace embeddings, or a human perceptual identity-matching study, reported for both target and source recovery. FID is a distributional image-quality metric and does not measure identity, so it cannot substitute for this.","section":"IV.A.3 and III.D"},{"comment":"The central claim that DFREC 'is the only scheme that can recover both pristine source and target faces' is not quantitatively supported for the source face. Table I reports FID and IDSim only between the recovered target face and the original target face; Tables II, V, VI, and VII likewise measure target recovery. Source recovery appears only in qualitative figures (Figs. 3-5 and Fig. 7). Because the claim of dual recovery is the paper's main novelty, the source-recovery branch needs direct quantitative evaluation: source FID and an independent source IDSim against the original source face, along with a comparison of what each baseline can or cannot produce for the source. Without this, the abstract's 'both faces' claim is an overstatement relative to the evidence.","section":"Abstract and Table I"},{"comment":"The limitations section states that when the forged face is not derived from the target face (e.g., direct cropping and splicing), performance is 'significantly constrained.' This is a substantial scope restriction relative to the unqualified 'deepfake identity recovery' claim in the abstract and conclusion. The evaluation only covers face-swapping forgeries generated by trained swapping models. The scope should either be qualified in the abstract and conclusion, or the paper should include a splicing/cropping condition in the experiments to characterize the method's actual boundary. This is not a request to solve splicing, but the claim 'high fidelity' should be scoped to the face-swapping setting that is actually tested.","section":"Section V"}],"minor_comments":[{"comment":"There are typos in the abstract and introduction: 'fadelity' should be 'fidelity,' and 'DEREC' in the contribution bullet should be 'DFREC.'","section":"Abstract and Section I"},{"comment":"The header 'FaceForensic++' in Table I should be 'FaceForensics++' to match the dataset name used elsewhere in the paper.","section":"Table I"},{"comment":"The definition of Accid is terse: it uses identity similarity between the input face and the recovered target face, but for a forged input the recovered target is intentionally different from the input, whereas for a pristine input it is the same. Please specify which feature extractor is used for this similarity and state the threshold rationale more explicitly.","section":"IV.A.3"},{"comment":"In Algorithm 1, when the global mean of the segmentation map is at least 0.9, all patch weights are replaced by random values (step 5-7). This behavior is surprising and should be explained: does it correspond to a nearly all-source image, and why are all patch weights randomized in that case?","section":"Algorithm 1"},{"comment":"The red highlighting used for intra-dataset evaluations should be described in the captions and must be distinguishable in grayscale printing.","section":"Tables II and IV"},{"comment":"The ablation table for losses reports single point estimates without error bars or statistical significance. Given that the differences between some configurations are small (e.g., IDSim 0.7192 vs 0.7109), a variance estimate across multiple seeds or test splits would strengthen the claim that all loss terms contribute.","section":"Section IV.F.3 / Table VII"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the IDSim metric is the most serious issue, but it is fixable by re-running evaluation with an independent identity encoder and by adding source-recovery metrics. The paper's architecture and dataset coverage are real assets, and the ablation study is informative. I would be willing to look at a revised version that addresses the metric neutrality and the dual-recovery quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2412.07260. The core idea is real: instead of another deepfake detector, DFREC tries to recover both the source and target face from a face swap, and none of the cited prior work attempts both. The architecture is a sensible composition of known parts — DeepLabv3 segmentation to split source/target information, a U-Net to reconstruct the source and pull latent target features, and an MAE that fuses those features with unmasked background patches to reconstruct the target. That is a plausible division of labor, and the ablations support each module earning its keep.\n\nWhat is genuinely good: they evaluate on three datasets and six swapping algorithms, including cross-forgery generalization and robustness to JPEG, blur, and noise. The generalization tables show large margins over DFI, which suggests the method is not just memorizing the training forgeries. The identification accuracy Accid is above 98% in most cells, which would be a strong forensic result if the metric stands.\n\nHere is where I push back. The stress-test note lands: the headline IDSim metric uses FaceNet embeddings of the recovered and original target faces, and the target identity loss trains on exactly those FaceNet layers. So Table I compares DFREC against baselines that never saw a FaceNet identity loss, while DFREC was trained to minimize L2 distance in that space. The margins are partly evidence of better identity recovery, but they also reflect metric alignment. An independent identity metric — ArcFace, a different checkpoint, human perceptual matching — would settle this. The reader's concern about source recovery is also fair: source recovery appears only as qualitative figures, never as a number. The paper claims 'only scheme that can recover both with high fidelity,' but high fidelity for the source face is asserted, not demonstrated.\n\nOther soft spots, in proportion: no error bars anywhere, and no code or data release, which makes the 0.98–1.00 Accid numbers hard to verify. The limitations section is honest about the splicing case degrading, and that should stay. None of these are fatal to the idea. The task is valuable, the design is coherent, and the qualitative results look like real recovery, not just color correction.\n\nWho is this for? Forensics researchers who need identity tracing for investigations will want to know this exists. It deserves a serious peer review, but the referee should push hard for an independent identity metric and for a quantitative source-recovery evaluation. I would not cite the numbers yet, but I would cite the task formulation.","headline":"DFREC is a genuinely new task formulation with decent engineering, but the headline IDSim numbers partly measure what the model was trained to optimize, and source recovery is never quantified.","tokens_in":21537,"tokens_out":2116,"would_cite":false,"duration_ms":19303,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single deepfake image can be unmixed into both the swapped-in face and the original target face.","keywords":["deepfake identity recovery","face swapping forensics","identity segmentation","masked autoencoder","deepfake provenance","identity tracing","source and target face recovery","image forensics"],"falsifier":"Take face-swapped images and replace the forged face region with a patch cropped from an unrelated person's photo, so that no target-identity signal remains in the swapped area, then run DFREC: if the identity similarity between the 'recovered target' and the true target does not drop to near-chance levels, the reported recovery is likely driven by background context rather than by separated target identity features. A second check is to compare the unsupervised segmentation map against ground-truth swap masks; if their overlap is no better than random, the claimed identity separation is not what the network actually learned.","tokens_in":20500,"feed_emoji":"🎭","tokens_out":5526,"duration_ms":54226,"temperature":0.7,"pith_summary":"DFREC is a system for turning a single face-swapped image into two recovered faces: the source face (the person whose identity was transplanted in) and the target face (the person whose appearance was stolen). The paper argues that even after a face-swap, the forged image carries a separable trace of the target's identity, entangled with the target's pose, expression, and lighting. DFREC's three modules—an identity segmentation module, a source reconstruction module, and a target reconstruction module built on a masked autoencoder—learn to split the forged face into these two identity streams and rebuild each person at high fidelity. If correct, this gives forensic investigators a non-repudiable, intuitive link from a deepfake video to the actual faces involved, a step beyond today's detectors that only say \"fake.\" The method reports higher identity similarity between recovered and real target faces than prior inpainting and deepfake-recovery baselines on three public benchmarks, and is the only scheme in its comparison that recovers both faces directly.","feed_headline":"One network recovers both faces hidden in a deepfake","feed_subtitle":"DFREC separates the swapped-in identity from the victim's leftover features and rebuilds each face at high fidelity.","key_machinery":"The load-bearing mechanism is the identity-aware masked autoencoder in the Target Identity Recovery Module. A semantic-guided masking stage uses the source-affiliation map to decide which image patches to hide, preferring patches most likely to belong to the swapped-in source face, while an identity fusion decoder concatenates the latent target identity features extracted by the source reconstruction module with the visible patch embeddings, so the model is conditioned on who the target is rather than simply inpainting texture. The segmentation map is trained with no direct supervision; it is driven only by the downstream reconstruction losses for the source and target faces.","core_discovery":"The central claim is that a face-swapped image is not merely a new identity on an old body: it is a mixture of two identities, and the mixture can be unmixed. The paper supports this by measuring identity similarity, finding that a forgery is roughly as similar to its target face as to its source face, which indicates that target-identity information survives the swap. DFREC operationalizes the unmixing with an identity segmentation map that classifies each pixel's affiliation with the source face, a source reconstruction network that splits its own latent features into a source stream and a target stream, and a masked autoencoder that hides the source-related patches and fills them in using visible background patches plus the extracted target stream. Trained with identity, perceptual, attribute, and patch-recovery losses, the system reconstructs both faces from the single forged image, and reports identity similarity superior to existing baselines on FaceForensics++, CelebaMegaFS, and FFHQ-E4S across six face-swapping algorithms.","pith_inferences":["If the identity-splitting premise holds beyond face-swapping, the same three-module design could be applied to other attribute-manipulation forgeries—expression reenactment, aging, makeup transfer—where a target identity still underlies the edited image.","Because the segmentation is unsupervised, a testable implication is that DFREC's success across unseen forgeries depends on a consistent swap-region signature rather than on true identity understanding; a perfectly blended diffusion-based edit would stress whether the target trace is still separable.","A practical extension the paper does not explore: embedding the recovered source face and matching it against a known-identity database would turn recovery from a visualization tool into an automated suspect-ranking step for investigators.","The recovered target face could be used as a privacy safeguard, allowing platforms to reconstruct and blur the victim's identity in a deepfake before the image spreads further."],"forward_implications":["Forensic investigators could obtain both the suspect's face and the victim's face from a single deepfake frame, giving a concrete link between a forged video and real identities.","The same network can flag a face as real or fake by checking whether its recovered target reconstruction differs from the input, so recovery doubles as an interpretable detector.","Recovery generalizes across face-swapping algorithms in the paper's experiments, meaning a model trained on a few swap types can still reconstruct targets from unseen forgeries.","The recovered target face remains identifiable even after common post-processing such as blur, JPEG compression, and Gaussian noise, per the paper's robustness tests.","Pristine images pass through the system unchanged, so the method can be applied to a mixed batch of real and fake faces without corrupting the genuine ones."],"supporting_citations":[{"why":"Supplies the premise that face-swapping fuses source and target identities because attribute features correlate with identity, motivating the extraction of latent target features.","marker":"[8]"},{"why":"FaceShifter is both one of the six evaluated forgery algorithms and a source of the observation that high-fidelity swaps preserve target attributes such as expression, pose, and lighting.","marker":"[10]"},{"why":"MegaFS generates the CelebaMegaFS dataset and is one of the swap algorithms evaluated; its LCR variant is used in the experiments.","marker":"[12]"},{"why":"Face X-ray is used during training to continuously generate forgery faces and masks, which the authors credit with improving the segmentation performance of the identity segmentation module.","marker":"[17]"},{"why":"RECCE is a reconstruction-based deepfake detection baseline whose outputs are limited to distorted faces, motivating the goal of proper target-identity recovery.","marker":"[24]"},{"why":"Delocate is the masked-autoencoder deepfake recovery baseline that DFREC extends and directly compares against in the experiments.","marker":"[25]"},{"why":"DFI is the deepfake inversion baseline that traces the target face by decoupling identity and attributes; it is the strongest existing method that DFREC outperforms in generalization and recovery accuracy.","marker":"[27]"},{"why":"The masked autoencoder architecture, including the Vision Transformer encoder and patch masking procedure, is the technical foundation of the target identity recovery module.","marker":"[29]"},{"why":"FaceNet provides the identity feature extraction used both in the identity loss and in the IDSim evaluation metric.","marker":"[30]"},{"why":"FaceForensics++ is one of the three evaluation datasets and provides DeepFake and FaceShifter forged videos used in the experiments.","marker":"[35]"}],"fun_headline_variants":["Deepfake unmixed: one model reconstructs both faces","Split a deepfake into its two real faces","Unmasking deepfakes: recover both identities","One network, two recovered faces from a single deepfake","Two faces in one deepfake? This model pulls both out"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a face-swapped image retains enough of the target person's identity, mixed into the swapped area, that it can be separated out and used to rebuild the target face; if the forged face never came from the target face, for example through direct cropping and splicing, the recovery breaks down.","fun_headline_variants_meta":{"raw":{"variants":["Deepfake unmixed: one model reconstructs both faces","Split a deepfake into its two real faces","Unmasking deepfakes: recover both identities","One network, two recovered faces from a single deepfake","Two faces in one deepfake? This model pulls both out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4572,"prompt_tokens":1005,"completion_tokens":3567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":3488}},"tokens_in":621,"tokens_out":3567,"duration_ms":31059,"temperature":1.0,"reasoning_tokens":3488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:26.047711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take face-swapped images and replace the forged face region with a patch cropped from an unrelated person's photo, so that no target-identity signal remains in the swapped area, then run DFREC: if the identity similarity between the 'recovered target' and the true target does not drop to near-chance levels, the reported recovery is likely driven by background context rather than by separated target identity features. A second check is to compare the unsupervised segmentation map against ground-truth swap masks; if their overlap is no better than random, the claimed identity separation is not what the network actually learned.","supporting_citations":[{"cited_title":"Implicit identity driven deepfake face swapping detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that face-swapping fuses source and target identities because attribute features correlate with identity, motivating the extraction of latent target features."},{"cited_title":"Advancing high fidelity identity swapping for forgery detection,","cited_arxiv_id":null,"evidence_quote":"FaceShifter is both one of the six evaluated forgery algorithms and a source of the observation that high-fidelity swaps preserve target attributes such as expression, pose, and lighting."},{"cited_title":"One shot face swapping on megapixels,","cited_arxiv_id":null,"evidence_quote":"MegaFS generates the CelebaMegaFS dataset and is one of the swap algorithms evaluated; its LCR variant is used in the experiments."},{"cited_title":"Deepfake face prove- nance for proactive forensics,","cited_arxiv_id":null,"evidence_quote":"DFI is the deepfake inversion baseline that traces the target face by decoupling identity and attributes; it is the strongest existing method that DFREC outperforms in generalization and recovery accuracy."}],"review_version":1}