{"id":"20e6dca9-c71b-4485-ab54-376f579cb90a","arxiv_id":"2412.13174","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ORFormer uses per-patch messenger tokens to detect and recover occluded face regions, reducing landmark error on WFLW and COFW.","lead":"This paper introduces ORFormer, a transformer module that detects occluded parts of a face and reconstructs their missing features using extra messenger tokens. When its output heatmaps are plugged into standard facial landmark detectors, it improves accuracy on occluded-face benchmarks like WFLW and COFW.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline WFLW NME 3.86 in Table 1 is obtained with L2+NME loss while the reproduced STAR baseline uses STAR loss; without a no-ORFormer L2+NME baseline the SOTA gain is not attributable to ORFormer.","rationale":"The central claim is an empirical one: adding ORFormer heatmaps to an existing FLD model yields state-of-the-art accuracy. The strongest evidence for that claim should compare ORFormer-integrated models to the same base model trained with the same loss and capacity. Table 10 includes three ORFormer rows but omits the key control: the base HGNet without ORFormer under L2 heatmap + NME landmark loss. Since the headline 3.86 is produced under that loss, the reader cannot tell whether the gain versus the dagger-STAR baseline comes from ORFormer or from the loss switch. The same-loss row that does exist, STAR loss (4.03 to 3.92), shows a real but modest improvement, so the method is not obviously invalid; nevertheless, the SOTA claim in Table 1 is conditional on the missing control. I chose this over the alpha-map-validation concern because no amount of alpha-map accuracy would resolve the attribution problem: even a perfectly correct occlusion map would not show that the 3.86 result is due to ORFormer rather than L2+NME loss. Conversely, if the missing L2+NME baseline is already at 3.86, the central quantitative claim fails regardless of how well the mechanism works. The COFW and 300W claims rely on the same integration protocol, so the WFLW control is the decisive test. This matches the first issue in the reader's rationale, though the reader's formal weakest_assumption field points to the alpha map, hence partial agreement. The concern is presented as a missing control, not as any suggestion of misreporting; the authors disclose the loss in Table 10, which is why the test is straightforwardly constructible from the paper's own setup.","tokens_in":17866,"tokens_out":6043,"duration_ms":61951,"concrete_test":"Retrain the exact four-stack Hourglass baseline (no ORFormer) on WFLW using the same L2 heatmap loss + NME landmark loss, same from-scratch initialization, augmentation, and schedule as the 3.86 ORFormer row in Table 10, and report full-set and occlusion-subset NME. If the baseline reaches 3.86 or below, the Table 1 SOTA gain is not attributable to ORFormer; if the baseline remains near 4.03 or worse, the ORFormer integration accounts for the gain. The same check should be repeated for COFW if the claim extends there.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is not the internal occlusion mechanism but the comparability of the headline empirical result. Table 1 reports ORFormer WFLW NME 3.86, FR 1.76, AUC 0.622, and Table 10 shows that this 3.86 row is trained with L2 heatmap loss plus NME landmark loss. The reproduced STAR baseline, marked with a dagger at 4.03, uses STAR's AWing+STAR losses. The paper never trains the same four-stack Hourglass backbone without ORFormer under the L2+NME loss. Therefore the 3.86 number, and the state-of-the-art claim built on it, may reflect the loss change rather than the messenger-token/alpha mechanism. This is a missing control, not a backbone mismatch: Table 12 controls parameter count against a five-stack HGNet, but parameter count is not the confound here; loss function is. The controlled comparison that does exist, Table 6 with STAR loss (4.03 vs 3.92), provides partial support for ORFormer, so the concern does not by itself reject the method, but it does prevent clean attribution of the Table 1 SOTA numbers. The alpha-map validation issue is real but secondary: even a perfect alpha map would not resolve this attribution problem, because the headline gain could still be driven by the loss choice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ORFormer, a ViT-based module for facial landmark detection under partial occlusion. Each image patch token is paired with a learnable messenger token that attends to all patches except its own; an occlusion detection head derives a per-patch occlusion map alpha from the squared difference between the regular and messenger embeddings, and a feature-recovery step merges the two quantized feature streams with alpha. The recovered features are decoded into edge heatmaps, which are concatenated into existing FLD backbones (ADNet/STAR). Experiments on WFLW, COFW, and 300W report state-of-the-art or competitive NME/FR/AUC, with ablation studies showing consistent gains when ORFormer is added to STAR under the same loss, and efficiency gains over a parameter-matched five-stack Hourglass.","tokens_in":17994,"tokens_out":4503,"duration_ms":43595,"significance":"If the empirical attribution is sound, the paper makes a useful contribution: the messenger-token design is a generic mechanism for detecting and recovering corrupted or occluded patches, and the controlled comparison in Table 6 (STAR 4.03 -> 3.92 on WFLW full, 4.82 -> 4.66 on the occlusion subset, with matching loss) is direct evidence that the mechanism helps. The paper is also transparent about implementation details, reports reproduced baselines, and includes ablations for the distance function, integration strategy, and loss choices. The main significance concern is whether the headline Table 1 numbers are attributable to ORFormer or to the change of loss function, and whether the cross-attention 'exclusion' is implemented as described.","major_comments":[{"comment":"The claimed exclusion of the corresponding patch is not implemented by the equations as written. Setting Across(Q_M,K_X)_{i,i}=0 before softmax assigns a finite score of 0; after softmax, the own-patch token still receives a positive attention weight unless all other scores are also 0. Similarly, Eq. (8) multiplies the cross-attention score by (1-alpha_j) rather than applying a true mask, so the messenger token does not strictly aggregate 'features from all but its patch.' Please either replace the zero assignment with -inf masking (or the equivalent) or clarify that the intended operation is a score discount rather than exclusion, and confirm that the reported experiments use the described operation.","section":"§3.2, Eqs. (5)–(6) and Eq. (8)"},{"comment":"The headline WFLW NME of 3.86 in Table 1 is produced by the L2+NME loss configuration listed in Table 10, whereas the reproduced STAR baseline (marked with a dagger, 4.03) uses STAR's AWing+STAR losses. The paper never trains the same four-stack Hourglass backbone without ORFormer under the L2+NME loss, so the 3.86 number, and the state-of-the-art claim built on it, cannot be attributed to ORFormer rather than to the loss change. Please add the missing no-ORFormer L2+NME control, or report the SOTA comparison only with matched loss functions, as in Table 6, where the ORFormer gain is 4.03 to 3.92.","section":"§4.3 and §6.2.2, Tables 1 and 10"},{"comment":"The occlusion map alpha is load-bearing for feature recovery in Eq. (9), yet it is never quantitatively validated against ground-truth occlusion information. COFW provides per-landmark occlusion labels and WFLW provides subset labels, but the paper only visualizes alpha (Fig. 6). Because alpha is an unconstrained learned signal, please add a direct evaluation of alpha against occlusion annotations (e.g., patch-level precision/recall or AUC, or NME grouped by alpha threshold) to support the claim that ORFormer 'detects non-visible regions'; at minimum, report an ablation where alpha is replaced by a constant or by oracle-like values.","section":"§3.2, Eq. (7), and §4.4"}],"minor_comments":[{"comment":"In Eq. (10), S_k is a code index, not a one-hot vector; the expression '-S_k log(hat S_k)' should be written as '-log(hat S_{k,S_k})' or with explicit one-hot/cross-entropy notation.","section":"§3.2, Eq. (10)"},{"comment":"The subscript notation in Eq. (1) is inconsistent: 'Z i,j' and 'Z_{i,j}' appear side by side; please use a single typesetting for subscripts throughout.","section":"§3.1, Eq. (1)"},{"comment":"The heading 'Ackknowledgements' contains a typo and should read 'Acknowledgements'.","section":"Acknowledgements"},{"comment":"Tables 4 and 11 are identical duplicate ablation tables; please keep only one copy and cross-reference the other.","section":"§4.4 and §6.2.2, Tables 4 and 11"},{"comment":"The caption contains 'the he landmark NME loss'; it should read 'the landmark NME loss'.","section":"Table 4 caption"},{"comment":"The term 'mult-add' is unclear; please use 'MACs' or 'multiply-accumulate operations' for precision.","section":"§6.2.3, Table 12"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the Table 6 controlled comparison is the strongest evidence for the method. The missing L2+NME baseline is a straightforward fix, and the attention-masking issue in Eq. (6) needs clarification or correction because it affects the central mechanism as described. If the authors can add the missing control and fix the masking inconsistency, I would be willing to reconsider; the current manuscript's headline claim is not cleanly supported as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The messenger-token idea is the real thing here. Attaching a learnable token to each patch that attends to all patches except its own, then using the divergence between the regular and messenger embeddings to produce a per-patch occlusion map, is a genuine architectural departure from CodeFormer and HandOccNet. The paper also earns credit for the controlled ablations: with the STAR loss held fixed, ORFormer improves WFLW NME from 4.03 to 3.92 and the occlusion subset from 4.82 to 4.66 (Table 6). Those are modest but consistent gains, and the complexity comparison against a 5-stack Hourglass (Table 12) is fair. The supplementary material is unusually complete for this area, with index mappings, hyperparameters, and complexity analysis. The soft spots are real but not fatal. The main one is the headline claim. Table 1 reports WFLW NME 3.86, which beats the reproduced STAR baseline's 4.03, but Table 10 shows that 3.86 comes from L2 heatmap loss plus NME landmark loss, not STAR's AWing+STAR losses. The paper never trains the same backbone without ORFormer under L2+NME. So the state-of-the-art improvement in Table 1 cannot be attributed to the messenger-token mechanism; the loss change alone could explain part or all of it. The missing control is one training run. This is a confound, not a fatal flaw, because the same-loss Table 6 comparison already provides positive evidence for ORFormer. Second, the paper claims ORFormer detects non-visible regions but never quantifies the alpha map against ground-truth occlusion labels, even though COFW provides them. Figure 6 is qualitative only. That is a secondary issue, since a perfect alpha map would not fix the loss-function attribution problem, but it would strengthen the detection story. Minor discrepancy: Section 3.1 says the quantized heatmap generator is trained on faces without occlusions, while Section 4.1 describes adding random occlusion for the generator and ORFormer. The text should clarify whether the generator sees occluded images during pre-training. This is a solid incremental-to-substantial empirical paper with an honest evaluation scaffold. It deserves a serious referee. I would recommend major revision: add the no-ORFormer L2+NME baseline, validate alpha on COFW occlusion labels, and clarify the training-data discrepancy. With those changes, the contribution would be cleanly established.","headline":"The messenger-token mechanism is a real architectural contribution, but the headline SOTA number is confounded by a loss-function change and needs a missing control before the central claim is clean.","tokens_in":715,"tokens_out":885,"would_cite":false,"duration_ms":22871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer that detects and repairs occluded facial patches reaches state-of-the-art landmark accuracy.","keywords":["facial landmark detection","occlusion robustness","transformer","messenger token","heatmap regression","feature recovery","vector quantization"],"falsifier":"Feed ORFormer a set of faces with known synthetic occlusion masks and compare its predicted alpha map to the masks: if visibly covered patches do not receive high alpha, or if forcing alpha to zero (no recovery) does not raise NME on the WFLW occlusion subset, the claimed occlusion-detection and recovery mechanism is not what drives the accuracy gain.","tokens_in":17463,"feed_emoji":"🎯","tokens_out":6533,"duration_ms":55401,"temperature":0.7,"pith_summary":"ORFormer claims that facial landmark detection under occlusion can be made robust by having the transformer explicitly identify which image patches are non-visible and then rebuild their features from the rest of the face. The method adds one learnable messenger token per image patch; each messenger attends to every patch except its own, so the gap between a patch's own embedding and its messenger embedding reveals whether that patch is corrupted. A predicted occlusion map then blends the patch embedding with the messenger embedding, producing recovered features that yield high-quality facial-edge heatmaps. These heatmaps, fed into existing landmark detectors, push normalized mean error to 3.86 on WFLW and 4.46 on COFW, outperforming prior state-of-the-art results. The appeal is a general, annotation-free way to make transformer-based detectors resilient to partial occlusion.","feed_headline":"Messenger tokens reconstruct hidden face regions, top WFLW and COFW","feed_subtitle":"Recovered heatmaps push facial landmark accuracy past state-of-the-art results on WFLW and COFW benchmarks.","key_machinery":"The messenger token is the load-bearing object: for each image patch $X_i$ a learnable token $M_i$ runs cross-attention over all patch tokens except $X_i$ (attention score set to 0 at the diagonal). The difference between the resulting messenger embedding and the patch's own embedding, passed through a linear layer and sigmoid, gives the per-patch occlusion probability $\\alpha$. That $\\alpha$ is used twice: multiplicatively in the next layer's cross-attention to stop messengers from borrowing features from already-occluded patches, and as the interpolation weight in the final feature recovery $Z_{\\mathrm{rec}} = (1-\\alpha)\\otimes Z_I + \\alpha\\otimes Z_M$, where $Z_I$ and $Z_M$ are codebook-quantized features from regular and messenger embeddings. The codebook and decoder are pre-trained on unoccluded faces via vector quantization and frozen, so the recovery step draws on a prior of what intact faces look like.","core_discovery":"The central claim is that a pair of embeddings per patch — the regular self-attended token and a messenger token that deliberately excludes the patch itself — carries enough information to locate occluded regions and to repair them. ORFormer quantifies the discrepancy between the two embeddings with a squared-difference head, converts it into a patch-wise occlusion probability $\\alpha$ in $[0,1]$, and uses $\\alpha$ to form a convex combination $Z_{\\mathrm{rec}} = (1-\\alpha)\\otimes Z_I + \\alpha\\otimes Z_M$ of the regular and messenger quantized features before decoding heatmaps. Because the messenger gathers evidence only from visible patches, the recovered representation is robust to corruption; because $\\alpha$ is predicted from the input, no ground-truth occlusion labels are required. On WFLW, COFW, and the 300W challenging subset, the method reports the best normalized mean error among the compared methods, and on the WFLW occlusion subset it gains more than the full-set average over the VQVAE and CodeFormer baselines.","pith_inferences":["Because alpha is never supervised with occlusion masks, a natural extension is to add a weak or self-supervised loss on alpha using COFW's existing per-landmark occlusion labels; the paper's own numbers suggest this could further close the gap on the occlusion subset.","The same messenger-token design could apply to other partially occluded recognition tasks, such as object keypoint detection or pose estimation, wherever a quantized prior over intact appearances can be pre-trained.","The method's robustness to occlusion types absent from training (sunglasses, masks, hands) is untested; a synthetic-occlusion stress test with varying rectangle sizes could show whether the learned alpha generalizes beyond the augmentation used.","The paper's stated limitation that deformed facial features still hurt suggests a possible boundary: the messenger prior is a prior over intact geometry, so appearance deformation may be harder to model than simple occlusion."],"forward_implications":["Any heatmap-based facial landmark detector can adopt ORFormer's recovered heatmaps as auxiliary inputs; in the paper both ADNet and STAR improve under their own loss functions, with the largest gains on occlusion-heavy subsets.","The messenger mechanism needs no occlusion annotations at training or test time, so it transfers to datasets (like WFLW) that do not label occlusions.","Compared with a plain ViT, ORFormer adds about 10% more parameters, and the separate attention streams keep the extra compute linear in the token count.","Against a matched-parameter hourglass baseline, the full pipeline uses 20.6% fewer mult-adds and 15.9% less inference time while improving NME.","The strongest configuration comes from training the whole landmark model from scratch with simple L2 heatmap loss plus NME landmark loss, rather than fine-tuning only the merging block."],"supporting_citations":[{"why":"Supplies the vector-quantization codebook formulation used to pre-train the heatmap generator and serves as the baseline.","marker":"[34]"},{"why":"Provides the quantized-codebook plus transformer paradigm that ORFormer extends and the main heatmap-generation baseline.","marker":"[46]"},{"why":"Provides the patch-token self-attention backbone to which the messenger tokens are added.","marker":"[5]"},{"why":"Defines the WFLW benchmark and the edge-heatmap definition used for supervision.","marker":"[40]"},{"why":"The reproduced baseline and integration target whose STAR loss and hourglass backbone ORFormer is tested with.","marker":"[47]"},{"why":"The other integration target, contributing the ADNet loss and backbone used in the ablation studies.","marker":"[11]"},{"why":"Provides the COFW dataset and the consensus-among-patches idea for occlusion robustness that the messenger token builds on.","marker":"[3]"},{"why":"Supplies the stacked hourglass architecture used as the landmark detection backbone.","marker":"[26]"}],"fun_headline_variants":["Messenger tokens spot occluded face parts, boost landmark accuracy","ORFormer finds hidden facial landmarks under occlusion","Recover occluded face features with messenger tokens for accurate landmarks","Self-excluding tokens identify and repair occluded face patches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gap between a patch's own embedding and its messenger embedding reliably marks occluded patches, yet that occlusion map is never supervised with ground-truth labels, so a wrong alpha would make the recovery step hurt rather than help.","fun_headline_variants_meta":{"raw":{"variants":["Messenger tokens spot occluded face parts, boost landmark accuracy","ORFormer finds hidden facial landmarks under occlusion","Recover occluded face features with messenger tokens for accurate landmarks","Self-excluding tokens identify and repair occluded face patches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2557,"prompt_tokens":960,"completion_tokens":1597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1531}},"tokens_in":576,"tokens_out":1597,"duration_ms":11509,"temperature":1.0,"reasoning_tokens":1531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:21:09.824322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed ORFormer a set of faces with known synthetic occlusion masks and compare its predicted alpha map to the masks: if visibly covered patches do not receive high alpha, or if forcing alpha to zero (no recovery) does not raise NME on the WFLW occlusion subset, the claimed occlusion-detection and recovery mechanism is not what drives the accuracy gain.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the vector-quantization codebook formulation used to pre-train the heatmap generator and serves as the baseline."},{"cited_title":"Towards robust blind face restora- tion with codebook lookup transformer","cited_arxiv_id":null,"evidence_quote":"Provides the quantized-codebook plus transformer paradigm that ORFormer extends and the main heatmap-generation baseline."},{"cited_title":"Look at boundary: A boundary-aware face alignment algorithm","cited_arxiv_id":null,"evidence_quote":"Defines the WFLW benchmark and the edge-heatmap definition used for supervision."},{"cited_title":"Star loss: Reducing semantic ambigu- ity in facial landmark detection","cited_arxiv_id":null,"evidence_quote":"The reproduced baseline and integration target whose STAR loss and hourglass backbone ORFormer is tested with."},{"cited_title":"Adnet: Leveraging error-bias towards nor- mal direction in face alignment","cited_arxiv_id":null,"evidence_quote":"The other integration target, contributing the ADNet loss and backbone used in the ablation studies."},{"cited_title":"Robust face landmark estimation under occlusion","cited_arxiv_id":null,"evidence_quote":"Provides the COFW dataset and the consensus-among-patches idea for occlusion robustness that the messenger token builds on."},{"cited_title":"Stacked hour- glass networks for human pose estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the stacked hourglass architecture used as the landmark detection backbone."}],"review_version":1}