{"id":"2229c81a-6a5c-4fb8-9569-d5d84f1499e0","arxiv_id":"2507.16095","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A feedback fine-tuning method adds detector-based pose, interaction, identity, and gaze losses, time-gated by noise level, to SSR-Encoder, reporting modest gains on HICO-DET, GazeFollow, and Concept101.","lead":"This paper fine-tunes a personalized image generation model with extra losses from pose, face, gaze, and interaction detectors, applying each loss only at denoising timesteps where that signal is reliable. The authors report gains on three datasets, but the evaluation metrics reuse the same detectors that provide the training feedback.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Tables 1 and 2 show the full model does not improve gaze accuracy (53.73 vs 53.81) or HICO facial-identity similarity (0.6398 vs 0.6531), so the §4.4 claim of improved gaze and identity is contradicted; the only positive attribute-specific gain (interaction mAP) is measured with…","rationale":"I agree with the reader's rejection but would shift the emphasis slightly. The deepest problem is not only that detector outputs might be biased; it is that the paper's own quantitative results already fail to support the headline. The full model is worse than SSR-Encoder on gaze accuracy on GazeFollow and on facial identity similarity on HICO-DET, so the §4.4 statement that the feedback-guided model improves the generated images in each aspect is directly false under the paper's chosen metrics. The one consistently positive attribute-specific number, interaction mAP, comes from the same detector (CMMP) that defines the training loss and uses a deliberately lax evaluation rule (label presence without bounding-box IoU). This makes that improvement an expected consequence of fitting CMMP's preferences rather than evidence of better interaction rendering. The timestep-gating idea and some CLIP/DINO gains are real contributions, but they do not rescue the central social-context claim. The appropriate disposition is unchanged: the paper should be rejected as written; the claims could be revised and supported with independent evaluation.","tokens_in":11394,"tokens_out":6231,"duration_ms":66684,"concrete_test":"Run a human perceptual evaluation on a stratified sample (e.g., 200 HICO-DET and 200 GazeFollow test images) comparing Ours versus SSR-Encoder side-by-side in randomized order on three axes: interaction correctness, facial identity match, and gaze plausibility. If human judgments do not show a significant improvement for Ours, or if they show declines on identity and gaze consistent with Tables 1 and 2, the circular detector-based metrics cannot support the central claim. Independently, recompute the SSR-Encoder and Ours rows of Tables 1 and 2 from released checkpoints; if they reproduce, the contradiction between Tables 1/2 and §4.4 stands even before any new experiment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, repeated in the Abstract and §4.4, is that feedback-based finetuning improves generated interactions, facial identities, and gaze. The strongest evidence against it is internal to the paper. In Table 1, the full model (Ours) has lower facial cosine similarity than SSR-Encoder: 0.6398 vs 0.6531. In Table 2, it has lower gaze accuracy: 53.73% vs 53.81%. Thus, under the paper's own evaluation, two of the three attribute-specific claims fail on the relevant benchmark; only interaction mAP improves. That interaction gain is not trustworthy because Eq. 6 trains L_interaction against CMMP logits, and Table 1 evaluates interaction generation with CMMP mAP under a no-IoU rule that counts a label as correct regardless of where it is predicted. The same training/evaluation-detector overlap applies to gaze (Sharingan) and identity (ArcFace), although those metrics already fail to show consistent gains. The paper provides no independent measurement of whether generated interactions, identities, and gaze are actually better, either by human raters or by detectors not used in defining the losses.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning the SSR-Encoder personalized image generation model with additional feedback losses derived from frozen pretrained detectors: a boundary loss computed from segmentation gradients (Eq. 2), an identity loss using ArcFace embeddings (Eq. 3), a gaze loss using Sharingan targets and vectors (Eq. 4), a pose loss using X-Pose keypoints (Eq. 5), and an interaction loss using CMMP logits (Eq. 6). These losses are applied only within manually selected timestep ranges, under the rationale that coarse cues (pose, interaction) are recovered earlier in denoising than fine cues (identity, gaze). The method is evaluated on HICO-DET, GazeFollow, and Concept101, with ablations of each loss. The central claim, stated in the Abstract and Section 4.4, is that the feedback-guided model improves generated interactions, facial identities, and gaze patterns.","tokens_in":11649,"tokens_out":3075,"duration_ms":36306,"significance":"If the claims were well supported, the timestep-aware use of multiple social-cue detectors as feedback would be a useful contribution to personalized generation, and the ablation structure is reasonably thorough. The paper also contains a potentially useful observation, illustrated in Figure 1, that different semantic attributes become reliable at different denoising timesteps. However, the quantitative evaluation is not trustworthy for the attribute-specific claims because the metrics reuse the same detectors that define the training losses, and the full model's reported identity and gaze scores are internally inconsistent with the paper's headline claims. The interaction mAP metric is especially weak because it ignores localization entirely. These issues bear directly on the main contributions, so the current manuscript does not establish its claims.","major_comments":[{"comment":"The evaluation of interaction, identity, and gaze is circular with respect to the training objectives. Equations 3, 4, and 6 train the diffusion model to match outputs of ArcFace, Sharingan, and CMMP respectively, and Section 4.2 evaluates the same attributes with the same detectors: CMMP mAP for interactions, ArcFace cosine similarity for identity, and Sharingan-based gaze-target matching. Improving these metrics therefore partly reflects better alignment with the frozen detectors' outputs rather than independently verified improvements in true image attributes. The paper provides no human evaluation and no comparison against detectors that were not used in the loss functions. This is load-bearing because every attribute-specific quantitative gain is reported through these detectors.","section":"Section 3 (Eqs. 3-6) and Section 4.2"},{"comment":"The central claim that the method improves facial identity and gaze is contradicted by the paper's own tables. In Table 1, the full model 'Ours' has facial cosine similarity 0.6398 versus 0.6531 for SSR-Encoder, and in Table 2 it has gaze accuracy 53.73% versus 53.81% for SSR-Encoder. Thus, under the paper's own evaluation, two of the three attribute-specific claims fail on the relevant benchmarks; only interaction mAP improves for the full model. The Abstract and Section 4.4 nevertheless state that the method improves identity and gaze, which is internally inconsistent.","section":"Abstract, Section 4.4, Tables 1 and 2"},{"comment":"The interaction mAP metric is not a valid measure of generated interactions because Section 4.2 explicitly counts a CMMP label prediction as positive irrespective of the predicted bounding box location, i.e., with no IoU constraint. Since Eq. 6 trains the model against CMMP logits, the reported interaction mAP gain of 16.43 versus 15.87 in Table 1 may reflect improved alignment with CMMP's label distribution rather than correct spatial interaction placement. A localization-aware metric, or an independent interaction detector, is needed to support the interaction claim.","section":"Section 4.2 and Eq. 6"},{"comment":"The timestep thresholds for applying each feedback loss are 'determined through manual inspection,' and all loss weights are fixed at 0.01. The inverse-timestep ablation in Table 1 shows that a different weighting strategy degrades performance, but this does not establish that the chosen thresholds are robust or that the results are not tuned to the specific benchmarks. A sensitivity analysis over thresholds and weights would strengthen the paper's central methodological claim, though this issue is secondary to the circular evaluation and internal contradictions above.","section":"Section 3F and Table 1"}],"minor_comments":[{"comment":"The phrase 'timestep-based inculcation' is unusual; 'incorporation' or 'application' would be clearer.","section":"Abstract"},{"comment":"The description of Figure 3 (right) says the boundary loss curve is 'increasing' after Gaussian filtering, but the visual trend is not easy to verify at the printed figure resolution; a clearer plot or explicit numbers would help.","section":"Section 3F and Figure 3"},{"comment":"The text says 'the row 3 in the table indicates the performance of SSR-model when it is finetuned on the hicodet dataset' but the table caption labels this row 'SSR-Encoder*'; the terminology should be consistent.","section":"Section 4.3"},{"comment":"MIP-Adapter is based on SDXL and is described as a 'topline'; because it uses a different base architecture and training data, direct comparison with the SDV1.5-based methods is not apples-to-apples and should be presented as an upper reference only.","section":"Tables 1 and 2"},{"comment":"In Table 2, the full model's gaze accuracy is slightly lower than SSR-Encoder, but the text emphasizes the gain in facial embedding similarity; this selective emphasis obscures the fact that the headline claim of improved gaze is not supported.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: the timestep-gated detector-feedback idea is worth knowing about, but the paper's headline claims do not survive its own tables. The full model does not improve gaze accuracy on GazeFollow (53.73 vs 53.81) and it lowers facial identity similarity on HICO-DET (0.6398 vs 0.6531) relative to SSR-Encoder. The one attribute that does improve — interaction mAP — is trained and evaluated with the same detector under a no-IoU rule, so that gain cannot be read as evidence of better interactions. The abstract and Section 4.4 overstate what the data show.\n\nWhat is genuinely new: fusing four pretrained detectors (pose, identity, gaze, interaction) as feedback losses with timestep-based gating, plus the observation that different social signals become reliable at different noise levels. That is a sensible extension of the SSR-Encoder program. The boundary loss is a nice addition. The ablation is honest: each individual loss delivers its intended attribute (L_gaze alone raises gaze accuracy to 56.39), and the inverse-timestep baseline doing worse supports the gating choice. The paper is clearly written and the figures help.\n\nThe soft spots are proportional to the claims. The evaluation circularity is load-bearing: ArcFace, Sharingan, and CMMP define the training losses and the metrics. For interaction, mAP ignores localization entirely, so pasting a person and object anywhere in the frame can score well. For gaze, the accuracy is a permissive bbox-label match. The non-circular metrics (CLIP-T, CLIP-I, DINO) show only marginal, inconsistent gains — on GazeFollow, DINO actually drops. The timestep thresholds are chosen by manual inspection of loss curves; acceptable, but closer to tuning than derivation.\n\nWho is this for? Someone working on detector-based guidance for personalized generation will find the gating idea worth borrowing. As a paper, it needs a different evaluation: independent detectors, human ratings, or IoU-aware interaction metrics, plus a corrected abstract. I would send it to review because the method deserves vetting, but I would expect major revision or rejection as-is.\n\nRecommendation: engage with the work, but ask the authors to fix the circularity first.","headline":"A genuinely new timestep-gated feedback idea undercut by a circular evaluation and the paper's own tables; the full model does not improve gaze or identity.","tokens_in":12164,"tokens_out":3138,"would_cite":false,"duration_ms":30155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Detector feedback improves pose, gaze, and identity in personalized image generation.","keywords":["personalized image generation","diffusion models","social context feedback","human-object interaction","gaze estimation","face identity preservation","timestep-dependent supervision","detector-guided fine-tuning"],"falsifier":"Train the model exactly as described, then evaluate identity, gaze, and interaction with detectors not used in training and with human raters; if the improvements over SSR-Encoder shrink to noise, the reported gains are detector-specific rather than real image improvements.","tokens_in":11197,"feed_emoji":"🖼️","tokens_out":6568,"duration_ms":64439,"temperature":0.7,"pith_summary":"Personalized image generation gets subjects' appearances right but often fails when the prompt describes social actions: poses are wrong, faces no longer match the reference person, and gaze does not follow the described interaction. This paper claims those failures can be reduced by fine-tuning a diffusion-based personalization model with feedback from pretrained detectors for pose, human-object interaction, identity, and gaze. The detector outputs on the reconstructed clean image are compared with detector outputs on the real training image, and the resulting losses are added to the usual denoising objective only during timesteps where each signal is reliable. If the claim holds, adding these social-context feedback losses to an existing personalization framework improves interaction correctness, identity preservation, gaze consistency, and overall image quality on HICO-DET, GazeFollow, and Concept101.","feed_headline":"Detector feedback fixes pose, gaze, identity in personalized images","feed_subtitle":"Fine-tuning with frozen pose, gaze, face, and interaction detectors stabilizes social-scene generation.","key_machinery":"The central mechanism is timestep-gated detector feedback. During training, at each noise level the model predicts the noise, converts the latent back to an approximate clean image $\\hat{x}_0$ through the VAE decoder, and runs frozen detectors on that image; the losses push $\\hat{x}_0$'s detector outputs toward the same detectors' outputs on the ground-truth image. Because diffusion recovers coarse structure at high noise and fine details at low noise, the paper gates each loss to timesteps where the signal is trustworthy: gaze for $t \\in [0,200]$, identity for $t \\in [0,400]$, interaction for $t \\in [0,500]$, and pose for $t \\in [0,700]$. The total loss is the denoising loss plus these detector losses, each weighted at 0.01, together with the SSR-Encoder regularization loss.","core_discovery":"The paper's central claim is that feedback from frozen pretrained detectors can act as a training signal that fixes three known weaknesses of personalized generation: incorrect human poses and interactions, lost facial identity, and unnatural gaze. The authors fine-tune SSR-Encoder with additional losses for boundary-gradient consistency, ArcFace identity matching, Sharingan gaze target and vector matching, X-Pose keypoint matching, and CMMP interaction logit matching, and they apply each loss only in the timestep range where the corresponding signal is still recoverable from the noisy latent: gaze for $t \\in [0,200]$, identity for $t \\in [0,400]$, interaction for $t \\in [0,500]$, and pose for $t \\in [0,700]$. On the HICO-DET test set the full method raises interaction mAP from 15.87 to 16.43, on GazeFollow the gaze-only feedback raises gaze accuracy from 53.81% to 56.39%, and on Concept101 the method improves CLIP-I and DINO scores over the non-feedback baselines including an SDXL-based model. The paper concludes that its feedback-guided diffusion model, leveraging pretrained gaze, identity, and interaction detectors, improves the generated images in each of these aspects.","pith_inferences":["A natural extension, not tested in the paper, is to apply the same detector-feedback recipe to other relational cues such as body contact, relative depth, or object state changes, where the same coarse-to-fine timestep logic should apply.","Because the evaluation uses the very detectors that supervised training, part of the reported gains may be specific to those detectors; measuring identity with a different face recognition model or gaze with human annotators would show whether the improvement transfers.","The timestep cutoffs were selected by inspecting loss curves; an adaptive scheduler that learns when each detector's signal is reliable could replace the manual ranges.","The boundary-gradient loss's sensitivity to fine detail suggests the same feedback idea could be used for texture or edge quality in non-social personalized generation, though the paper does not explore that."],"forward_implications":["Complex activities in prompts, such as 'pushing a motorcycle,' will be generated with the correct pose and interaction rather than showing the subject riding or ignoring the object.","The method is a fine-tuning wrapper, so it can be applied to other personalization backbones beyond SSR-Encoder without changing their inference-time pipeline.","Timestep-gated supervision stabilizes training, so combining coarse and fine social signals does not require reweighting or adversarial scheduling.","Generated images keep reference identities in social scenes, while gaze follows the person or object implied by the caption.","Overall image quality, measured by CLIP-I and DINO, improves or holds relative to the baselines while the social cues improve."],"supporting_citations":[{"why":"Supplies the SSR-Encoder base model being fine-tuned, including the image-text aligner and cross-attention conditioning that carry reference-subject information.","marker":"[32]"},{"why":"Supplies ArcFace facial identity embeddings used for the identity loss and for identity-similarity evaluation.","marker":"[4]"},{"why":"Supplies Sharingan gaze targets and vectors used for the gaze loss and for gaze-accuracy evaluation.","marker":"[24]"},{"why":"Supplies X-Pose skeleton keypoints used for the pose loss.","marker":"[30]"},{"why":"Supplies CMMP human-object-interaction logits and labels used for the interaction loss and for mAP evaluation.","marker":"[13]"},{"why":"Supplies the HICO-DET interaction training and test data for the interaction experiments.","marker":"[3]"},{"why":"Supplies the GazeFollow gaze training and test data for the gaze experiments.","marker":"[21]"},{"why":"Supplies the Concept101 multi-concept personalization benchmark used for the image-quality evaluation.","marker":"[12]"}],"fun_headline_variants":["Detector feedback sharpens pose, gaze, identity in generated images","Frozen detectors train diffusion model to fix social-scene flaws","Feedback fine-tuning stabilizes pose and gaze in personalized images","Detector signals correct pose, identity, gaze in image generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the frozen detectors' outputs on real and generated images are accurate enough that bringing the generated image's detector signals close to the real image's signals truly improves pose, identity, gaze, and interaction, rather than just making those detectors happier.","fun_headline_variants_meta":{"raw":{"variants":["Detector feedback sharpens pose, gaze, identity in generated images","Frozen detectors train diffusion model to fix social-scene flaws","Feedback fine-tuning stabilizes pose and gaze in personalized images","Detector signals correct pose, identity, gaze in image generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1453,"prompt_tokens":975,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":591,"tokens_out":478,"duration_ms":5075,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:17:59.824221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the model exactly as described, then evaluate identity, gaze, and interaction with detectors not used in training and with human raters; if the improvements over SSR-Encoder shrink to noise, the reported gains are detector-specific rather than real image improvements.","supporting_citations":[{"cited_title":"Where are they looking? In Advances in Neural Information Processing Systems (NIPS), 2015","cited_arxiv_id":null,"evidence_quote":"Supplies the GazeFollow gaze training and test data for the gaze experiments."},{"cited_title":"Arcface: Additive angular margin loss for deep face recognition","cited_arxiv_id":null,"evidence_quote":"Supplies ArcFace facial identity embeddings used for the identity loss and for identity-similarity evaluation."},{"cited_title":"Sharingan: A transformer archi- tecture for multi-person gaze following","cited_arxiv_id":null,"evidence_quote":"Supplies Sharingan gaze targets and vectors used for the gaze loss and for gaze-accuracy evaluation."},{"cited_title":"X-pose: Detecting any keypoints","cited_arxiv_id":null,"evidence_quote":"Supplies X-Pose skeleton keypoints used for the pose loss."},{"cited_title":"Exploring conditional multi-modal prompts for zero-shot hoi detection","cited_arxiv_id":null,"evidence_quote":"Supplies CMMP human-object-interaction logits and labels used for the interaction loss and for mAP evaluation."},{"cited_title":"Learning to detect human-object interactions","cited_arxiv_id":null,"evidence_quote":"Supplies the HICO-DET interaction training and test data for the interaction experiments."},{"cited_title":"Multi-concept customization of text-to-image diffusion","cited_arxiv_id":null,"evidence_quote":"Supplies the Concept101 multi-concept personalization benchmark used for the image-quality evaluation."}],"review_version":1}