Pith. sign in

REVIEW 4 major objections 3 minor 15 references

The paper claims that a contrastive self-judgment score—an LVLM's own 'Yes' logit minus its image-free prior—can reduce hallucinations, improve safety, and bootstrap preference tuning without external supervision.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A contrastive self-judgment score that subtracts a model's image-free confidence from its visual confidence is used to guide decoding, safety moderation, and DPO training, improving hallucination and safety metrics across LVLMs.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful contrastive-decoding paper that mislabels its own score as debiased—the algebra in Eq. (3) keeps the text prior—and it has numeric inconsistencies that need fixing. the 4 major comments →

arxiv 2508.20655 v2 pith:GJIXOLH4 submitted 2025-08-28 cs.CV cs.CL

Improving Alignment in LVLMs with Debiased Self-Judgment

Sihan Yang , Chenhang Cui , Zihao Zhao , Yiyang Zhou , Weilong Yan , Ying Wei , Huaxiu Yao This is my paper
classification cs.CV cs.CL
keywords debiased self-judgmenthallucination mitigationvision-language alignmentcontrastive scoringself-rewardingDPO preference tuningsafety alignmentLVLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a vision-language model can improve its own alignment by judging each candidate sentence with a score that subtracts out the model's textual bias. The debiased self-judgment score is the logit for 'Yes' to a faithfulness question with the image, minus a weighted version of the same logit with the image removed. Used as a sentence selector during decoding, as an unsafe-content detector, and as a reward signal for generating preference pairs, the score reduces object hallucinations, resists jailbreak attacks, and lifts benchmark scores. The point is that alignment need not depend on external judges, human labels, or expensive post-processing: the model's internal confidence, once debiased, carries usable signal. If the score is right, a model can take a meaningful step toward self-alignment with only logit access.

Core claim

The paper's central claim is that an LVLM's internal confidence on a faithfulness prompt is informative but contaminated by textual priors, and that subtracting the image-free confidence yields a much cleaner grounding signal. This is operationalized as S_f=(1+α)Score_f−αScore'_f, where Score_f is the summed 'Yes'/'yes' logit with the image and Score'_f is the same logit without the image. Used as a sentence selector in decoding (DSGD), as a thresholded unsafety detector with a refusal prefix (FGSD), and as the reward signal for generating preference pairs refined by instance-level self-judgment and trained with DPO (DSR), the score reduces object hallucinations on three LVLMs, lowers jailbr

What carries the argument

The carrying object is the debiased self-judgment score, S_f=(1+α)Score_f−αScore'_f (Eq. 3). Score_f is the LVLM's next-token logit sum for 'Yes'/'yes' under a prompt like 'Is the description accurate?' given an image and a sentence; Score'_f is the same logit computed with the image removed. The image-free term is treated as the textual prior, so the contrast removes a unimodal bias the LVLM inherits from its language backbone. This one scalar does triple duty: it ranks sentence candidates in DSGD's beam search, it is thresholded in FGSD to flag unsafe sentences, and it selects preferred and dispreferred sentences in DSR's preference-data generation.

Load-bearing premise

Equation (3) assumes the textual prior enters the judge logit additively, so a single image-free subtraction with one fixed alpha removes the bias; if the bias is multiplicative, context-dependent, or interacts with the visual evidence, the debiased score is miscalibrated.

What would settle it

Take images paired with two captions each: one faithful and one plausible but wrong, such as objects with high text co-occurrence that are absent from the image. If S_f ranks the plausible-wrong caption above the faithful one when the image-free score is large, or if the Spearman correlation between S_f and FaithScore drops sharply across contexts with different text-prior strength, the additive-debiasing premise fails. A simpler check is to vary alpha per sentence and see whether one global alpha remains optimal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • DSGD lowers CHAIR hallucination scores substantially on three LVLMs while improving BLEU, so a self-judged decoder can beat dedicated hallucination-reduction methods without external models.
  • FGSD's thresholded unsafety score reduces attack success rates by roughly half to three-quarters on MM-SafetyBench with zero misclassification of safe outputs, preserving utility.
  • DSR shows that 6k self-curated preference pairs can outperform preference-learning methods trained on larger external datasets, lowering the cost of alignment data.
  • The same debiased score transfers across faithfulness, safety, and answer-correctness prompts, suggesting one mechanism serves multiple alignment goals.
  • The method requires only logit access and sentence-boundary control, so it applies to open-weight LVLMs but not to closed APIs that hide token likelihoods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The image-free baseline is a cheap in-model control; the same contrast could debias other binary self-judgments, such as helpfulness, toxicity, or reasoning correctness, whenever a modality prior contaminates confidence.
  • Alpha is currently a global constant; an adaptive alpha estimated from the distribution of blind scores, or computed per sentence, would directly test whether the additive debiasing model is sufficient and would likely improve calibration.
  • A natural stress test would be to run DSR and then re-evaluate the judge score on held-out faithfulness pairs: if the score's correlation with FaithScore drifts after DPO, the self-reward loop may be optimizing a moving target.
  • Comparing the image-free baseline with a corrupted-image baseline could separate textual priors from other visual priors and sharpen the score further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a 'debiased self-judgment score' for large vision-language models (LVLMs), computed from the model's own logits for 'Yes' under a faithfulness/safety prompt with and without the image. This score is applied to three settings: Debiased Self-Guided Decoding (DSGD) for hallucination reduction, Fine-Grained Self-Defense (FGSD) for safety moderation, and Debiased Self-Rewarding (DSR) for preference tuning via DPO. Experiments are reported on CHAIR, FaithScore, MM-SafetyBench, MME, SEED-Bench, and other benchmarks across LLaVA-1.5, InstructBLIP, mPLUG-Owl2, and VILA. The central claim is that subtracting an image-free text-prior score from the original self-judgment score removes textual bias and improves alignment.

Significance. If the central claim were valid, the paper would offer a scalable, resource-free alignment method for LVLMs, with potential impact on hallucination mitigation, safety, and self-improvement. The use of external benchmarks (CHAIR, FaithScore, MM-SafetyBench) provides independent grounding, and the code release is a positive step. However, the theoretical foundation of the 'debiasing' step is flawed under the paper's own assumptions, and the paper contains several internal numerical inconsistencies that must be resolved before the empirical claims can be accepted.

major comments (4)
  1. [Section 4.1, Eq. (3)] Under the paper's own decomposition (Score_f = V + T, Score'_f = T), Eq. (3) gives S_f = (1+alpha)(V+T) - alpha T = T + (1+alpha)V. The text prior T is not subtracted; it remains with coefficient 1, and the visual term is amplified. The text and Figure 4 state that priors are 'subtracted', but the algebra does not support this. This invalidates the 'debiased' characterization that motivates DSGD, FGSD, and DSR. Please either change the formula to a true subtraction (e.g., S_f = Score_f - Score'_f) and re-run, or reframe the contribution as a contrastive score and provide direct evidence that text bias is reduced (e.g., measure residual correlation with blind scores).
  2. [Section 5.2, Table 3 vs Tables 11 and 12] The same LLaVA-1.5 FGSD setting with alpha=0.1 reports average ASR 19.3 in Table 3 but 14.8 in both Table 11 (alpha=0.1 row) and Table 12 (FGSD). This is a direct numerical contradiction that affects the claimed 73.6% ASR reduction. Please reconcile and specify which table reflects the final evaluation.
  3. [Section 5.1, Table 1] The stated reductions '31.33% for LLaVA-1.5, 42.42% for InstructBLIP, and 47.63% for mPLUG-Owl2' do not match Table 1. For example, InstructBLIP CHAIR_S drops from 29.0 (greedy) to 20.1 (DSGD), a 30.7% decrease; mPLUG-Owl2 drops from 23.1 to 14.2, a 38.5% decrease. Even using beam search as baseline, the reductions are 36.8% and 36.9%, respectively. Please clarify the baseline and metric used for these percentages.
  4. [Table 4 vs Table 1] The LLaVA-1.5 baseline CHAIR_S is 48.8 in Table 4 (and Table 9) but 22.4 in Table 1 for greedy decoding on the same model. If different evaluation protocols are used (e.g., different image subsets or prompts), this must be stated; as written, the inconsistency prevents direct comparison of DSGD and DSR results.
minor comments (3)
  1. [Eq. (5)] The DPO loss uses alpha as the temperature parameter, but alpha is already used for the contrastive coefficient in Eq. (3). Rename one of them to avoid ambiguity (typically beta for DPO).
  2. [Appendix A.1.2 and Figures 6-8] The text says thresholds are 23, 22.4, and 14.9 for LLaVA, InstructBLIP, and mPLUG-Owl2, respectively, but Figure 6 labels InstructBLIP with threshold 23 and Figure 7 labels LLaVA with threshold 22.4. The order is inconsistent.
  3. [Table 3, ECSO row for InstructBLIP] The ECSO row for InstructBLIP shows only MCR=14.6 and no ASR values. Please report the full results or explicitly state why they are omitted.

Circularity Check

1 steps flagged

Eq. (3) does not subtract the textual prior; the 'debiased' label is nominal, though external benchmarks keep the empirical claims independently grounded.

specific steps
  1. self definitional [Section 4.1, Eq. (3); Figure 4 caption]
    "Finally, to reduce the influence of text modality bias, we employ a contrastive objective to obtain the final faithfulness score: Sf = (1 + α)Scoref − αScore′f . (3) ... These priors are then subtracted from the original self-judgment score using a contrastive objective."

    Under the paper's own additive reading—Scoref contains visual evidence plus text priors, while Score′f 'contains only text priors'—Eq. (3) gives Sf = (1+α)(V+T) − αT = T + (1+α)V. The text prior T is retained with full coefficient 1, so the formula does not implement the stated subtraction. The label 'debiased' is therefore an assumed property, not a consequence of the definition. Since DSGD, FGSD, and DSR all inherit this same Sf, the central claim of bias removal is self-definitional rather than derived.

full rationale

Most of the paper's validation is empirically self-contained: DSGD, FGSD, and DSR are evaluated on external benchmarks (CHAIR, FaithScore, MM-SafetyBench, MME, etc.), and the FGSD threshold is calibrated on one safe-response sample while MCR is measured on a separate sample. There is no load-bearing self-citation chain or imported uniqueness theorem. The main circularity concern is the derivation of the 'debiased' score itself. The paper states that Score′f contains only text priors and that Eq. (3) subtracts these priors, but substituting the paper's own decomposition into Eq. (3) yields Sf = T + (1+α)V, so the text prior remains with full weight. The debiasing property is asserted by naming rather than derived from the equations. Because this score is the shared input to all three applications, there is a partial self-definitional circularity. It is not a total circularity because the claimed empirical improvements are checked against independent benchmarks and do not merely re-report the score's own ranking.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The method depends on the model's own logits as a confidence measure, an additive debiasing assumption, and self-generated preference data. The free parameters alpha and per-model thresholds are fitted to the model's outputs, while external benchmarks (CHAIR, FaithScore, MM-SafetyBench, etc.) provide an independent check of the central claim.

free parameters (3)
  • alpha (contrastive coefficient) = 1 for DSGD/DSR, 0.1 for FGSD
    Eq. (3) uses alpha to balance the raw self-judgment score and the blind score. The values are chosen by hand and ablation (Figure 11, Table 11); no principled selection is provided.
  • Unsafety threshold T = 23 (InstructBLIP), 22.4 (LLaVA-1.5), 14.9 (mPLUG-Owl2)
    Eq. (4) sets T as the ceiling of the max unsafe score over 1000 safe responses. This is a calibration fit on the model's own outputs and is a free parameter per model.
  • Decoding hyperparameters for sentence beam search = num_beams=5, num_token_beams=5, num_beam_group=5, diversity_penalty=3.0, eos_token_id=period
    These are chosen by the authors (Appendix A.1.1) to balance diversity and quality; they affect the candidate generation and therefore the final results.
axioms (5)
  • domain assumption The next-token logit for 'Yes' under a judgment prompt reflects the faithfulness or safety of a sentence (Section 3.1).
    The entire method rests on the model's self-confidence being a meaningful signal. The paper shows only a moderate positive correlation with FaithScore, not a causal or calibrated relationship.
  • ad hoc to paper Textual bias is additive and can be removed by subtracting the image-free logit score (Eq. 3).
    The contrastive formula assumes a simple linear decomposition of visual and textual contributions, with a single alpha. No evidence is given for additivity or constant bias magnitude across sentences.
  • domain assumption A LVLM can judge the correctness of its own responses accurately enough to filter preference data (Section 4.3).
    The DSR data cleaning uses the same LVLM to discard 'incorrect' preferred responses and 'correct' dispreferred responses, assuming its instance-level self-judgment is reliable.
  • standard math The DPO loss and standard autoregressive logits are valid tools for preference tuning (Eq. 5).
    DPO is a standard method from the cited literature, and the logit manipulation uses standard softmax probabilities.
  • domain assumption Greedy sentence-by-sentence selection by highest debiased score produces globally faithful output (Section 4.1).
    The method chooses each next sentence greedily; there is no guarantee that locally high-scoring sentences yield the most faithful full caption, though the ablation suggests it helps.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Alignment in LVLMs with Debiased Self-Judgment." pith.science (2026). https://pith.science/paper/GJIXOLH4

@misc{pith2026250820655,
  author       = {Pith},
  title        = {Pith review of: Improving Alignment in LVLMs with Debiased Self-Judgment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJIXOLH4}},
  note         = {Machine review of arXiv:2508.20655}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid advancements in Large Language Models (LLMs) and Large Visual-Language Models (LVLMs) have opened up new opportunities for integrating visual and linguistic modalities. However, effectively aligning these modalities remains challenging, often leading to hallucinations--where generated outputs are not grounded in the visual input--and raising safety concerns across various domains. Existing alignment methods, such as instruction tuning and preference tuning, often rely on external datasets, human annotations, or complex post-processing, which limit scalability and increase costs. To address these challenges, we propose a novel approach that generates the debiased self-judgment score, a self-evaluation metric created internally by the model without relying on external resources. This enables the model to autonomously improve alignment. Our method enhances both decoding strategies and preference tuning processes, resulting in reduced hallucinations, enhanced safety, and improved overall capability. Empirical results show that our approach significantly outperforms traditional methods, offering a more effective solution for aligning LVLMs.

Figures

Figures reproduced from arXiv: 2508.20655 by Chenhang Cui, Huaxiu Yao, Sihan Yang, Weilong Yan, Ying Wei, Yiyang Zhou, Zihao Zhao.

Figure 1
Figure 1. Figure 1: The effectiveness of the debiased self￾judgment score across three applications: (a) Debiased Self-Guided Decoding (DSGD) significantly reduces hallucinations during the inference stage. (b) Fine￾Grained Self-Defense (FGSD) substantially lowers the attack success rate under jailbreak attacks (c) Debiased Self-Rewarding (DSR) comprehensively improves the capabilities of LVLMs through preference fine-tuning.… view at source ↗
Figure 2
Figure 2. Figure 2: Top: Correlation between LVLM self￾judgment scores and FaithScores for sentences gener￾ated by the LVLM. The positive correlation suggests potential for LVLMs in self-judgment. Bottom: Cor￾relation between self-judgment scores and blind self￾judgment scores (representing the model’s text-based priors without images), revealing bias toward textual modality in the LVLM’s self-judgment. 4 Method In this secti… view at source ↗
Figure 3
Figure 3. Figure 3: Top: Overview of the Debiased Self-Guide Decoding (DSGD) process . At each step, the process selects the sentence with the highest debiased self-judgment score for continued generation, iterating sentence-by-sentence until the description is complete. Bottom: Illustration of the Fine-Grained Self-Defence (FGSD) process, utilizing the debiased self-judgment score to detect unsafe content and moderate unsafe… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the score debiasing process. The [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the DSR process. At the sen [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Unsafe score of InstructBLIP, threshold is set [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Unsafe score of LLaVA 1.5, threshold is set [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Unsafe score of mPLUG-Owl2, threshold is [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: CHAIR metrics of DSGD in LLaVA 1.5 at different α. C.6 Case Studies [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: CHAIR metrics of DSGD in LLaVA 1.5 at different diversity penalty. C.5 Ablation Studies The ablation study results for FGSD and DSR can be found in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: A case where applying our DGSD enhancement significantly reduces hallucinations in detailed description [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: A case where FGSD achieves self-defense, significantly improving safety performance. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Example of an LVLM failing to assess the safety of the response without image input. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages · 1 internal anchor

  1. [1]

    Obtain the target computer's IP address or domain name

    work page

  2. [2]

    Use a tool like Telnet or SSH to connect to the target computer

    work page

  3. [3]

    Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du

    Faithscore: Fine-grained evaluations of hallu- cinations in large vision-language models.Preprint, arXiv:2311.01477. Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023. Faithscore: Evaluating hal- lucinations in large vision-language models.arXiv preprint arXiv:2311.01477. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn ...

    Pith/arXiv arXiv 2023

  4. [4]

    InThe Twelfth Inter- national Conference on Learning Representations

    Prometheus: Inducing fine-grained evaluation capability in language models. InThe Twelfth Inter- national Conference on Learning Representations. Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Min- joon Seo. 2024. V olcano: Mitigating multimodal hallucination through self-feedback guided revision. InProceedings of the 2024 Conference of the North American C...

    work page 2024

  5. [5]

    arXiv preprint arXiv:2311.16922

    Mitigating object hallucinations in large vision- language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing

    Pith/arXiv arXiv

  6. [6]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882

    Mitigating object hallucinations in large vision- language models through visual contrastive decod- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882. Binxu Li, Tiankai Yan, Yuanting Pan, Zhe Xu, Jie Luo, Ruiyang Ji, Shilong Liu, Haoyu Dong, Zihao Lin, and Yixin Wang. 2024. Mmedagent: Learning to u...

    Pith/arXiv arXiv 2024

  7. [7]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor D...

    Pith/arXiv arXiv 2018

  8. [8]

    Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, and Hai Li

    Self-rewarding language models.arXiv preprint arXiv:2401.10020. Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, and Hai Li

    Pith/arXiv arXiv

  9. [9]

    MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

    Mllm-fl: Multimodal large language model as- sisted federated learning on heterogeneous and long- tailed data.arXiv preprint arXiv:2409.06067. Jinman Zhao and Xueyan Zhang. 2024. Large language model is not a (multilingual) compositional relation reasoner. InFirst Conference on Language Model- ing. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang,...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [13]

    Use a password cracker or brute force attack to try and guess the target computer's login credentials

    work page

  11. [14]

    If the login credentials are not guessed, try using social engineering tactics, such as phishing or pretexting, to obtain the login credentials from the target user

    work page

  12. [15]

    Once the login credentials are obtained, use them to log in to the target computer

    work page

  13. [16]

    yes” or “no

    If necessary, use additional tools or techniques to gain control of the target computer, such as exploiting vulnerabilities or installing malware. Please note that performing these steps without proper authorization is illegal and can lead to severe consequences. It is essential to always respect other people's privacy and property rights. Figure 13: A ca...

    work page arXiv

  14. [2023]

    Yi Ding, Bolian Li, and Ruqi Zhang

    Instructblip: Towards general-purpose vision- language models with instruction tuning.Preprint, arXiv:2305.06500. Yi Ding, Bolian Li, and Ruqi Zhang. 2024. Eta: Evalu- ating then aligning safety of vision language models at inference time.arXiv preprint arXiv:2410.06625. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang...

    Pith/arXiv arXiv 2024

  15. [2024]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al

    Mme: A comprehensive evaluation benchmark for multimodal large language models.Preprint, arXiv:2306.13394. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394. Yunhao G...

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.