Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DAMA claims that making DPO's β depend on each example's measured hardness and on the model's current reward gap reduces hallucination and improves general-task scores in multimodal LLMs.

desk verdict A sensible adaptive-β extension of DPO with clean ablations, but the abstract misreports the headline numbers and the data-hardness term is length-sensitive; worth reviewing, not worth rejecting. read the letter →

arxiv 2502.01943 v2 pith:AABFIGWO submitted 2025-02-04 cs.CV

Jinda Lu , Junkang Wu , Jinghan Li , Xiaojun Jia , Shuo Wang , YiFan Zhang , Junfeng Fang , Xiang Wang
show 1 more author
Xiangnan He
This is my paper · ORCID
classification cs.CV
keywords multi-modallargelanguagemodelhallucinationmitigationdirectpreferenceoptimizationdatahardnessresponsivenessadaptivebetaCLIPimage-textsimilarityLLaVAalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard Direct Preference Optimization (DPO) for multimodal LLMs, the paper argues, does not treat its training examples equally: examples that are easy to distinguish between chosen and rejected responses get overfit, while hard-to-distinguish examples barely move, so hallucination persists exactly where it is hardest to remove. DAMA is a proposed fix that makes DPO's regularization strength β vary per example, scaled up on easy examples and down on hard ones, using two measured quantities: data hardness from CLIP image-text similarity at sub-sentence level, and model responsiveness from the current reward gap between chosen and rejected responses. On its evaluations, this one change raises LLaVA-1.5-7B response-level non-hallucination from 78.3% (vanilla DPO) to 90.9% on Object HalBench and improves AMBER, MMHal-Bench, LLaVA-Bench, and MM-Vet, with 7B and 13B Object HalBench results above GPT-4V. The point of caring is that hallucination limits trust in multimodal assistants, and DAMA claims to reduce it through an optimization adjustment rather than new data, decoders, or reward models.

What carries the argument

The carrying object is a per-instance regularization coefficient, $\beta_C = \beta \cdot \alpha_D \cdot \alpha_M$, that replaces the single global β of the DPO loss. $\alpha_D$ is data hardness: responses are split into sub-sentences, each is scored by a CLIP model against the image, the scores of both responses are softmax-normalized together, and the chosen-minus-rejected sum forms $\delta$, with $\alpha_D = \sigma(\delta)/\sigma(\bar{\delta})$ where $\bar{\delta}$ is the dataset mean. $\alpha_M$ is model responsiveness: per-batch implicit reward gaps from $\log \pi_\theta(y)/\pi_{\mathrm{ref}}(y)$ are normalized, outlier-filtered by squared distance, and compared to a moving-average gap, giving $\alpha_M = \sigma(\bar{R}_B)/\sigma(\bar{R})$. Element-wise multiplication of the two factors produces one β per example in the batch; large β enforces reference-model proximity, small β lets the policy update further, which is the mechanism claimed to balance easy and hard examples.

What would settle it

Run the data-aware component with δ divided by the number of sub-sentences and check whether Object HalBench gains persist; separately, correlate each instance's δ with human hallucination ratings — if the correlation is weak or negative, or if length normalization erases the gains, the hardness premise is falsified.

Watch

Extended reading notes

Core claim

DAMA's central claim is that DPO's imbalanced responsiveness to data hardness is a correctable optimization defect. The paper defines data hardness as the difference between the chosen and rejected response in softmax-normalized CLIP image-text similarity, computed on sub-sentences to avoid CLIP's length constraints, and defines model responsiveness as the outlier-filtered, mean-normalized implicit reward gap in the current batch. Both quantities are folded into β by element-wise multiplication, so easy examples keep a large β (stay close to the reference model, avoid overfitting) and hard examples get a small β (keep moving while the model is still wrong). The reported outcome is that DAMA-7B reaches 90.9% response-level and 95.3% mentioned-level non-hallucination on Object HalBench, versus 78.3% and 89.5% for vanilla DPO, and that DAMA-13B also improves on AMBER and MMHal-Bench while holding or improving general-task scores on LLaVA-Bench and MM-Vet.

Load-bearing premise

The load-bearing premise is that the sum of softmax-normalized CLIP image-text similarity scores over sub-sentences is a valid measure of data hardness; if length confounds or CLIP's proxy errors dominate, the hardness weights mis-rank easy and hard examples and the data-aware component's benefit would weaken.

Editorial extensions

If this is right

  • Per-example β can be added to existing DPO pipelines using signals already at hand — CLIP scores and the policy/reference log-ratios — with no new reward model, decoder, or data-cleaning stage.
  • Hard-to-distinguish examples receive smaller β and therefore larger effective updates, which predicts the largest hallucination drops on generative benchmarks where borderline rejections dominate.
  • Ablations show the multiplication combination beats weighted-sum blending and both single components, so the data- and model-aware corrections are jointly, not additively, useful.
  • The reported gains on LLaVA-Bench and MM-Vet indicate that the trustworthiness improvements are not accompanied by a collapse of general task performance.
  • The 7B model exceeds GPT-4V on Object HalBench, and the 13B model does so on both Object HalBench and AMBER.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because $\alpha_D$ is computed from softmax-normalized CLIP scores summed over sub-sentences, responses with more sub-sentences can accumulate more probability mass purely by length; a length-normalized hardness variant is the natural check on whether the gains are about hallucination content or response length.
  • Inference: The per-example β schedule behaves like a curriculum in which hard examples receive larger effective gradients, so DAMA could be combined with curriculum ordering or used as a drop-in weighting rule for other preference losses such as SimPO or KTO.
  • Inference: The paper limits itself to image-text MLLMs; if the mechanism transfers, the same hardness/responsiveness weighting should reduce hallucination in video and audio multimodal models, which the paper lists as future work.
  • Inference: The paper reports Object HalBench numbers evaluated with GPT-4-turbo-2024-04-09; re-running that benchmark with an open-source judge or human ratings would test whether the DPO-to-DAMA gap persists across evaluators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes DAMA, a modification of DPO for multi-modal LLMs in which the per-instance DPO temperature beta is adapted by two factors: a data-aware factor derived from CLIP image-text similarities over sub-sentences of the chosen and rejected responses, and a model-aware factor derived from the current implicit reward gaps. The two factors are combined multiplicatively. Experiments on five benchmarks (Object HalBench, AMBER, MMHal-Bench, LLaVA-Bench, MM-Vet) report large gains over vanilla DPO on LLaVA-1.5 7B/13B, with the 7B model exceeding GPT-4V on Object HalBench non-hallucination rates. The paper also includes ablations of the two components, the probability transformation, and the outlier filtering.

Significance. If the reported results hold, DAMA offers a simple and general way to make DPO adaptive at instance and batch level, with a released implementation and a clearly described algorithm. The idea of adjusting beta by an external CLIP-based hardness estimate plus a model-state estimate is a useful contribution, and the ablations give initial support for each component. However, the quantitative claims currently rest on a small number of runs, a potentially length-biased hardness metric, and an inconsistent outlier-filtering formula, so the significance is conditional on addressing these issues.

major comments (4)
  1. [Section 3.1, Equations 4-6] The hardness measure delta is length-biased. Because the softmax in Eq. 5 is computed over the concatenated score vectors Cw and Cl and the probabilities are summed per response in Eq. 6, responses with more sub-sentences accumulate more probability mass irrespective of whether the extra sub-sentences are hallucinated. The paper provides no length-normalization control (e.g., dividing by p/q or per-response softmax) and no analysis showing that delta correlates with human or automatic hallucination judgments. Since alpha_D scales beta instance-wise, this concern directly affects the central claim that DAMA focuses training on genuinely hard hallucinated samples. A length-controlled ablation or a correlation analysis is needed.
  2. [Section 3.2, Equations 11-12] The outlier-filtering formula is internally inconsistent. Eq. 10 defines Rbar_i = Ri/Rbar, making Rbar_i dimensionless, but Eq. 11 compares (Rbar_i - Rbar)^2 to a threshold. Moreover, Eq. 12 divides by N-K while summing over the N instances where M_i=1; if K=12 selected instances out of N=16 (as stated in Section 4.1 and Figure 3), the denominator should be K. As written, the formula does not compute the average reward gap of the kept instances, and the resulting alpha_M is not well-defined. This needs to be corrected and the experiments rerun or confirmed with the corrected formula.
  3. [Abstract and Section 4.3] The abstract says DAMA-7B 'reduces response-level and mentioned-level hallucination by 90.0% and 95.3%', but Table 4 lists non-hallucination rates: 90.9% response-level and 95.3% mentioned-level. Those are not reductions; the corresponding hallucination rates would be 9.1% and 4.7%. The wording should be corrected throughout (including Section 4.3, which repeats 'reduces ... by 90.9% and 95.3%').
  4. [Section 4.1 and Tables 1/4] No error bars, multiple seeds, or statistical tests are reported for any experiment, although the text uses 'significant' for the gains. Additionally, the baseline LLaVA-1.5-7B mention-level non-hallucination rate differs across tables (73.08 in Table 1 vs. 71.2 in Table 4), and baselines are partly copied from prior papers with different evaluator versions, making it hard to assess whether the improvements are robust. At minimum, a multi-seed comparison of DPO vs. DAMA on the main benchmarks and a consistent evaluation protocol table are needed.
minor comments (6)
  1. [Eq. 15] The text says 'H is initialized to 0' but the variable is Rbar; please fix the notation.
  2. [Figures 4 and 5] The x-axis label 'Addition Ratio' is unclear; the text refers to a mixing parameter rho for the weighted-sum baseline. Rename the axis and clarify the range of rho.
  3. [Table 4] There are inconsistencies in the formatting of the improvements row (e.g., '+5.1 %' with a space) and the column header 'NON-MEN.' should be spelled consistently as 'Non-Men.'.
  4. [Table 1 vs Table 4] The LLaVA-1.5-7B mention-level number differs between Table 1 (73.08) and Table 4 (71.2); please resolve this discrepancy.
  5. [Section 3.1] The phrase 'dynamically scaling β inversely with hardness' conflicts with the described behavior (larger β for easy-to-distinguish data); please rephrase to state the direction explicitly.
  6. [Section 4.1] The implementation details say 'four epochs' and batch size N=16, but no learning rate or optimizer settings are given beyond referring to the LLaVA GitHub; please specify these for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: hardness and responsiveness signals are external or open-loop states, and reported gains are external benchmarks.

full rationale

DAMA's two adaptive signals do not reduce to the quantities they are claimed to predict. The data-aware term (Eqs. 4-8) computes hardness from CLIP image-text similarity scores, an external frozen model; no parameter of the method is fitted to the reported benchmark outcomes. The model-aware term (Eqs. 9-15) uses the DPO implicit reward gap as a state estimate to rescale beta; while this is a feedback loop (the rescaled beta is fed back into the same DPO loss), the batch-level scalar alpha_M is a filtered, normalized mean and does not equal the per-instance loss or reward by construction, so the benchmark results are not forced by an algebraic identity. The paper contains no load-bearing self-citations and no uniqueness theorem imported from the authors' prior work. Concerns that the softmax in Eq. 5-6 is length-sensitive and that the CLIP gap is a proxy for human-judged hallucination are validity and robustness issues, not circularity; they concern whether the mechanism does what is claimed, not whether the claim is assumed into the input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three empirical assumptions that are not independently validated: CLIP scores as hallucination proxies, length-insensitive hardness aggregation, and reward-gap estimates as model responsiveness. The paper also inherits the DPO framework and the RLAIF-V dataset unchanged. The free parameters beta, K, gamma, and epochs are set by hand or from prior work without sensitivity analysis.

free parameters (4)
  • beta (base DPO penalty) = 0.1
    Chosen from prior DPO and RLAIF-V settings; the paper does not tune it. It is the base scale for all per-instance beta adjustments.
  • K (outlier filtering keep count) = 12
    Set to 12 with batch size N=16. No ablation or sensitivity analysis for K is reported.
  • gamma (moving average momentum) = 0.9
    Chosen by hand for updating the mean reward gap estimate; no sensitivity analysis.
  • epochs = 4
    Training length chosen for full fine-tuning; not ablated.
assumptions (5)
  • domain assumption CLIP image-text similarity is a valid proxy for whether a response is hallucinated relative to the image.
    Invoked in Section 3.1, Equations 4-6, where hardness is defined entirely through CLIP scores.
  • domain assumption Summing softmax-normalized CLIP scores across sub-sentences gives a hardness measure that is not biased by response length.
    Eq. 6 sums probabilities over all sub-sentences of each response with no length normalization; the paper does not test this assumption.
  • domain assumption The implicit DPO reward gap of the current model (Eq. 9) is a reliable, stable measure of model responsiveness.
    Section 3.2 uses these gaps, after outlier filtering, to set the batch-level beta multiplier.
  • domain assumption The 22k RLAIF-V preference dataset is a sound base for alignment and the five benchmarks are valid, leakage-free evaluation tools.
    Section 4.1 uses the released dataset and standard benchmarks without contamination checks.
  • standard math The DPO loss and implicit reward formulation from Rafailov et al. (2024) are correct and applicable to MLLMs.
    Equation 3 is the starting point of the paper; no re-derivation is offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DAMA: Data- and Model-aware Alignment of Multi-modal LLMs." pith.science (2026). https://pith.science/paper/AABFIGWO

@misc{pith2026250201943,
  author       = {Pith},
  title        = {Pith review of: DAMA: Data- and Model-aware Alignment of Multi-modal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AABFIGWO}},
  note         = {Machine review of arXiv:2502.01943}
}
read the original abstract

Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large language models (MLLM) with human preferences. However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending to overfit on the easy-to-distinguish data while underfitting on the hard-to-distinguish data. In this paper, we propose Data- and Model-aware DPO (DAMA) to dynamically adjust the optimization process from two key aspects: (1) a data-aware strategy that incorporates data hardness, and (2) a model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effectively adapt to data with varying levels of hardness. Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, on the Object-HalBench, our DAMA-7B reduces response-level and mentioned-level hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V.

Figures

Figures reproduced from arXiv: 2502.01943 by the authors.

Figure 1
Figure 1. (1) Preference data (Prompt, Image, Preferred response yw, Rejected response yl) with different hardness: “easy-to￾distinguish” data denotes a large Image-Text sim score gap be￾tween yl and yw; “hard-to-distinguish” data indicates a low score gap between yl and yw. (2) Implicit reward across the optimiza￾tion stage: the reward gap for “easy-to-distinguish” data enhances significantly during optimization, while for “… view at source ↗
Figure 2
Figure 2. Overview of our data-aware preference optimization. For each preference instance: (1) We first break the preferred and rejected response into sub-sentences by prompting a large language model (LLM); (2) Next, we estimate the similarity scores between each sub-sentence and the given image using the CLIP classifier, and then calculate the differences between the preferred and rejected response as the hardness of the d… view at source ↗
Figure 3
Figure 3. Overview of our model-aware preference optimization. Given N preference instances: (1) we first calculate the reward gap of each instance using the implicit reward model; (2) To ensure stable modeling, we filter out the outliers (i.e. the instance with excessively high or low gaps) and then estimate the average gap; (3) To enable the model to be aware of its current responsiveness, we integrate such estimation into … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Experimental results of the combination strategies with the response-level non-hallucination rates. DPO and the improvements achieved by DAMA over DPO, are listed in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Experimental results of the combination strategies with the mentioned-level non-hallucination rates. 28% gain for 7B models, which is significant; (2) DAMA achieves new state-of-the-art over various benchmarks. Our DAMA-7B reduces the response-level and mentioned-level…
Figure 6
Figure 6. Figure 6: Qualitative Comparisons between DAMA and vanilla DPO. Data source: Object HalBench. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Comparisons between DAMA and vanilla DPO. Data source: Object HalBench. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A VLM-LLM agentic pipeline and a new 251-person benchmark show that ordinary personal photo sets can reveal private attributes, including abstract traits like income and MBTI, at rates above human evaluators.

Reference graph

Works this paper leans on

15 extracted references · 13 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308.12966, 1(2): 3,

    arXiv

  2. [3]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  3. [4]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306,

    arXiv

  4. [5]

    Token preference optimization with self-calibrated visual-anchored rewards for hallucination mitigation

    Gu, J., Wang, Y ., Cao, M., Bu, P., Song, J., He, Y ., Li, S., and Zheng, B. Token preference optimization with self-calibrated visual-anchored rewards for hallucination mitigation. arXiv preprint arXiv:2412.14487,

    arXiv

  5. [6]

    Vlfeedback: A large- scale ai feedback dataset for large vision-language models alignment

    Li, L., Xie, Z., Li, M., Chen, S., Wang, P., Chen, L., Yang, Y ., Wang, B., Kong, L., and Liu, Q. Vlfeedback: A large- scale ai feedback dataset for large vision-language models alignment. arXiv preprint arXiv:2410.09421,

    arXiv

  6. [8]

    A., Burns, K., Darrell, T., and Saenko, K

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156,

    arXiv

  7. [10]

    Y ., Xu, N., Zhang, S., Poon, H., and Chen, M

    Wang, F., Zhou, W., Huang, J. Y ., Xu, N., Zhang, S., Poon, H., and Chen, M. mdpo: Conditional preference opti- mization for multimodal large language models. arXiv preprint arXiv:2406.11839,

    arXiv

  8. [11]

    An llm-free multi-dimensional benchmark for mllms hallucination evaluation

    Wang, J., Wang, Y ., Xu, G., Zhang, J., Gu, Y ., Jia, H., Yan, M., Zhang, J., and Sang, J. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397,

    arXiv

Show all 15 references
  1. [12]

    Hallucidoctor: Mitigating hallu- cinatory toxicity in visual instruction data

    Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., and Zhuang, Y . Hallucidoctor: Mitigating hallu- cinatory toxicity in visual instruction data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12944–12953, 2024a. Y...

  2. [13]

    Automated multi-level preference for mllms

    Zhang, M., Wu, W., Lu, Y ., Song, Y ., Rong, K., Yao, H., Zhao, J., Liu, F., Sun, Y ., Feng, H., et al. Automated multi-level preference for mllms. Advances in Neural Information Processing Systems, 2024a. Zhang, R., Wei, X., Jiang, D., Guo, Z., Li, S., Zhang, Y ., Tong, C., L...

    arXiv

  3. [14]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization

    Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., and He, C. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839,

    arXiv

  4. [15]

    Provide a thorough description of the given image

    Zhou, Y ., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via pref- erence fine-tuning. arXiv preprint arXiv:2402.11411 , 2024a. Zhou, Y ., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., Wang, X., Li, Y ., Zhang, L., and Ya...

    2024 arXiv

  5. [2018]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  6. [2023]

    Liu, H., Li, C., Li, Y ., Li, B., Zhang, Y ., Shen, S., and Lee, Y . J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tun- ing. Advances in neural information processing systems , 36, 2024b. Liu, H....

    arXiv

  7. [2024]

    Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment

    Cui, C., Zhang, A., Zhou, Y ., Chen, Z., Deng, G., Yao, H., and Chua, T.-S. Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment. arXiv preprint arXiv:2410.14148,

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.