{"id":"8e2997e7-a949-48e7-a689-70c2370bc2fd","arxiv_id":"2412.15739","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VORD filters hallucinated tokens by requiring that a token's probability on the original image exceed its probability on a corrupted image, with an adaptive margin.","lead":"VORD is a new decoding rule, plus an optional training loss, that reduces object hallucinations in vision-language models by penalizing any word that becomes more likely when the input image is corrupted. It reports higher accuracy and better confidence calibration than two earlier methods, VCD and OPERA, on the POPE and MME benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VORD's core ordinal assumption is contradicted by its own motivation in Sec. 3.4: correct tokens are reported to violate Eq. (3), yet Eq. (4) rejects precisely those tokens.","rationale":"The reader's weakest_assumption correctly identifies Eq. (3) as the load-bearing premise, and my analysis agrees: the paper's own Sec. 3.4 states that desired tokens violate the expected ordering, which is precisely the failure mode that Eq. (4) punishes. This makes the concern more than a theoretical possibility; it is an internal inconsistency between the motivating observation and the proposed fix. I considered whether to move the verdict to REJECT, but the empirical results, including consistent gains across POPE and MME and the adaptive plausibility constraint, leave room for the possibility that rejected correct tokens are rare in practice (e.g., because the plausibility mask or the margin keeps most correct tokens above threshold). The missing piece is an explicit measurement of correct-token rejection rates, which the authors did not provide. Since the reader already gave a CONDITIONAL verdict with the same core concern, my read does not change the verdict; it sharpens the condition: the authors should quantify how often Eqs. (4)-(5) reject ground-truth tokens. The convexity proof in Sec. 4.2.1 is also flawed (convexity in θ does not follow from the scalar second derivative of the penalty), but that is a secondary issue because the loss's practical utility is empirical. I did not find the missing recent baselines or GPT-4o evaluation noise to be as load-bearing as the self-contradicted ordinal assumption, and I agree with the reader's overall assessment that the method is plausible but insufficiently validated on its central premise.","tokens_in":22239,"tokens_out":2505,"duration_ms":25564,"concrete_test":"Run LLaVA-v1.5-7B on the COCO captions used in POPE (or on a held-out caption set). For each ground-truth token yt in the gold caption, compute the VORD rejection predicate: Pθ(yt|v,x) + mθ < Pθ(yt|v_hat,x), using the same Mixup corruption (α=1.0) and adaptive margin as the paper. Report the fraction of ground-truth tokens rejected, both with and without the Eq. (5) plausibility constraint. Repeat across 100+ images. If the rejection rate of correct tokens is non-negligible (e.g., >2-3%), VORD is destroying valid output and the empirical accuracy gains need to be re-examined for whether they reflect hallucination suppression or over-truncation. An even more direct check is to compare BLEU/ROUGE or ChatGPT-judged faithfulness of VORD-decoded vs regular-decoded captions on the same images; a sharp drop in caption quality would confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of VORD is Eq. (4), which accepts a token only if Pθ(yt|v,x) + mθ ≥ Pθ(yt|v_hat,x), under the ordinal assumption of Eq. (3). However, Sec. 3.4 presents the motivating observation that desired/correct tokens (shown in green in Fig. 2) 'did not behave as expected' under Mixup, i.e., they had lower probability on the clean image than on the modified image. This is an internal admission that the load-bearing premise fails for the very tokens the method must preserve: if a correct token violates the inequality, VORD sets its probability to zero and destroys valid content. The adaptive plausibility constraint (Eq. 5) only restricts rejection to tokens above a probability threshold; it does not repair the ordering and can still remove correct, high-probability tokens. The paper provides no quantitative measurement of how often correct tokens are rejected by Eq. (4), and its own Table 8 shows VORD underperforms VCD on mild corruptions (e.g., Gaussian Blur F1 80.83 vs 82.18), consistent with the ordinal assumption being corruption-dependent. Without a per-token or per-sequence validation that correct tokens survive the filter, the claimed gains on POPE/MME could arise from aggressively suppressing all non-trivial tokens rather than from a sound ordinal property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VORD (Visual Ordinal Calibration), a method to reduce object hallucinations in large vision-language models by enforcing an ordinal relationship between token probabilities under clean and corrupted images. VORD has two forms: a training-free decoding filter that zeroes out tokens whose probability on the modified image exceeds their probability on the original image plus an adaptive margin, and a trainable loss that penalizes such violations. The adaptive margin is computed from the angular similarity between the visual features of the original and modified images. Experiments on POPE, MME, and LLaVA-Bench with LLaVA-v1.5, Qwen-VL, and InstructBLIP report consistent improvements in accuracy, F1, and ECE over regular decoding and against VCD and OPERA baselines, along with GPT-4oV-assessed gains in response accuracy and detailedness.","tokens_in":22499,"tokens_out":5919,"duration_ms":51124,"significance":"If the empirical claims hold, VORD offers a lightweight, training-free intervention and a complementary regularizer that improve both hallucination metrics and calibration across several backbone LVLMs. The code is publicly available, and the evaluation includes error bars over reruns, which is a strength. However, the significance is currently conditional because the central mechanistic justification—the ordinal assumption in Eq. (3)—is internally contradicted by the paper's own motivating observations, and the formal analysis of the loss function contains mathematical errors. The method may still work as an effective heuristic, but the paper does not yet establish the ordinal-calibration story it advertises.","major_comments":[{"comment":"The motivating observation in Sec. 3.4 states that desired tokens (shown in green in Fig. 2) 'did not behave as expected' under Mixup, meaning they violate the ordinal relation in Eq. (3). Yet the decoding rule in Eq. (4) rejects exactly those tokens by setting their probability to zero when Pθ(yt|v,x) + mθ < Pθ(yt|v̂,x). The adaptive plausibility constraint in Eq. (5) only limits rejection to high-probability tokens; it does not repair the ordering. This is an internal inconsistency in the central mechanism: the paper argues from the failure of ordinality and then enforces ordinality as the acceptance criterion. The paper provides no quantitative measurement of how often ground-truth or human-judged correct tokens are rejected by Eq. (4). Without such a per-token or per-sequence analysis, the observed benchmark gains could result from aggressive suppression of all non-trivial tokens rather than from a sound ordinal property. This needs to be addressed with a direct measurement of correct-token survival rates or an equivalent analysis.","section":"Sec. 3.4 and Eq. (4)"},{"comment":"The convexity proof is mathematically unsupported. In Eq. (9), θ is the full parameter vector of the LVLM, so ∂²gθ/∂²θ is a Hessian matrix, not a scalar, and the statement that 'the second derivative is always positive' is not well-defined for a matrix. Moreover, (∂gθ/∂θ)² is an outer product, and the coefficient (ψ² − ψ) is negative when 0 < ψ < 1, so the second term is not necessarily positive semidefinite. The first term ψ g^{ψ−1} H is positive semidefinite only if H is positive semidefinite, which is not established. The claim that the VORD loss is convex in θ is therefore not proven. The authors should either provide a correct convexity argument (e.g., convexity in the functional output space) or remove the convexity claim and instead describe the loss as a heuristic penalty.","section":"Sec. 4.2.1, Eqs. (8)–(9)"},{"comment":"The gradient analysis in Eq. (8) treats the margin mθ as a constant when differentiating gθ = Pθ(y|v̂,x) − Pθ(y|v,x) + mθ. However, Eq. (10) defines mθ as a function of the visual encoder parameters θ (through fθ(v) and fθ(v̂)). The correct gradient should include ∂mθ/∂θ, which is absent. Consequently, the reported gradient of Lvord is incomplete, and the same issue propagates to the second-derivative analysis. This is load-bearing for the training variant: the actual loss that is optimized differs from the analyzed one, and the claimed behavior (e.g., not penalizing when the ordinal property is obeyed) may not hold if the margin changes during optimization. The authors should correct the gradient derivation or explicitly state that they treat mθ as a fixed, non-differentiable scalar during backpropagation.","section":"Sec. 4.2, Eq. (8) and Eq. (10)"},{"comment":"The corruption ablation shows that VORD underperforms VCD on mild corruptions: Gaussian Blur (F1 80.83 vs. 82.18), Jpeg Compression (81.99 vs. 82.25), and Saturate (81.46 vs. 82.11). The authors attribute this to smaller margins and state that the corruption must be 'severe enough' for VORD to work. This is a direct acknowledgment that the ordinal assumption in Eq. (3) is corruption-dependent, and it weakens the paper's central claim that VORD calibrates based on a general ordinal relationship between modified image pairs. The paper should provide a quantitative characterization of when Eq. (3) holds and when it fails, and should report the survival rate of correct tokens under the recommended Mixup setting. Without such evidence, the choice of Mixup as the default corruption appears to be tuned to make the method work, rather than derived from the proposed principle.","section":"Appendix B, Table 8"},{"comment":"The abstract and Sec. 5.3 claim that VORD delivers 'better calibration,' but Table 7 shows that the VORD loss consistently increases ECE relative to the baseline: for example, on A-OKVQA Random with LLaVA-1.5-7B, ECE rises from 2.82 to 2.94 (ψ=1) and 3.01 (ψ=2); on MSCOCO Popular it rises from 3.14 to 3.31 and 3.56; and on GQA Adversarial from 5.63 to 5.82 and 5.85. The calibration improvement is only demonstrated for VORD decoding in Table 1, not for VORD loss. The claims should be qualified to distinguish between the two variants, and the ECE results for the loss variant should be discussed honestly.","section":"Sec. 5.3 and Table 7"}],"minor_comments":[{"comment":"The algorithms use the notation 'ˆv = α ∗ v1 + (1 − α) ∗ v2', but the text defines the Mixup coefficient as λ drawn from Beta(α, α). Using α both for the Beta distribution shape parameter and for the mixing weight is confusing; please rename the mixing weight (e.g., λ) consistently.","section":"Algorithm 1 and Algorithm 2"},{"comment":"The statement '1.0 ≥ λi ≥ λj ⇔ Pθ(yt|v,x) ≥ Pθ(yt|v̂,x)' uses λi and λj without a clear definition. Only a single λ is introduced in the preceding sentence; please clarify what λi and λj denote (e.g., two different mixing strengths) and whether the equivalence is an empirical claim or a definition.","section":"Eq. (3)"},{"comment":"The text states that VORD generates shorter responses, but Table 3 shows that for Qwen-VL, VORD has a longer average length (13.13) than regular decoding (10.41) and VCD (10.36). Please reconcile this observation with the 'Short & Sweet' claim.","section":"Table 3 and Sec. 6"},{"comment":"Figure 5 is referenced as showing the 'squared variant (ψ = 2) highlighted in magenta,' but the manuscript appears to be in black-and-white; consider using a distinguishable marker pattern or a color-blind-safe palette.","section":"Sec. 5.3"},{"comment":"The paper claims the adaptive margin 'eliminates the need for hyperparameter tuning,' but the method still requires choosing the corruption type (e.g., Mixup), the Mixup strength α, and the plausibility threshold β. Please temper this claim to reflect the remaining hyperparameters.","section":"Sec. 4.3"},{"comment":"There are several minor grammatical issues: the abstract reads 'we present VORD a simple and effective method' (missing comma), and Sec. 4.1.2 has 'Higher values ofβ results' (grammar/spacing). A careful proofread is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism of the paper is internally inconsistent with its own motivating observations, and the formal derivations contain errors (the convexity proof and the gradient analysis). However, the empirical results are plausible and the method may be a useful heuristic. These issues are fixable with additional analyses (e.g., measuring survival of correct tokens, correcting the gradient derivation, and removing the convexity claim). The paper is not suitable for acceptance in its current form, but I do not see a load-bearing error that would force rejection; a major revision with targeted experiments and mathematical corrections could make it publishable. The code release and the use of multiple backbones and rerun statistics are strengths that should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, VORD is a genuinely simple and practical contrastive decoding method: it compares token probabilities on clean vs corrupted images, rejects tokens that violate an ordinal inequality with an adaptive margin, and reports consistent gains over VCD and OPERA on POPE and MME across LLaVA, Qwen-VL, and InstructBLIP. The paper includes reruns with error bars, which is more than many papers in this area do, and the code is public. Second, the paper's core assumption is in trouble from its own Sec. 3.4. There, the authors say that desired tokens 'did not behave as expected' under Mixup, meaning correct tokens had lower probability on the clean image than on the modified image. Yet Eq. (4) sets exactly those tokens to zero. The adaptive plausibility constraint does not repair this; it only limits rejection to high-confidence tokens. The paper never measures how often correct tokens are killed by the filter, and Table 8 shows VORD falls below VCD on mild corruptions like Gaussian blur and JPEG, which is consistent with the ordinal assumption being corruption-dependent. That is a real soft spot, maybe the central one. The convexity proof in Sec. 4.2.1 is also incorrect as stated: the second derivative with respect to a high-dimensional parameter vector is not a scalar, and 'always positive' does not follow. That is a minor theoretical blemish if the loss works, but it should be fixed or removed. I also note the choice of strong corruptions appears post-hoc based on results, and the GPT-4o evaluation lacks error bars. On the positive side, the adaptive visual-similarity margin is a nice twist and the VORD loss is a clean, trainable regularizer. The empirical story is plausible and not circular: evaluation is on held-out benchmarks. Who is this for? Anyone working on LVLM hallucination who wants a plug-in decoding filter without retraining. It deserves a serious referee: the method is simple, reproducible, and shows real gains, but the ordinal premise needs quantitative validation and the theoretical claim needs correcting. My recommendation: send to peer review with major revision expected, and ask for a per-token analysis of how often correct tokens are rejected, plus experiments on mild corruptions that the current story cannot explain.","headline":"A simple contrastive decoding method with consistent gains, but the paper's own motivating observation undercuts its core mechanism and needs to be resolved before I would trust it.","tokens_in":23045,"tokens_out":1557,"would_cite":true,"duration_ms":15361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VORD claims that enforcing one ordinal inequality on token probabilities suppresses object hallucinations in large vision-language models.","keywords":["object hallucination","large vision-language models","calibration","contrastive decoding","ordinal ranking loss","Mixup","expected calibration error","POPE benchmark"],"falsifier":"Take a set of images with known ground-truth object labels and corrupt each image with the same Mixup or diffusion recipe used in the paper; count how often a ground-truth object token has strictly lower probability on the clean image than on the corrupted image. If that violation rate is non-negligible on correct tokens, then the ordinal mask in Eq. (4) is deleting valid content, and the observed POPE gains would be explained by filtering rather than by a true ordinal regularity.","tokens_in":22004,"feed_emoji":"🖼️","tokens_out":6532,"duration_ms":52734,"temperature":0.7,"pith_summary":"VORD claims that a single ordinal constraint can suppress object hallucinations in large vision-language models: for each candidate token, the probability the model assigns when viewing the true image should be at least as high as the probability it assigns when viewing a deliberately corrupted version of the same image, plus a margin. If this constraint is enforced at decoding time by setting violating tokens to zero, or trained into the model as a ranking penalty added to cross-entropy, the paper argues that generated text becomes more faithful and better calibrated. On the POPE hallucination benchmark, VORD decoding improves accuracy and F1 by up to about +2.2 and +2.3 points over regular decoding across three backbone models, while achieving lower expected calibration error. VORD loss fine-tuning raises POPE accuracy by up to +2.9 points, and combining both forms gives the best results. The practical interest is that hallucination control and confidence calibration are treated as one mechanism rather than separate add-ons.","feed_headline":"Ordinal rule cuts object hallucinations in vision-language models","feed_subtitle":"Penalizing tokens that score higher on a corrupted image boosts accuracy and calibration on POPE, MME, and LLaVA-Bench.","key_machinery":"The load-bearing object is the ordinal mask δθ = Pθ(yt|v,x) + mθ >= Pθ(yt|v_hat,x), where v_hat is a corrupted copy of the image built by Mixup or diffusion noise and mθ is an adaptive margin mθ = (1/π) arccos( (f̄θ(v)·f̄θ(v_hat))/(||f̄θ(v)|| ||f̄θ(v_hat)||) ), the normalized angular distance between the averaged visual tokens of the clean and corrupted images. In VORD Decoding the mask is applied at each autoregressive step, zeroing any token that violates the inequality and then truncating further with an adaptive plausibility constraint; in VORD Loss the same inequality is converted into a convex, piecewise-quadratic ranking penalty max(·,0)^ψ whose gradients pull the clean-image probabilities above corrupted-image probabilities. The adaptive margin is what lets the method set its own strictness from the visual similarity of the image pair, which the ablations show is necessary because fixed margins either under-filter or over-penalize.","core_discovery":"The paper's central claim is that token probabilities in LVLMs are not naturally ordinal under visual corruption, and that enforcing ordinality fixes hallucination. Concretely, for a clean image v, a corrupted image v_hat, and a text prompt x, the paper posits that tokens consistent with the image should satisfy Pθ(yt|v,x) >= Pθ(yt|v_hat,x); VORD turns this into a hard mask P_VORD = Pθ(yt|v,x) if the inequality with margin mθ holds, and 0 otherwise, and into a convex loss L_vord = max(Pθ(y|v_hat,x) - Pθ(y|v,x) + mθ, 0)^ψ added to cross-entropy. The paper reports that this suppresses hallucinated objects like <person> or <loudspeaker>, improves accuracy, precision, recall and F1 on POPE, raises overall MME perception scores, and produces shorter, more detailed open-ended answers on LLaVA-Bench, all while lowering expected calibration error.","pith_inferences":["Editorial extension: the ordinal rule is not tied to object categories; it should apply to attribute-level hallucinations such as color, position, or count whenever the corrupted image lowers the true attribute's probability, so VORD may generalize beyond the existence-focused POPE benchmark.","Editorial extension: a testable implication the paper does not pursue is that using a corruption which leaves semantically irrelevant tokens nearly unchanged could make VORD behave like a pure linguistic-prior filter; comparing Mixup against text-only corruptions would separate visual grounding from language-prior suppression.","Editorial extension: if the ordinal assumption holds only for severe corruptions, as the ablation suggests, a practical extension would be to schedule corruption intensity per image, adapting the margin so that mild corruptions still activate the constraint without deleting valid tokens.","Editorial extension: the authors note VORD could extend to LLMs and NLP via text modifications; an implicit consequence is that any input perturbation that selectively damages signal while preserving structure could support the same ordinal calibration, making the idea a general confidence-grounding recipe."],"forward_implications":["A training-free decoder can lower object hallucination rates on POPE and MME without retraining, by comparing one forward pass on the image with one forward pass on a corrupted image.","The same ordinal principle works as a fine-tuning objective: adding max(Pθ(y|v_hat,x) - Pθ(y|v,x) + mθ, 0)^ψ to cross-entropy improves POPE accuracy and F1, with the squared variant (ψ=2) slightly ahead.","Calibration improves alongside accuracy: VORD reports the lowest expected calibration error among compared decoders on the tested backbone LVLMs, so hallucination suppression does not come at the cost of overconfident predictions.","Combining VORD loss with VORD decoding yields the best overall scores, so the training-time and decoding-time forms reinforce each other.","Because the method only needs image pairs and token probabilities, it is portable to any LVLM with a vision encoder, which is why the paper tests three backbone architectures."],"supporting_citations":[{"why":"The visual contrastive decoding baseline whose corruption-based comparison VORD generalizes; supplies the main comparison on POPE, MME, and LLaVA-Bench.","marker":"[26]"},{"why":"POPE, the polling-based object probing benchmark that provides the accuracy, precision, recall, F1, and ECE numbers used to measure hallucination suppression.","marker":"[30]"},{"why":"The LLaVA-v1.5 backbone and visual instruction tuning recipe on which the VORD loss fine-tuning experiments run.","marker":"[37]"},{"why":"Mixup, the image modification used to build the corrupted image v_hat and to motivate the ordinal assumption in Eq. (3).","marker":"[70]"},{"why":"OPERA, a beam-search hallucination-reduction baseline that VORD is compared against on POPE and MME.","marker":"[22]"},{"why":"InstructBLIP, one of the three backbone LVLMs used to show the method transfers across architectures.","marker":"[8]"}],"fun_headline_variants":["VORD: ordinal token rule reduces object hallucinations","Enforce token ordering to stop LVLM object hallucinations","Calibrate token probabilities to fix object hallucinations","Ordinal loss and masking cut hallucinations in LVLMs","Simple ordinal trick stops vision-language model hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"VORD stands on the assumption that for every token that matters, the model's probability on the clean image exceeds its probability on the corrupted image by at least the margin; if a genuinely correct token ever scores higher on the corrupted image, the decoding rule sets its probability to zero and the correct answer is destroyed.","fun_headline_variants_meta":{"raw":{"variants":["VORD: ordinal token rule reduces object hallucinations","Enforce token ordering to stop LVLM object hallucinations","Calibrate token probabilities to fix object hallucinations","Ordinal loss and masking cut hallucinations in LVLMs","Simple ordinal trick stops vision-language model hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1687,"prompt_tokens":917,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":533,"tokens_out":770,"duration_ms":6970,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:07:50.952048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of images with known ground-truth object labels and corrupt each image with the same Mixup or diffusion recipe used in the paper; count how often a ground-truth object token has strictly lower probability on the clean image than on the corrupted image. If that violation rate is non-negligible on correct tokens, then the ordinal mask in Eq. (4) is deleting valid content, and the observed POPE gains would be explained by filtering rather than by a true ordinal regularity.","supporting_citations":[{"cited_title":"Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding","cited_arxiv_id":null,"evidence_quote":"The visual contrastive decoding baseline whose corruption-based comparison VORD generalizes; supplies the main comparison on POPE, MME, and LLaVA-Bench."},{"cited_title":"Evaluating object hallucination in large vision- language models","cited_arxiv_id":null,"evidence_quote":"POPE, the polling-based object probing benchmark that provides the accuracy, precision, recall, F1, and ECE numbers used to measure hallucination suppression."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"The LLaVA-v1.5 backbone and visual instruction tuning recipe on which the VORD loss fine-tuning experiments run."},{"cited_title":"Dauphin, and David Lopez-Paz","cited_arxiv_id":null,"evidence_quote":"Mixup, the image modification used to build the corrupted image v_hat and to motivate the ordinal assumption in Eq. (3)."},{"cited_title":"Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation","cited_arxiv_id":null,"evidence_quote":"OPERA, a beam-search hallucination-reduction baseline that VORD is compared against on POPE and MME."},{"cited_title":"InstructBLIP: Towards general-purpose vision-language models with instruction tuning","cited_arxiv_id":null,"evidence_quote":"InstructBLIP, one of the three backbone LVLMs used to show the method transfers across architectures."}],"review_version":1}