{"id":"08f78eba-b0fd-4e16-b481-31f460fa3035","arxiv_id":"2507.12796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DeQA-Doc adapts DeQA-Score, an MLLM-based image quality scorer, to document images using pseudo-variance soft labels and resolution-flexible encoders, and reports top scores on DIQA-5000.","lead":"DeQA-Doc adapts an existing multimodal language model scorer to rate document image quality, reporting higher scores than internal baselines on the DIQA-5000 benchmark. The work matters because automatic, generalizable document quality scoring would benefit OCR, digitization, and archival systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (4) defines the KL loss as Σ p·log(q/p), which is the negative KL divergence; as written, the training objective would push predictions away from the soft label and cannot explain the reported results without a code check.","rationale":"The reader's weakest assumption is the fixed 20% pseudo-variance. I regard that as a sensitivity issue rather than the most load-bearing one: even if the true DIQA-5000 variance differs, the validation ablation still shows pseudo-variance beating linear interpolation, so the comparative claim is not destroyed by miscalibration; at worst the margin changes. The more fundamental problem is Eq. (4). A KL loss written with numerator and denominator swapped is not a KL divergence; it is the negative KL. Under the stated training setup, the level token is supervised only by L_kl, so this sign error would prevent the model from learning the target distribution at all. The strong reported correlations therefore cannot be explained by the method exactly as written. The likely resolution is that Eq. (4) is a transcription error and the released code uses the correct direction, and the provided GitHub URL makes this verifiable directly. I keep the reader's CONDITIONAL verdict rather than escalating because the concern is concrete but potentially typographical. External baselines and error bars remain a secondary limitation, but the internal inconsistency is the most load-bearing because it determines whether the described method is even the method that ran.","tokens_in":12896,"tokens_out":8905,"duration_ms":107194,"concrete_test":"Inspect the released code at https://github.com/Junjie-Gao19/DeQA-Doc for the level-token loss computation. If it uses torch.nn.functional.kl_div with log-probabilities of the prediction and the soft label as the target, equivalently Σ p log(p/ppred), then Eq. (4) is a typo and the central training claim is intact. If it literally evaluates Σ p log(ppred/p), retrain the mPLUG-Owl2 pseudo-variance configuration from Table 3 on DIQA-5000 validation and check whether the Final Score is approximately 0.8989; if training diverges or the score drops well below that, the reported gains are not produced by the described method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2, Eq. (4), defines L_kl = Σ_i p_i log(ppred_i / p_i), where p_i is the soft label and ppred_i is the model-predicted probability. For two distributions, this expression is -KL(p || ppred), not a KL divergence. Minimizing it maximizes KL(p || ppred), which drives the predicted distribution toward regions where the soft label has low probability. Since the level token receives only this KL loss and no cross-entropy loss, a model trained literally as written could not learn to match the provided quality labels. Yet the paper reports strong correlations in Tables 1-7, so either the implementation uses the conventional KL(p || ppred) = Σ p log(p/ppred) and Eq. (4) is a sign or transcription error, or the experiments rely on an objective different from the one described. This is load-bearing because the central claim that DeQA-Doc's soft-label training outperforms baselines depends on the training loss being exactly what is stated; an unverified sign error leaves the method as specified unable to produce the reported results.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeQA-Doc, an adaptation of the authors' earlier DeQA-Score MLLM-based image quality scorer to the document quality assessment task. Since DIQA-5000 provides only mean opinion scores without per-image variance, the authors construct soft labels either by assigning a fixed pseudo standard deviation of 20% of the score range or by linear interpolation between adjacent quality levels. They also relax input-resolution constraints by removing absolute position embeddings from the CLIP encoder or by switching to Qwen2.5-VL, and they combine model and prompt ensembles at inference. On the DIQA-5000 validation and hidden test splits, the paper reports final scores up to 0.9288 for an ensemble of mPLUG-Owl2 and Qwen2.5-VL variants, and claims significant improvements over existing baselines.","tokens_in":13155,"tokens_out":6301,"duration_ms":69460,"significance":"If the results are reproducible, the paper offers a practical recipe for adapting score-distribution-regression MLLMs to document images and demonstrates that a MOS-only dataset can be used with an assumed variance model. The release of code and model weights is a strength, as is the use of an external benchmark (DIQA-5000). However, the significance of the empirical claim is currently limited by the absence of external baselines on the hidden test set, the lack of uncertainty estimates, and a training objective that is written with an incorrect sign in a load-bearing equation.","major_comments":[{"comment":"The KL divergence loss is written as L_kl = Σ_i p_i log(ppred_i / p_i), which equals -KL(p || ppred). Minimizing this expression maximizes the KL divergence between the soft label and the model prediction. Because the level token receives only this KL loss, a model trained literally as specified would be pushed away from the target soft label and could not produce the reported correlations. Please correct the sign or reverse the argument order, and verify in the released code that the implemented loss matches the corrected formula. As currently written, the central training objective is inconsistent with the reported results.","section":"Section 3.2, Eq. (4)"},{"comment":"The test-phase evaluation compares only internal variants of DeQA-Doc. No published external baseline (for example, Q-Align, DeQA-Score, or a traditional DIQA method) is evaluated on the hidden DIQA-5000 test set in the same protocol. The only comparison with Q-Align and DeQA-Score appears in Table 1 on the validation split. The abstract's claim that DeQA-Doc 'significantly outperforms existing baselines' is therefore not supported by the test-phase evidence presented here; please add external baseline scores on the same test protocol or temper the claim accordingly.","section":"Section 4.4, Tables 5–7"},{"comment":"The pseudo standard deviation is fixed at 20% of the score range based on statistics from natural-image datasets (KonIQ, SPAQ, KADID). No evidence is given that document-image annotation variance is similar, and no sensitivity analysis with other pseudo-variance values is reported. Since the pseudo-variance soft label is consistently preferred over linear interpolation (Table 3), the reported advantage could depend on the imported 20% constant. Please report results for a range of pseudo standard deviation values (for example, 0.1, 0.2, and 0.3 of the score range) and, if possible, validate the assumption on a document dataset that contains per-item variance information.","section":"Section 3.1.2"},{"comment":"The text states that 'these four score results ... are obtained through submitting inferred quality scores to the official workshop website. Therefore, the SRCC and PLCC results are not available.' This is contradictory because Tables 1–7 report dimension scores that are derived from SRCC and PLCC. Please clarify how the reported numbers were actually obtained and whether the final score in Eq. (7) is computed from independently computed correlations or from a single submitted score. In addition, no error bars or significance tests are provided for any table; differences of 0.002–0.008 (for example, m0 versus m1 in Table 5) are presented as improvements without statistical support.","section":"Section 4.2"}],"minor_comments":[{"comment":"The phrase 'resolution constrains' should be 'resolution constraints', and 'challenge lays in' should be 'challenge lies in'.","section":"Abstract and Section 3.2"},{"comment":"The text says LoRA yields improvements across all attributes, but LoRA's overall score (0.9054) is lower than full fine-tuning's overall score (0.9063). Please correct this description.","section":"Section 4.3, Table 4"},{"comment":"The '5-fold ensemble' is not described precisely; please specify how the five models are trained (for example, different seeds or data subsets) and how their predictions are combined.","section":"Section 4.4, Table 6"},{"comment":"The pretraining on KonIQ used in Table 5 (model m3) is not described in the implementation details; please specify the pretraining data split, input resolution, loss, and number of epochs.","section":"Section 4.1"},{"comment":"Reference [30] is cited as 'B Series'; this should be a proper citation, for example ITU-R Recommendation BT.500, with the title and year.","section":"References"},{"comment":"The sentence describing Table 1 says 'The soft label in DeQA-Score is calculated by pseudo variance.' Q-Align uses one-hot labels rather than pseudo-variance soft labels, so the wording should be clarified to avoid implying that the same label construction applies to both methods.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially an incremental adaptation of the authors' own DeQA-Score, and the current test-phase tables are internal ablations rather than comparisons against external baselines. The sign error in Eq. (4) is the most urgent issue to verify against the released code; it should be resolved before further review. Please also ask the authors to clarify the official-workshop metric protocol in Section 4.2, since the current wording is contradictory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, DeQA-Doc is a direct, well-scoped adaptation of the authors' own DeQA-Score to document images, and the paper is mostly honest about what is inherited and what is new. Second, the training loss in Eq. (4) is written as L_kl = Σ p_i log(ppred_i / p_i). That is negative KL divergence, not KL. If the level token were trained with that objective, the model would be pushed away from the soft label and the reported correlations would not materialize. Since code is released and the results are consistent, this is probably a sign/transcription error in the paper, but it is load-bearing and has to be checked against the implementation before the method description is trusted.\n\nWhat is actually new: fixed pseudo-variance for datasets without annotation variance, linear interpolation as a cheap alternative, removal of absolute position embeddings so the CLIP-based encoder can handle large document resolutions, and ensembling across mPLUG-Owl2 and Qwen2.5-VL. The ablations are clean and useful: 1024x1024 beats 448 and 1536, LoRA matches full finetuning, model ensembling helps, prompt ensembling does not. Releasing code and weights is real evidence. The paper also reports an honest null result for prompt ensembles, which is rare and welcome.\n\nWhere it is soft: the test-phase tables compare only DeQA-Doc variants; no external baseline appears on the hidden test set. The only published baseline comparison is on the validation split, with re-implemented Q-Align and DeQA-Score. So the abstract's \"significantly outperforms existing baselines\" is not supported by the test phase as reported. There are no error bars or significance tests. The pseudo-variance ratio is a heuristic borrowed from natural-image datasets; the paper flags it and compares it against linear interpolation, but it is not validated for document annotations. The final scores come from the workshop site, so raw SRCC/PLCC numbers per dimension are unavailable, which limits reproducibility.\n\nBottom line: this is a competent incremental paper for people working on document image quality or MLLM-based quality scoring. The central mechanism is inherited, but the adaptation details are a real contribution. I would send it to peer review with the requirement that Eq. (4) be corrected and verified against the released code, and ideally that at least one external baseline be added to the test phase. I would not cite it in my own work until the loss issue is resolved.","headline":"Useful and honest adaptation of DeQA-Score to documents, but Eq. (4) is sign-flipped as written and the test phase only has internal comparisons.","tokens_in":13625,"tokens_out":5599,"would_cite":false,"duration_ms":62993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adapting an MLLM image-quality scorer to document images reaches Final Score 0.9288 on DIQA-5000.","keywords":["document image quality assessment","multi-modal large language model","score distribution regression","soft label","pseudo-variance","DIQA-5000","ensemble inference","resolution-flexible vision encoding"],"falsifier":"Measure per-image annotator variance on a random subset of DIQA-5000 by collecting multiple quality ratings per document. If the true standard deviation differs substantially from the assumed 0.2 times the score range, or if retraining with the measured variances fails to reproduce the reported gains over Q-Align and DeQA-Score, the central claim would be undercut.","tokens_in":12723,"feed_emoji":"📄","tokens_out":9945,"duration_ms":91856,"temperature":0.7,"pith_summary":"DeQA-Doc sets out to show that a multimodal-large-language-model quality scorer can be moved from natural images to document images without redesigning the core. The paper adapts DeQA-Score, which regresses a continuous quality score from a predicted distribution over five quality levels, to the document domain by supplying soft labels even though DIQA-5000 lacks annotator variance. It does this with a fixed pseudo-variance borrowed from natural-image datasets, or with linear interpolation between adjacent levels, and it relaxes CLIP's fixed resolution so high-resolution document layouts are not crushed into 448 by 448. On the DIQA-5000 test set the full ensemble of mPLUG-Owl2 and Qwen2.5-VL variants reaches a Final Score of 0.9288, which the paper reports as a clear improvement over its internally re-implemented Q-Align and DeQA-Score baselines. If this holds, MLLMs offer a practical route to document-quality assessment for OCR, digitization, and archival.","feed_headline":"MLLM scorer adapted to documents hits 0.9288 on DIQA-5000","feed_subtitle":"Variance-free soft labels and flexible resolution let MLLMs rate document readability for OCR and archival.","key_machinery":"The load-bearing mechanism is score-distribution regression with soft labels. DeQA-Score discretizes a Gaussian MOS distribution over five rating levels, applies a linear post-adjustment so the discrete expectation equals the mean score, and trains the MLLM's level token against this distribution with a KL divergence loss while other tokens use standard cross-entropy. DeQA-Doc keeps this mechanism and replaces the missing variance in two ways: a fixed pseudo-standard-deviation of 20 percent of the score range, and linear interpolation between the two adjacent levels around the mean score. To handle large document images, it removes the absolute position embeddings from the CLIP vision encoder or switches to Qwen2.5-VL, which accepts original resolution. At inference, predicted probabilities over the five levels are converted to a mean score, and ensemble inference averages the predicted distributions across models and prompt variants.","core_discovery":"The central claim is that document image quality can be scored accurately by an MLLM trained with distribution-based soft labels, even when the only supervision is a mean opinion score. DeQA-Doc treats quality as a discrete five-level distribution and supervises the level token with KL divergence against a soft label; the paper's contribution is to construct that soft label without per-image variance data and to make the vision encoder resolution-flexible. On the DIQA-5000 benchmark, the best configuration — averaging predictions from mPLUG-Owl2 and Qwen2.5-VL variants, with KonIQ pretraining and LoRA or full fine-tuning at 1024 resolution or original resolution — achieves a Final Score of 0.9288, outperforming the paper's own Q-Align baseline at 0.8523 and DeQA-Score baseline at 0.8849 on the validation split. The paper interprets this as evidence that a single MLLM-based scoring framework can generalize across image and document quality, and that ensembling complementary backbones plus preserving high input resolution matters more than prompt variation.","pith_inferences":["If document annotators have noticeably different score variance than natural-image annotators, the fixed 20-percent pseudo-variance could be miscalibrated; the paper does not test this directly, so a measurement of per-image variance on DIQA-5000 would be the natural next experiment.","Because prompt ensemble did not help while model ensemble did, the main source of DeQA-Doc's robustness may be architectural diversity rather than prompt diversity — a hypothesis the paper leaves implicit.","The resolution-flexible score-distribution recipe could plausibly transfer to other high-resolution visual quality tasks such as screen-content, medical, or archival image assessment, since none of the components is document-specific."],"forward_implications":["Document quality assessment can be driven by an MLLM trained only on mean opinion scores, so variance-free datasets are no longer a barrier to distribution-based soft-label training.","Removing absolute position embeddings from a CLIP-based vision encoder lets a fixed-resolution MLLM process 1024 or 1536 resolution documents, with 1024 performing best in the paper's ablations.","Averaging across mPLUG-Owl2 and Qwen2.5-VL variants yields consistent gains over any single model, while averaging across ten prompt templates yields no measurable gain.","Pretraining on natural-image quality data (KonIQ) and using LoRA tuning each give a small further improvement, suggesting cross-domain transfer works in both directions.","The full ensemble's 0.9288 Final Score on the DIQA-5000 test set is the paper's evidence that MLLM-based scoring can serve OCR, digitization, and archival pipelines."],"supporting_citations":[{"why":"Supplies the score-distribution regression method, soft-label discretization and post-adjustment, pseudo-variance statistics, and fidelity loss that DeQA-Doc adapts.","marker":"[46]"},{"why":"Serves as the internal Q-Align baseline and supplies the closed-set softmax inference over the five level tokens.","marker":"[37]"},{"why":"Provides the mPLUG-Owl2 backbone whose CLIP encoder's fixed resolution motivates the resolution-flexible modification.","marker":"[43]"},{"why":"Provides the Qwen2.5-VL backbone that accepts original-resolution document images and complements mPLUG-Owl2 in the ensemble.","marker":"[4]"},{"why":"Defines the CLIP vision encoder whose absolute position embeddings are removed to allow flexible input resolution.","marker":"[27]"},{"why":"Contributes the KonIQ dataset both as the source of pseudo-variance empirical statistics and as pretraining data.","marker":"[11]"},{"why":"Supports the Gaussian model of human quality scores that underlies the soft-label distribution construction.","marker":"[50]"}],"fun_headline_variants":["MLLM document quality scorer tops DIQA-5000 at 0.9288","Soft-label MLLM rates document quality, hits 0.9288","DeQA-Doc: variance-free soft labels boost document scoring","Ensembled MLLMs excel at document IQ assessment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that human score variance on document images is roughly the same as on natural images — about 20 percent of the score range — even though the DIQA-5000 dataset provides no variance to check this against.","fun_headline_variants_meta":{"raw":{"variants":["MLLM document quality scorer tops DIQA-5000 at 0.9288","Soft-label MLLM rates document quality, hits 0.9288","DeQA-Doc: variance-free soft labels boost document scoring","Ensembled MLLMs excel at document IQ assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000513,"raw_usage":{"total_tokens":2515,"prompt_tokens":991,"completion_tokens":1524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1445}},"tokens_in":607,"tokens_out":1524,"duration_ms":12293,"temperature":1.0,"reasoning_tokens":1445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:37:48.105747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure per-image annotator variance on a random subset of DIQA-5000 by collecting multiple quality ratings per document. If the true standard deviation differs substantially from the assumed 0.2 times the score range, or if retraining with the measured variances fails to reproduce the reported gains over Q-Align and DeQA-Score, the central claim would be undercut.","supporting_citations":[{"cited_title":"Teaching large language models to regress ac- curate image quality scores using score distribution","cited_arxiv_id":null,"evidence_quote":"Supplies the score-distribution regression method, soft-label discretization and post-adjustment, pseudo-variance statistics, and fidelity loss that DeQA-Doc adapts."},{"cited_title":"Q-Align: Teaching LMMs for visual scoring via discrete text-defined levels","cited_arxiv_id":null,"evidence_quote":"Serves as the internal Q-Align baseline and supplies the closed-set softmax inference over the five level tokens."},{"cited_title":"mPLUG-Owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration","cited_arxiv_id":null,"evidence_quote":"Provides the mPLUG-Owl2 backbone whose CLIP encoder's fixed resolution motivates the resolution-flexible modification."},{"cited_title":"Learning transferable visual models from natural lan- guage supervision","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP vision encoder whose absolute position embeddings are removed to allow flexible input resolution."},{"cited_title":"KonIQ-10K: An ecologically valid database for deep learning of blind image quality assessment","cited_arxiv_id":null,"evidence_quote":"Contributes the KonIQ dataset both as the source of pseudo-variance empirical statistics and as pretraining data."},{"cited_title":"Uncertainty-aware blind image quality assessment in the laboratory and wild","cited_arxiv_id":null,"evidence_quote":"Supports the Gaussian model of human quality scores that underlies the soft-label distribution construction."}],"review_version":1}