{"id":"680d8d9c-1e43-4acb-bf4a-e353d6ebd5cd","arxiv_id":"2412.01249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UA-MABSA weights training samples by image quality and cross-modal relevance, achieving state-of-the-art results on the Twitter-2015 dataset.","lead":"This paper presents a training trick for multimodal sentiment analysis that gives less weight to noisy or low-quality image-text pairs. The method measures image quality with OpenCV and relevance with CLIP, then reweights each sample's loss during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not shown to come from the quality scores: without a random/permuted-weight control and with thresholds seemingly tuned on test, the SOTA claim may be noise or overfitting.","rationale":"The reader's weakest assumption concerns the validity of OpenCV/CLIP scores as proxies for sample quality. I agree that this is a real risk, but the more precise and more load-bearing issue is that the experimental design cannot distinguish the specific quality signal from any arbitrary per-sample reweighting of the same scale. The paper's ablations only remove one component at a time within the proposed weighting scheme; they do not compare against a control where the weights are randomly permuted or drawn from the same distribution. Such a control is standard for sample-reweighting claims and is directly testable. Second, the threshold selection protocol is underspecified: Figure 4 is described as selecting the best threshold, but the paper does not state whether this selection used the validation or test split. Because the final reported numbers are test-set numbers, any threshold chosen by inspecting test performance constitutes a form of test-set leakage. Third, the reported improvements are small, and the paper provides averages over 5 runs but no variance or significance tests, so it is unclear whether the 0.2 to 0.6 point gaps over FITE and KEF are meaningful. These concerns are addressable rather than fatal, which is why I would keep the reader's conditional verdict: the paper should be accepted only if the authors provide the missing control, clarify or move threshold selection to a held-out split, and report error bars or significance tests. I do not see evidence of dishonesty; the method is clearly described and the transfer to TomBERT is a positive sign. The concern is about what the experiments can establish, not about the authors' intent.","tokens_in":11130,"tokens_out":4925,"duration_ms":44983,"concrete_test":"Run a control experiment on Twitter-2015 with identical training settings (same backbone, epochs, seeds) but replace each sample's weight in Eq. 9 with a random permutation of the computed UA weights, preserving the weight distribution and removing score-sample correspondence. Repeat for 5 runs and compare Accuracy/Macro-F1 against UA-MABSA and the unweighted baseline. If the permuted-weight model matches UA-MABSA within one standard deviation, the specific quality signal is not the driver. Additionally, rerun threshold selection on the validation split only, then report test results for the chosen thresholds with per-run standard deviations; if validation-chosen thresholds change the final numbers or the confidence intervals overlap KEF/FITE, the SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the OpenCV image-quality scores and CLIP image/text and aspect/image similarities in Eq. 9 identify samples whose loss should be reduced. The paper only compares the weighted loss to an unweighted loss and ablates the three score components; it never tests whether the specific assignment of scores to samples matters. A uniform or randomly permuted weighting with the same score distribution could improve robustness by acting as a regularizer or by rebalancing effective class proportions, especially given the skewed Twitter-2015 label distribution (#Neutral 1883 vs #NEG 368 in train). If such a control matches UA-MABSA, the 'data uncertainty' explanation is unsupported and the gains are not attributable to quality assessment. The concern is sharpened by the threshold-selection protocol in Section 4.4: Figure 4 is presented without stating whether thresholds were chosen on the validation or test split, and the final numbers are test-set results. With only about 0.2 to 0.6 point gaps over KEF and FITE, and no standard deviations or significance tests reported for the 5 runs, the SOTA claim may be within run-to-run noise. The 'remaining thresholds can refer to the appendix' is also unresolved in this version, leaving the full procedure unspecified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UA-MABSA, a training-time sample-reweighting method for multimodal aspect-based sentiment analysis (MABSA). For each sample, the method computes a data-quality weight from three sources: an OpenCV-based image-quality score (brightness, contrast, sharpness, color constancy, resolution, plus an OCR-derived in-image text-length term), a CLIP-based coarse-grained image-text relevance score, and a CLIP-based fine-grained aspect-image relevance score; the three terms are averaged in Eq. (9) and multiply the standard cross-entropy loss, so that low-quality samples contribute less to training. The backbone is a BERT-based caption-fusion model in the style of CapBERT and FITE. Experiments on Twitter-2015 and Twitter-2017 report the best accuracy and macro-F1 among the compared BERT-based systems on Twitter-2015 (78.88 and 74.49), smaller gains on Twitter-2017, and a loss to VLP-MABSA on Twitter-2017; the authors also show that the reweighting transfers to a second backbone (UA-TomBERT) and provide ablations of each score component. The announced contribution is a first treatment of data uncertainty in fine-grained multimodal sentiment analysis, implemented as a quality-aware loss reweighting.","tokens_in":11397,"tokens_out":9284,"duration_ms":81474,"significance":"If the empirical claims hold, this is a useful and simple plug-in contribution: it introduces a quality/uncertainty dimension that prior MABSA work does not explicitly model, and the two-backbone evaluation (UA-MABSA and UA-TomBERT) together with component ablations gives reasonable evidence that each of the three score terms carries some signal. The authors also deserve credit for excluding BERTweet-based baselines to keep the comparison fair, for including a case study that shows what the weights capture, and for an explicit Limitations paragraph acknowledging the manual thresholds and the reliance on CLIP supervision. That said, the significance is currently capped by three unaddressed points: the reported margins over the best baselines are comparable in size to run-to-run variation and no variances or significance tests are reported; the threshold-selection protocol is under-specified and appears to be test-based; and there is no control showing that the specific quality assignment, rather than generic per-sample loss reweighting, is what produces the gains.","major_comments":[{"comment":"The paper reports 'average results of 5 independent training runs' but gives no standard deviations, per-run results, or significance tests. The headline margins over the strongest baselines on Twitter-2015 are Acc +0.20 over KEF and +0.39 over FITE, and Macro-F1 +0.74 over FITE and VLP-MABSA; these gaps are of the same order as typical run-to-run variation when fine-tuning BERT-sized models, so the SOTA claim is not yet supported. Please report per-run variances and run a paired significance test (e.g., bootstrap or approximate randomization over the five runs) for the comparisons that back the SOTA claim, and report the test for both metrics.","section":"Section 4.1 / Table 2"},{"comment":"The threshold-selection protocol is not specified. It is not stated whether the thresholds for resolution score and OCR text length, and the 'remaining thresholds' deferred to an appendix that this version does not contain, were chosen on the validation split, and the performance curves in Figure 4 appear to be computed on test data; if so, the final test numbers are optimistically selected. The Limitations section itself concedes that 'the assessment of sample quality still requires manual threshold judgment.' Please specify the selection protocol, provide the full threshold table, and report test results for thresholds chosen on validation only.","section":"Section 4.4 / Figure 4"},{"comment":"The quality weights are computed using learnable projections W_v, W_s, W'_a, W'_v 'during the training process', but the paper never states whether these projections receive gradients through the reweighted loss L' in Eq. (9). If they do, the weighting is an adaptive training signal rather than a fixed quality estimate, which changes the interpretation of the method; if they are detached, the phrase 'learnable weights' is misleading and should be clarified. In addition, the image-text and aspect-image terms are temperature-scaled cosine similarities that can be negative, so the averaged weight in Eq. (9) can be negative for some samples, which would invert the sign of that sample's loss contribution; no normalization, lower bound, or clipping of the weights is described. The paper should state the effective range of the final weight and the clipping procedure, if any.","section":"Section 3.4 / Eqs. (5), (8), (9)"},{"comment":"The ablations compare the full weighting only against removing one score component or removing all weighting; they do not test whether the specific assignment of weights to samples matters. A control that randomly permutes the computed weights across training samples, preserving the weight distribution and the loss scale, would determine whether the gains come from the quality signal or from the regularizing effect of per-sample loss reweighting per se. Since the central claim is that the quality scores identify which samples should be down-weighted, this control is required to rule out the alternative explanation that any sample-dependent reweighting improves robustness on this skewed dataset.","section":"Section 3.5 / Table 3"}],"minor_comments":[{"comment":"The symbol N is used both for the number of words in the text (Section 3.1) and for the in-batch sample size (Eqs. (4) and (7)); please rename one of them.","section":"Section 3.1 / Eqs. (4), (7)"},{"comment":"The expression '* et' is presumably intended to denote multiplication by an exponential temperature e^t; please use unambiguous notation and state the temperature value used in the experiments.","section":"Eqs. (5) and (8)"},{"comment":"For the 'w/o Image Quality' ablation rows it is unclear whether the OCR text-length factor is part of the image-quality score that is removed; please specify the exact composition of each ablated term in Eq. (9).","section":"Table 3"},{"comment":"The axes and curves in Figure 4 are not fully labeled; it is unclear which metric (Acc or Macro-F1) is plotted, whether both curves share the x-axis, and which threshold values were scanned. Please add axis labels and legends and state the scan ranges.","section":"Figure 4"},{"comment":"Section 4.3 conjectures that Twitter-2017 has a high proportion of low-quality data, but Section 4.4 states that the quality-weight distributions of the two datasets are similar; please reconcile these statements or remove the unsupported conjecture.","section":"Section 4.3 vs. Section 4.4 / Figure 5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the 'data uncertainty' framing is used loosely; the reweighting in Eq. (9) is an engineering heuristic rather than a probabilistic uncertainty model, and the paper would be better positioned as sample reweighting by external quality heuristics. I would also encourage requesting a code release, since the number of unstated preprocessing details (thresholds, in-batch negative sampling, temperature values, weight clipping) makes exact reproduction unlikely without it. The paper is within the journal's scope, and with the requested variance reporting, the permutation control, and a clean threshold-selection protocol, a revision could plausibly establish the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a sample-reweighting paper for multimodal aspect-based sentiment analysis (MABSA). It scores each training sample by image quality (OpenCV criteria), image-text correlation (CLIP coarse), and aspect-image correlation (CLIP fine), then multiplies the cross-entropy loss by the average of these scores. That combination is new for MABSA, and the authors test it on two backbones (their own CapBERT-style model and TomBERT), which is a nice touch. The paper is clearly written, the ablations are sensible, and the limitations section is honest about the manual thresholds and the reliance on CLIP.\n\nThe soft spots are real but addressable. The claimed gains over the best baselines on Twitter-2015 are about 0.2 to 0.7 points in accuracy and F1, with no standard deviations or significance tests across the 5 runs. Those gaps are within run-to-run noise for this kind of model. The threshold analysis in Section 4.4 never states whether thresholds were chosen on validation or test; if on test, the final numbers carry selection bias. And the paper never runs the obvious control: a uniform or randomly permuted weighting with the same score distribution. Without that, saying the quality scores are doing the work is an interpretation, not a demonstrated mechanism. The ablation does show every component contributes, which is evidence against pure noise, but it is not the same as showing the specific scores matter — a regularizing effect of any weighting would produce a similar pattern. So the stress-test is right to flag this.\n\nThe \"data uncertainty\" framing overstates things: there is no probabilistic model, just a heuristic quality weighting. That is fine as engineering, but the label promises more. Also note that the SOTA claim holds only on Twitter-2015 macro-F1; on Twitter-2017, VLP-MABSA still beats them. The baseline set also stops around 2022.\n\nWho is this for? People working on MABSA or on sample reweighting in multimodal NLP. I would send it to review rather than desk reject; the idea is coherent and the weaknesses are fixable. For acceptance, I would want a permutation control, variance estimates, a clear validation protocol for thresholds, and ideally a more recent baseline comparison.\n\nRecommendation: engage with it, but require the extra controls before signing off.","headline":"A coherent quality-weighted loss for MABSA that gets modest gains on one benchmark, but the attribution to the quality scores is under-tested: no error bars, no permutation control, and threshold tuning that may leak test information.","tokens_in":11915,"tokens_out":4084,"would_cite":false,"duration_ms":36140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reweighting each training sample by image quality and cross-modal relevance improves multimodal aspect sentiment classification, giving the best macro-F1 among compared models on Twitter-2015.","keywords":["data uncertainty","multimodal aspect-based sentiment analysis","loss reweighting","sample quality assessment","CLIP cosine similarity","image quality","cross-modal relevance","Twitter-2015 benchmark"],"falsifier":"Run the same UA-MABSA training on Twitter-2015 with the quality weights randomly permuted across training samples while keeping the same weight distribution; if accuracy and macro-F1 stay at the reported level, the improvement is not attributable to the quality scores themselves.","tokens_in":10916,"feed_emoji":"📈","tokens_out":6837,"duration_ms":56827,"temperature":0.7,"pith_summary":"The paper claims that multimodal aspect-based sentiment models should not train every image-text sample with equal weight: samples with blurry or low-resolution images, or with weak links between text and image (or between a specific aspect and the image), inject noise that hurts fine-grained sentiment learning. To capture this, it proposes UA-MABSA, which scores each sample on image quality, coarse image-text relevance, and fine aspect-image relevance, averages the three scores, and multiplies the standard cross-entropy loss by that average. On Twitter-2015 the reweighted model reaches 78.88 accuracy and 74.49 macro-F1, the best macro-F1 in the comparison, and the same reweighting applied to TomBERT improves it as well. If the claim holds, cheap per-sample quality estimation is a portable way to make multimodal models robust to noisy real-world data.","feed_headline":"Data-quality weighting lifts multimodal sentiment accuracy to 78.88","feed_subtitle":"Weighing each sample's loss by image quality and image-text relevance beats prior multimodal sentiment models.","key_machinery":"The load-bearing identity is the uncertainty-aware loss $L' = \\mathrm{avg}(W^{\\mathrm{Image}}_i + W^{\\mathrm{IT}}_i + W^{\\mathrm{AI}}_i) \\cdot L$, where $L$ is the standard cross-entropy loss for aspect sentiment classification. Each of the three weights lies roughly in a comparable range: $W^{\\mathrm{Image}}_i$ comes from OpenCV quality heuristics and an OCR text-length penalty, while $W^{\\mathrm{IT}}_i$ and $W^{\\mathrm{AI}}_i$ come from CLIP cosine similarities computed with in-batch negative sampling. Averaging the three and multiplying the loss re-weights the training objective so that a sample that is visually clear, well matched to its text, and specifically matched to the target aspect gets a larger share of gradient update, and a noisy or weakly coupled sample gets a smaller share.","core_discovery":"The central discovery is that a per-sample training weight built from three quality signals improves aspect-level sentiment classification and can be grafted onto existing backbones. The image quality score $W^{\\mathrm{Image}}_i$ averages OpenCV-derived measures of brightness, contrast, sharpness, color, and resolution plus a penalty for the length of OCR-detected text inside the image. The coarse relevance score $W^{\\mathrm{IT}}_i$ is the CLIP cosine similarity between the image and the tweet text, and the fine relevance score $W^{\\mathrm{AI}}_i$ is the CLIP cosine similarity between the aspect phrase and the image. The final loss is $L' = \\mathrm{avg}(W^{\\mathrm{Image}}_i + W^{\\mathrm{IT}}_i + W^{\\mathrm{AI}}_i) \\cdot L$, which makes the model spend less effort on low-quality, weakly aligned samples while keeping full loss on clear and challenging ones.","pith_inferences":["A direct extension is to use the same three quality scores in other weakly supervised multimodal tasks, such as multimodal named entity recognition or visual question answering, since the weighting does not depend on sentiment labels.","The paper's threshold tuning (e.g., choosing 200 for resolution and OCR length) is manual; a learned calibrator that maps quality scores to weights without hand-set thresholds would test whether the gains persist.","Because CLIP supplies both relevance scores, an alternative experiment would use a quality estimator trained independently of the backbone, isolating whether the improvement comes from the quality signal or from CLIP's own textual priors leaking into the weighting.","The authors note that Twitter-2017, which they believe has a higher share of low-quality data, benefits less; artificially corrupting or restoring image quality in a controlled way would directly test whether reweighting helps most when low-quality samples are the minority."],"forward_implications":["On Twitter-2015, UA-MABSA reports 78.88 accuracy and 74.49 macro-F1, the best macro-F1 among the compared BERT-based models, and it stays competitive on Twitter-2017.","Slotting the same quality weights into TomBERT (UA-TomBERT) improves that baseline by 1.34 accuracy and 2.15 macro-F1 on Twitter-2015, so the reweighting transfers across backbone designs.","Ablation shows that each of the three scores contributes; removing the aspect-image relevance score hurts most on Twitter-2015, while removing the image quality score hurts most on Twitter-2017.","Because the method only changes the loss function, it can be combined with prior MABSA systems by reweighting their losses instead of redesigning their encoders."],"supporting_citations":[{"why":"Supplies the CLIP uni-modal encoders and temperature scaling used to compute coarse image-text and fine aspect-image relevance scores.","marker":"(Radford et al., 2021)"},{"why":"CapBERT is the caption-based backbone whose design UA-MABSA extends and is a primary baseline it outperforms.","marker":"(Khan and Fu, 2021)"},{"why":"FITE motivates focusing on visual sentiment and is the strongest recent baseline; UA-MABSA replaces its facial-expression reliance with direct quality assessment.","marker":"(Yang et al., 2022a)"},{"why":"TomBERT is both a compared baseline and the backbone used in the UA-TomBERT transfer experiments, and the source of the Twitter-2015/Twitter-2017 benchmark tasks.","marker":"(Yu and Jiang, 2019)"},{"why":"Establishes data uncertainty learning in face recognition, the formulation the paper transfers to multimodal sentiment.","marker":"(Chang et al., 2020)"},{"why":"VLP-MABSA is the task-specific pretraining baseline whose competitive results UA-MABSA matches with a simpler approach.","marker":"(Ling et al., 2022)"}],"fun_headline_variants":["Per-sample quality weights beat prior multimodal sentiment models","Triple quality signal lifts multimodal sentiment on Twitter-2015","Data-quality weighting sets new SOTA on Twitter-2015","Weighting loss by image quality and relevance beats SOTA","UA-MABSA: quality-aware loss for multimodal sentiment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weights rest on the premise that OpenCV image metrics and CLIP cosine similarities rank sample quality correctly; if they mislabel informative samples as low quality, reweighting would suppress useful learning and the gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Per-sample quality weights beat prior multimodal sentiment models","Triple quality signal lifts multimodal sentiment on Twitter-2015","Data-quality weighting sets new SOTA on Twitter-2015","Weighting loss by image quality and relevance beats SOTA","UA-MABSA: quality-aware loss for multimodal sentiment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00108,"raw_usage":{"total_tokens":4514,"prompt_tokens":934,"completion_tokens":3580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":3509}},"tokens_in":550,"tokens_out":3580,"duration_ms":24922,"temperature":1.0,"reasoning_tokens":3509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:31:32.995830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same UA-MABSA training on Twitter-2015 with the quality weights randomly permuted across training samples while keeping the same weight distribution; if accuracy and macro-F1 stay at the reported level, the improvement is not attributable to the quality scores themselves.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP uni-modal encoders and temperature scaling used to compute coarse image-text and fine aspect-image relevance scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CapBERT is the caption-based backbone whose design UA-MABSA extends and is a primary baseline it outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VLP-MABSA is the task-specific pretraining baseline whose competitive results UA-MABSA matches with a simpler approach."}],"review_version":1}