{"id":"c78453d1-3994-43e4-94da-c6ec802f8527","arxiv_id":"2507.17653","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"QuMAB models each annotator with a lightweight query in a cross-attention network, reconstructs missing labels, and reports accuracy gains over aggregation baselines on two new dense-label datasets.","lead":"This paper introduces QuMAB, a method that gives each data labeler its own small query model to learn that labeler's personal judgment patterns instead of averaging all labelers into one ground truth. It also contributes two dense multi-annotator datasets, aiming to cut annotation cost and make crowd-sourced labels more reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main accuracy comparisons do not control for backbone size; QuMAB's ViT-G/14 encoder may explain gains over baselines, not the annotator-wise query modeling.","rationale":"The reader's verdict is CONDITIONAL and notes that 'main accuracy tables do not control for the much larger backbone used by QuMAB' in its rationale; however, the reader's formally stated weakest assumption concerns temporal stability and spatial-focus reducibility of annotator behavior. I see the backbone confound as the more immediate, load-bearing threat to the paper's central claim because it undermines the primary quantitative evidence for the proposed method. The paper's own supplementary material explicitly states that different methods use different backbones and only standardizes them for the efficiency table, not for accuracy. The ablation study provides internal evidence that the Q-Former and self-attention components help, but it does not compare QuMAB against a strong baseline sharing its pretrained backbone, so the observed gains may reflect encoder capacity rather than annotator-wise modeling. This does not mean the idea is wrong; it means the current experiments cannot validate it. A controlled comparison is a concrete, feasible check that would settle the issue. I do not see grounds for rejection, because the paradigm and architecture are plausible and the ablation suggests some real effect, but the evidence is currently insufficient. Hence CONDITIONAL is appropriate, aligned with the reader's verdict.","tokens_in":114,"tokens_out":3308,"duration_ms":46880,"concrete_test":"Rerun D-LEMA, PADL, and MaDL using the same frozen EVA-CLIP ViT-G/14 features (or the same backbone and Q-Former initialization as QuMAB) with the same annotator-specific heads and training schedule, then recompute Tables 2 and 3, including standard errors over at least 5 random seeds. If QuMAB's average per-annotator accuracy and consensus accuracy no longer exceed the best baseline by a meaningful margin, the claimed advantage of annotator-wise query modeling is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that QuMAB's query-based annotator modeling outperforms existing multi-annotator methods—is not supported by the accuracy tables because the comparisons are not backbone-matched. QuMAB uses a frozen EVA-CLIP ViT-G/14 encoder and a Q-Former initialized from InstructBLIP (Section 5.1), which are very large, heavily pretrained components. The baselines D-LEMA, PADL, and MaDL are standard methods typically trained with much smaller backbones. The paper itself acknowledges in the supplementary efficiency analysis (Table 8) that 'different methods use different backbone networks' and only standardizes to ResNet-34 for that efficiency comparison—not for the accuracy experiments in Tables 2 and 3. Without a same-backbone comparison, the reported gains (e.g., Avg 0.84 vs. 0.80 on AMER, and CoPr improvements in Tables 3 and 6) could be entirely due to the stronger pretrained encoder rather than the proposed annotator-wise behavior modeling. The ablation study (Table 5) shows Q-Former and self-attention help within QuMAB's own architecture, but it does not establish superiority over strong baselines with the same backbone. This is load-bearing because the paper's headline claims of 'superiority' and 'paradigm shift' rest on these uncontrolled numerical comparisons; if the backbone confound is real, the empirical evidence for the central claim largely collapses.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QuMAB, a query-based architecture for multi-annotator learning in which each annotator is represented by a learnable query token processed by a shared Q-Former with cross-attention to input features and self-attention among annotator queries. The stated goal is a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling, with claimed benefits in reconstructing unlabeled labels, improving majority-vote consensus, and providing explainable visualizations of annotator focus regions. The authors also introduce two datasets, STREET (urban image impressions) and AMER (multimodal video emotion), and report experiments against D-LEMA, PADL, and MaDL on individual annotator accuracy/F1, consensus prediction, sparse-label robustness, ablations, and qualitative attention visualizations.","tokens_in":18925,"tokens_out":4807,"duration_ms":51062,"significance":"The annotator-wise perspective and the proposed query-based mechanism are conceptually interesting and could be valuable to the multi-annotator learning community if the empirical claims are supported. The datasets, if released, would also be a useful resource because per-annotator longitudinal labels are rare. The ablation study (Table 5) gives some internal evidence that the annotator queries, self-attention, and per-annotator classifiers contribute to the method's performance, and the consensus-reconstruction experiment in Table 6 is a legitimate prediction task. However, the paper does not provide code or data at submission time, the accuracy comparisons are not backbone-matched, and the AMER density claim is contradicted by the paper's own supplementary statistics. These issues currently prevent the central claims from being established as stated.","major_comments":[{"comment":"The main accuracy comparisons do not control for the backbone network. QuMAB uses a frozen EVA-CLIP ViT-G/14 encoder and an InstructBLIP-initialized Q-Former, while the baselines are not described as using the same backbone. The supplementary efficiency analysis (Section 7.4, Table 8) explicitly states that 'different methods use different backbone networks' and standardizes to ResNet-34 only for that efficiency comparison, not for the accuracy experiments. Because ViT-G/14 is a very large, heavily pretrained model, the reported gains in Tables 2 and 3 may be due to the encoder rather than the proposed annotator-wise query modeling. Please rerun the baselines with the same EVA-CLIP/Q-Former backbone, or otherwise provide a controlled comparison where the only difference is the annotator modeling mechanism.","section":"Section 5.1 and Tables 2–3"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the quantitative results. Several reported differences are small (e.g., AMER average accuracy 0.84 vs. 0.80 for MaDL; consensus CoPr 0.60 vs. 0.57), and the sparse-annotation results in Table 4 and the missing-label simulations in Table 6 are based on random removal without multiple runs or seeds. Please provide repeated runs with standard deviations and appropriate statistical tests, or at least mean±std over several random splits and masking seeds, so the reader can assess whether the differences are reliable.","section":"Tables 2–4 and Table 6"},{"comment":"The paper repeatedly states that AMER has 'average 3,118 labels per annotator' and calls the datasets 'dense per-annotator labels'. However, Supplementary Table 7 reports the average as 1,999.1 labels per annotator, with annotators A1–A10 each having roughly 1,000 labels (about 79–81% missing) and only A11–A13 having near-complete labels. The abstract and introduction should be corrected to match these numbers, and the term 'dense' should be qualified so that it does not overstate the coverage of most annotators.","section":"Abstract, Section 1, Table 1, and Supplementary Table 7"},{"comment":"The efficiency comparison is not consistent with the model evaluated in the main experiments. Table 8 standardizes all methods to ResNet-34, whereas Section 5.1 states that QuMAB uses EVA-CLIP ViT-G/14 and an InstructBLIP-initialized Q-Former. As a result, the reported 106.02M parameters for 'Ours' does not reflect the actual model in Tables 2 and 3, whose frozen encoder alone is far larger. The claim that the approach is 'lightweight' should either be limited to the query mechanism and trainable parameters, or the efficiency table should include the backbone parameters actually used in the main comparisons.","section":"Section 7.4, Table 8"}],"minor_comments":[{"comment":"The caption says 'k = 1, ..., 13' for the STREET dataset, but STREET has 10 annotators; this should be corrected to 10.","section":"Table 3 caption"},{"comment":"The text says 'as in AMER (1,040 vs. 5,195 labels for annotators 1–10 vs. 11–13)', but Supplementary Table 7 gives different per-annotator counts (e.g., A1 has 1,096, A11 has 5,187). Please align the numbers.","section":"Section 5.2"},{"comment":"References [21] and [22] appear to be duplicate entries for the same CIFAR-10H paper, and references [26] and [27] also appear to be the same paper on learning from multiple annotators. Please deduplicate and cite the original venues correctly.","section":"References"},{"comment":"The statement that under 40% removal 'our model’s average performance drops by 20.4%, whereas the best baseline PADL experiences a larger drop of 27.4%' should specify the exact per-method values it is computed from and how the average is taken across the STREET perspectives and AMER.","section":"Section 5.3, Table 4"},{"comment":"In the caption of Figure 7, the frame ranges and annotator numbers are described inconsistently with the main text (e.g., which annotators focus on early versus late frames). Please unify the descriptions.","section":"Section 7.6 and Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is worth considering, and the datasets could be a useful community asset if released. However, the current manuscript needs a same-backbone comparison to support the headline empirical claims, and the dataset statistics should be corrected. I would also check with the editor whether the 'we are negotiating to publish both datasets after acceptance' statement is acceptable for a data-contribution claim, since the datasets are not currently available for verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely new: represent each annotator with a learnable query in a Q-Former-style cross-attention, and use shared self-attention among queries as implicit regularization. That is a real architectural contribution to multi-annotator learning, and the two dense-label datasets (STREET and AMER) are useful additions, with AMER being the first multimodal multi-annotator dataset. Second, the empirical case for superiority over baselines is not yet made. The accuracy tables compare QuMAB, built on a frozen EVA-CLIP ViT-G/14 encoder plus an InstructBLIP-initialized Q-Former, against D-LEMA, PADL, and MaDL running on their own, typically much smaller backbones. The paper only standardizes the backbone to ResNet-34 in the efficiency table, not in the accuracy experiments. The ablation shows the query and self-attention help within QuMAB's own architecture, but it does not show that QuMAB beats a strong baseline with the same encoder. That is load-bearing because the claimed superiority and the 'paradigm shift' framing rest on those numbers.\n\nThe paper does some things well. The consensus-reconstruction experiment in Table 6 is a clean prediction task: models trained on full data predict masked test labels, and no fitted quantity is reused as the target, so there is no circularity. The visualizations of annotator focus regions are a nice qualitative complement, and the dense longitudinal annotation setup is genuinely valuable for studying annotator behavior. The related work section correctly identifies PADL, MaDL, and D-LEMA as predecessors in individual annotator modeling, though the abstract and introduction overstate the novelty by calling it a paradigm shift when the shift is more incremental.\n\nSoft spots, in proportion: the backbone confound is the biggest issue and should be fixed before publication. There are also no error bars or significance tests anywhere, the sparsity simulation appears to be single-seed, and the datasets are only promised after acceptance, not released. Those are fixable, but they matter for reproducibility.\n\nThis paper is for researchers in multi-annotator learning, crowdsourcing, and affective computing. The idea and the datasets merit referee time, but the experiments need major revision. My recommendation: send it to peer review, require same-backbone comparisons, variance estimates, and a concrete data-release plan, and let the authors either close the gap or soften the claims.","headline":"Genuinely new annotator-wise query architecture and two dense datasets, but the headline accuracy claims are undercut by uncontrolled backbone differences and missing uncertainty reporting.","tokens_in":19461,"tokens_out":1860,"would_cite":false,"duration_ms":20027,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that multi-annotator learning should shift from sample-wise aggregation to annotator-wise behavior modeling, with per-annotator queries reconstructing missing labels, improving consensus under sparse data, and…","keywords":["multi-annotator learning","annotator behavior modeling","query-based architecture","inter-annotator correlation","sparse labels","consensus prediction","multimodal emotion recognition","urban perception annotation"],"falsifier":"Ask the same annotators to re-label a held-out set weeks after their original labels, then compare QuMAB's predicted labels to the second-round labels: if agreement with second-round labels is no higher than agreement with a different annotator's original labels, the stable-behavior premise fails.","tokens_in":18418,"feed_emoji":"🏷️","tokens_out":8575,"duration_ms":83049,"temperature":0.7,"pith_summary":"The paper argues that multi-annotator learning should stop treating disagreements among annotators as noise to be averaged into a single ground truth. Instead, it proposes an annotator-wise paradigm in which each annotator's consistent judgment pattern is modeled directly, so that disagreements become informative signal. The QuMAB architecture represents each annotator by a lightweight learnable query that cross-attends to input content, while shared self-attention among queries uses inter-annotator correlations as implicit regularization against sparse labels. The intended payoff is that missing annotations can be reconstructed, consensus predictions become statistically reliable even with sparse coverage, and the queries' attention maps explain why annotators differ. To support this, the paper contributes two dense per-annotator datasets, STREET and AMER, with thousands of labels per annotator.","feed_headline":"Disagreement is data: per-annotator queries rebuild missing labels","feed_subtitle":"QuMAB models each annotator's focus pattern, reconstructs sparse labels, and makes consensus votes reliable.","key_machinery":"The load-bearing object is a set of learnable query tokens, one per annotator, inside a Q-Former-style module. Queries first attend to each other in a shared self-attention layer, which makes inter-annotator similarities act as a regularizer, and then attend to input features in multi-head cross-attention, producing annotator-specific representations that feed separate classifiers. Because the queries are vectors rather than full networks, the per-annotator cost stays low. The cross-attention weight maps are the same mechanism used for both prediction and explanation, since they indicate which input regions each annotator focuses on. The training objective is simply the sum of per-annotator cross-entropy losses.","core_discovery":"QuMAB is a query-based architecture for modeling individual annotator behavior rather than aggregating labels sample by sample. For each annotator, a small learnable query token is passed through a shared self-attention layer, then through cross-attention with features from a frozen image or video encoder, and finally into that annotator's own classifier. The paper's central hypothesis is that annotator judgment differences arise from varying focus on different regions of the input; the cross-attention weights therefore encode each annotator's behavior pattern and double as a visualization of which image patches or video frames they rely on. Shared self-attention lets annotator queries influence each other, capturing inter-annotator correlations as implicit structural regularization that prevents per-annotator models from overfitting to small label sets while preserving individual differences. The paper claims that this design outperforms aggregation-oriented baselines on individual annotator prediction, that majority voting over per-annotator predictions beats direct aggregation when test labels are missing, and that per-annotator queries degrade less under 40% label removal.","pith_inferences":["If the stable-focus hypothesis holds, the same query architecture could be used to detect annotator drift: a query whose attention pattern shifts markedly as new labels arrive would signal that an annotator's judgment criteria have changed and that their past labels should be down-weighted.","The reconstruction step suggests an active-labeling loop: use per-annotator uncertainties to choose which samples each annotator should label next, converting saved annotation cost into even denser coverage of the most informative cells of the annotation matrix.","Since AMER is multimodal (audio, video, text), the focus-region story could be extended across modalities to ask whether disagreement is explained by which modality an annotator weights, not just which frame or patch they attend to.","If per-annotator models are accurate, the consensus label could be replaced by a distribution over annotator types, which might be more honest than majority vote for subjective tasks without ground truth."],"forward_implications":["With enough dense per-annotator labels, each annotator's model can predict their label on unannotated samples, so the annotation matrix can be completed rather than averaged over disjoint subsets.","Consensus prediction by majority vote over reconstructed per-annotator predictions stays accurate when 20% to 40% of test labels are missing, where direct majority vote on remaining labels loses accuracy.","Training under sparse annotations (40% of labels removed) costs QuMAB a 20.4% average accuracy drop versus 27.4% for the strongest baseline, indicating inter-annotator regularization helps under data scarcity.","Ablation results show that disabling shared self-attention between annotator queries lowers performance, supporting the role of inter-annotator correlations as implicit regularization.","Attention visualizations link focus differences to label differences: annotators who attend to a dog in a street scene rate happiness higher, and annotators who attend to early versus late video frames label different emotions."],"supporting_citations":[{"why":"Supplies the Q-Former module that the query-based architecture is built on; frozen feature extraction with learnable queries.","marker":"[14]"},{"why":"Provides the frozen pre-trained image encoder (EVA-CLIP) used to extract input features.","marker":"[32]"},{"why":"PADL baseline that models annotator preference via Gaussian distributions; the main comparison for individual annotator modeling.","marker":"[17]"},{"why":"MaDL baseline that jointly optimizes consensus and annotator classifiers; a key comparison for aggregation-oriented methods.","marker":"[9]"},{"why":"D-LEMA baseline, an ensemble-based multi-annotator learner that the experiments must beat.","marker":"[19]"},{"why":"MER2024 supplies the raw video emotion data from which the AMER dataset is annotated.","marker":"[16]"},{"why":"Supports the claim that sparse, fragmented annotation coverage makes statistical aggregation unreliable.","marker":"[20]"},{"why":"CIFAR-10H dataset used in comparison to show existing datasets have far fewer labels per annotator.","marker":"[22]"},{"why":"QUBIQ medical segmentation datasets used in comparison to show sparse per-annotator coverage in prior resources.","marker":"[18]"},{"why":"Supports the claim that subjective tasks lack absolute ground truth, motivating a shift away from single ground-truth aggregation.","marker":"[24]"}],"fun_headline_variants":["Queries model each annotator's focus, not just a noisy average","Per-annotator queries turn sparse labels into reliable consensus","Annotator-by-annotator: QuMAB learns individualized behavior patterns","Sparse labels? QuMAB rebuilds them via annotator-specific queries","Disagreement is not noise: QuMAB models each annotator's unique focus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each annotator's judgment is stable over time and expressible as a consistent focus pattern over input regions, so that a model trained on their past labels can predict their labels on new samples.","fun_headline_variants_meta":{"raw":{"variants":["Queries model each annotator's focus, not just a noisy average","Per-annotator queries turn sparse labels into reliable consensus","Annotator-by-annotator: QuMAB learns individualized behavior patterns","Sparse labels? QuMAB rebuilds them via annotator-specific queries","Disagreement is not noise: QuMAB models each annotator's unique focus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000805,"raw_usage":{"total_tokens":3557,"prompt_tokens":985,"completion_tokens":2572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2474}},"tokens_in":601,"tokens_out":2572,"duration_ms":18825,"temperature":1.0,"reasoning_tokens":2474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:44:34.378399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask the same annotators to re-label a held-out set weeks after their original labels, then compare QuMAB's predicted labels to the second-round labels: if agreement with second-round labels is no higher than agreement with a different annotator's original labels, the stable-behavior premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PADL baseline that models annotator preference via Gaussian distributions; the main comparison for individual annotator modeling."},{"cited_title":"Multi-annotator Deep Learning: A Probabilistic Framework for Classification","cited_arxiv_id":"2304.02539","evidence_quote":"MaDL baseline that jointly optimizes consensus and annotator classifiers; a key comparison for aggregation-oriented methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"D-LEMA baseline, an ensemble-based multi-annotator learner that the experiments must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MER2024 supplies the raw video emotion data from which the AMER dataset is annotated."},{"cited_title":"Sparse Probability of Agreement","cited_arxiv_id":"2208.06161","evidence_quote":"Supports the claim that sparse, fragmented annotation coverage makes statistical aggregation unreliable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CIFAR-10H dataset used in comparison to show existing datasets have far fewer labels per annotator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QUBIQ medical segmentation datasets used in comparison to show sparse per-annotator coverage in prior resources."},{"cited_title":"Raykar, Shipeng Yu, Linda H","cited_arxiv_id":null,"evidence_quote":"Supports the claim that subjective tasks lack absolute ground truth, motivating a shift away from single ground-truth aggregation."}],"review_version":1}