{"id":"d8f91beb-2d67-43b7-b39d-f5a751fac96c","arxiv_id":"2607.08493","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Ensemble Diversity Optimization jointly learns ensemble weights, size, and a signed diversity regularizer, substantially improving calibration to annotator distributions on subjective text classification.","lead":"EDO learns ensemble weights and size with a signed diversity regularizer so models keep or damp annotator disagreement instead of collapsing it. It cuts cross-entropy 40–78% on four subjective NLP benchmarks while keeping competitive F1, useful for moderation and soft-label training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged s-selection issue; the central claim holds under the paper's stated conditions.","rationale":"The paper's central claim is empirical and compositional: jointly optimizing ensemble structure (weights + effective K via Gumbel–Softmax) together with a validation-tuned signed diversity regularizer yields large, stable gains in soft calibration (CE reductions of 40–78 % vs. Soft-CE/Soft-MD/Top-5/WEL) while preserving competitive micro-F1 and better MD/BS alignment on the four LeWiDi benchmarks. The supporting evidence (Tables 2–6, Appendices D–E, multi-seed stds) is consistent with that claim under the experimental protocol given. The single most load-bearing assumption is precisely the one the reader identified: that the development-set choice of s correctly matches the latent disagreement structure and transfers. Because the paper already states this limitation and because no deeper mathematical or experimental contradiction appears, the appropriate action is to leave the reader's CONDITIONAL verdict unchanged. The concrete test above would quantify how costly an s error actually is; if the degradation is large, the conditional acceptance would need stronger guidance or an automatic s selector before full acceptance.","tokens_in":25926,"tokens_out":635,"duration_ms":6487,"concrete_test":"For each of the four datasets, re-run the full Optuna + NSGA-II pipeline twice with s forced to the opposite of the selected sign (keeping all other search spaces identical); if the resulting Pareto-front CE on the held-out test set rises by more than ~15–20 % relative to the reported EDO-Random numbers in Table 5, the claimed navigation of the utility–calibration trade-off is shown to be brittle to s misspecification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption correctly isolates the most load-bearing premise: that a single fixed s ∈ {−1,+1} chosen on the development set correctly steers the optimizer toward preserving genuine subjectivity versus suppressing structural artifact, and that this choice generalizes to test without an automatic detector of disagreement type (Methodology §3.3; Limitations §6). No stronger internal inconsistency or experimental flaw is present. The joint optimization of weights, Gumbel–Softmax cardinality, class-weighted CE, soft F1, and reliability-weighted L_Div is well-specified (Eqs. 2–6, 10–12), the ablations (Tables 2–4, Appendix E) and multi-seed stability (Table 6) support the reported CE/BS reductions (Table 5), and the model-agnostic prediction-space design is coherent. The paper itself flags the s-dependence and lack of a disagreement-type detector as a limitation, so the concern is already surface-level rather than hidden.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces Ensemble Diversity Optimization (EDO), a model-agnostic prediction-space framework for subjective NLP classification under annotator disagreement. It jointly optimizes reliability-aware ensemble weights, effective cardinality (via Gumbel–Softmax relaxation of K), and a multi-objective loss combining soft micro-F1, class-weighted cross-entropy, L2 regularization, and a signed reliability-weighted pairwise L1 diversity term L^(s)_Div (s ∈ {−1,+1}). The sign and loss weights are validation-tuned. On four LeWiDi benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) with frozen BERT/AraBERT backbones, EDO-Random reports large CE reductions (40–78% vs Soft-CE/Soft-MD/Top-5/WEL) and lower Soft Brier scores while keeping competitive micro-F1 and competitive or better MD, with multi-seed stability and ablations on aggregation, weighting, and signed diversity.","tokens_in":26250,"tokens_out":1130,"duration_ms":11956,"significance":"If the empirical gains hold, EDO supplies a practical, lightweight, and model-agnostic way to improve probabilistic calibration and alignment with annotator distributions without requiring annotator metadata or end-to-end backbone fine-tuning. The combination of differentiable cardinality selection, class-weighted soft-label CE, and a signed diversity regularizer that can either preserve or suppress disagreement is a useful operationalization of the bias–variance–diversity decomposition for subjective supervision. Strengths include clear multi-objective formulation (Eqs. 2–6, 10–12), extensive ablations (Tables 2–4, App. D–E), multi-seed stability (Table 6), qualitative examples, and public code. The work is of clear interest to the perspectivist/soft-label and ensemble-calibration communities.","major_comments":[{"comment":"The central claim that signed diversity enables controlled navigation of the utility–calibration trade-off rests on a single fixed s ∈ {−1,+1} chosen on the development set (Methodology §3.3, Eq. 6; hyperparameter search §4.3). Limitations §6 correctly notes that EDO “does not distinguish principled subjective variation from artifact-driven divergence.” Table 2 and Appendix E show that the preferred sign is dataset-dependent (especially under severe imbalance on HS-Brexit). Without an automatic detector or a sensitivity analysis that reports how often the validation-chosen s is suboptimal on test, the claimed steering mechanism remains an untested modeling assumption rather than a demonstrated capability. A concrete addition—e.g., an oracle upper bound with test-set s, or a simple entropy/imbalance heuristic for choosing s—would make the claim load-bearing rather than contingent.","section":null},{"comment":"Table 5 reports large CE/BS gains for EDO-Random, yet the comparison mixes single-model soft-label baselines (Soft-CE, Soft-MD) with ensemble methods (Top-5 Voting, WEL). The paper freezes backbones and optimizes only in prediction space; it is therefore unclear how much of the CE reduction is attributable to the signed diversity + Gumbel-K machinery versus simply having a larger, re-weighted ensemble of already-trained models. An ablation that applies the same reliability weighting and learned K without L^(s)_Div (or with λ_Div = 0) on the identical frozen predictions would isolate the contribution of the signed regularizer that the abstract and introduction emphasize.","section":null}],"minor_comments":[{"comment":"Theorem 1 (Appendix A) only restates the triangle inequality relating pairwise L1 to barycenter dispersion; the “theoretical note” framing slightly overstates novelty. A shorter statement that L_Div is a monotone proxy for weighted predictive spread would suffice.","section":null},{"comment":"Figure 2 Spearman correlations are informative but the caption and surrounding text do not report sample size (number of Optuna trials) or multiple-testing correction; the asterisks for p < 0.05 should be interpreted cautiously.","section":null},{"comment":"Notation for soft labels is inconsistent: §3.1 uses ¯y_i while later equations use y_i,c for soft targets; a single convention would improve readability.","section":null},{"comment":"Table 1 reports Neg:Pos ratios and annotator ranges; adding the average number of annotators per instance (or total annotation count) would help readers judge the reliability of the soft labels.","section":null},{"comment":"The qualitative examples (Appendix G) are helpful; rendering the original Arabic script (or providing a reliable transliteration table) would avoid the ASCII workaround.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical multi-objective ensemble paper that already flags its main modeling assumption (fixed s). The two major points are fixable within revision and do not indicate hidden circularity or experimental error. Fit for a methods-oriented ML/NLP venue is good once the s-dependence and contribution isolation are tightened. No concerns about citation pattern or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that EDO actually moves the needle on soft calibration for annotator-disagreement tasks. On the four LeWiDi benchmarks it cuts cross-entropy 40-78% versus Soft-CE, Soft-MD, Top-5 and WEL while keeping F1 competitive and lowering Brier scores, all while staying model-agnostic and cheap because everything happens in prediction space on frozen backbones.\n\nWhat is new is the joint package: reliability-weighted ensemble weights, Gumbel-Softmax for learning effective cardinality end-to-end, a signed diversity term (s = +1 or -1) that can either preserve or suppress disagreement, class-weighted CE for imbalance, and a soft-F1 surrogate, all optimized together with Pareto selection. It cleanly extends WEL by making diversity an explicit, directional regularizer rather than an emergent side-effect. The ablations (aggregation variants, signed diversity marginals, weighted vs uniform) and multi-seed stability numbers are thorough and support the claim. Theorem 1 is just a clean triangle-inequality bound linking pairwise L1 to barycenter dispersion; nothing fancy, but it grounds the regularizer. Code is promised and the setup is reproducible.\n\nThe soft spot is exactly the one the paper names in the limitations: s is a single validation-chosen hyperparameter with no automatic detector of whether the observed disagreement is genuine subjectivity or structural artifact (imbalance, sparse coverage). If you pick the wrong sign the navigation of the utility-calibration frontier can go the wrong way. That is a real practical limitation, not a fatal flaw; it is already surface-level and the rest of the machinery still works. Hyperparameter count is high but handled by Optuna + NSGA-II in the usual way. Frozen backbones are intentional for isolation and cost, not a hidden weakness.\n\nThis is for anyone building or evaluating soft-label or multi-annotator pipelines in moderation, hate-speech or similar subjective classification. The math is elementary, the data are standard, the citations are appropriate, and the empirical case is solid. I would send it to peer review without hesitation; a good referee will push for clearer guidance or a simple heuristic for choosing s, but the contribution is already worth the time. Worth reading and, if you work in the area, worth citing.","headline":"Practical prediction-space ensemble that delivers real CE/BS gains on subjective NLP by jointly learning weights, size and signed diversity; the fixed s choice is the main caveat the authors already flag.","tokens_in":26840,"tokens_out":576,"would_cite":true,"duration_ms":11497,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Jointly optimizing ensemble weights, size, and a signed diversity regularizer improves calibration to annotator disagreement by 40–78% while keeping competitive F1.","keywords":["ensemble diversity","subjective supervision","annotator disagreement","probabilistic calibration","Gumbel-Softmax","soft labels","multi-objective optimization","prediction-space learning"],"falsifier":"On a held-out subjective dataset whose disagreement structure is independently known (true subjectivity versus imbalance-driven noise), if the validation-chosen sign yields higher cross-entropy and worse F1 than the opposite sign or than the same objective with diversity turned off, the steering claim is falsified.","tokens_in":26831,"feed_emoji":"📊","tokens_out":843,"duration_ms":20865,"temperature":0.7,"pith_summary":"Subjective NLP tasks produce systematic annotator disagreement that should be treated as signal rather than noise. This paper introduces Ensemble Diversity Optimization (EDO), a prediction-space method that learns ensemble weights and effective size end-to-end while a validation-tuned signed diversity term either preserves or suppresses disagreement. The joint multi-objective loss balances a soft F1 surrogate, class-weighted cross-entropy, and reliability-weighted diversity so the ensemble neither collapses nor overfits dominant labels. On four soft-label benchmarks the approach substantially lowers cross-entropy and Brier score relative to soft-label and ensemble baselines, keeps micro-F1 competitive, and aligns predictions more closely with the full annotator distributions. The result matters for any setting—content moderation, hate speech, sentiment—where a single hard label discards genuine human multiplicity and poorly calibrated uncertainty is costly.","feed_headline":"Signed diversity cuts calibration error 40-78% on subjective NLP","feed_subtitle":"EDO learns ensemble size and weights to match annotator distributions without collapsing disagreement","key_machinery":"The signed diversity regularizer L_Div^(s) = s · (reliability-weighted pairwise ℓ1 disagreement), with s ∈ {−1, +1} chosen on validation data, jointly optimized with soft F1 and class-weighted CE while ensemble weights w and size K are learned via Gumbel-Softmax relaxation.","core_discovery":"EDO demonstrates that treating diversity as a signed, reliability-weighted regularizer inside a single differentiable objective—while simultaneously learning ensemble cardinality via Gumbel-Softmax—produces convex combinations of frozen model predictions that match annotator soft-label distributions far more closely than fixed ensembles or single-model soft supervision, cutting cross-entropy 40–78 percent without sacrificing competitive micro-F1.","pith_inferences":["The same signed-diversity mechanism could transfer to other partial-label or multi-hypothesis settings where candidate labels arise from sources other than human annotators.","An automatic detector of disagreement type would eliminate the need for manual sign selection and improve robustness under distribution shift.","Lightweight adapters or heterogeneous experts could enlarge the hypothesis space while retaining the method’s prediction-space efficiency."],"forward_implications":["Practitioners can improve soft-label calibration on imbalanced subjective tasks by adding a cheap, validation-tuned signed diversity term rather than hand-designing ensemble size or architecture.","Because optimization occurs entirely in prediction space, any set of frozen probabilistic classifiers can be combined without retraining backbones.","Reliability weights emerge automatically from the joint gradients, down-weighting members that contribute unstructured noise.","The signed coefficient supplies an explicit, controllable direction along the utility–calibration Pareto frontier under genuine label multiplicity."],"fun_headline_variants":["EDO signed diversity cuts CE 40-78% while matching annotator distributions","Signed diversity regularizer learns ensemble size to curb calibration error 40-78%","Gumbel-Softmax EDO steers diversity to cut Brier and CE without F1 loss","Reliability-weighted signed diversity prevents collapse on subjective NLP tasks","Joint weight-cardinality optimization matches soft labels better than fixed ensembles"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"A single diversity sign picked once on the development set correctly decides whether observed disagreement is genuine subjectivity worth preserving or structural artifact worth suppressing, and that choice generalizes to test data.","fun_headline_variants_meta":{"raw":{"variants":["EDO signed diversity cuts CE 40-78% while matching annotator distributions","Signed diversity regularizer learns ensemble size to curb calibration error 40-78%","Gumbel-Softmax EDO steers diversity to cut Brier and CE without F1 loss","Reliability-weighted signed diversity prevents collapse on subjective NLP tasks","Joint weight-cardinality optimization matches soft labels better than fixed ensembles"]},"model":"grok-4.5","effort":"low","cost_usd":0.007282,"raw_usage":{"total_tokens":1763,"prompt_tokens":776,"num_sources_used":0,"completion_tokens":85,"cost_in_usd_ticks":72820000,"prompt_tokens_details":{"text_tokens":776,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":902,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":776,"tokens_out":85,"duration_ms":9938,"temperature":1.0,"reasoning_tokens":902,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T06:33:29.063694+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out subjective dataset whose disagreement structure is independently known (true subjectivity versus imbalance-driven noise), if the validation-chosen sign yields higher cross-entropy and worse F1 than the opposite sign or than the same objective with diversity turned off, the steering claim is falsified.","supporting_citations":[],"review_version":1}