{"id":"e17c38aa-6d2b-41e4-82ff-817711d06d5e","arxiv_id":"2501.13638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A bag representation built from averaged Gaussian likelihoods in latent spaces achieves state-of-the-art quantification on the LeQua T1B and T2 multiclass benchmarks.","lead":"Researchers built a neural network that estimates the share of each class in an unlabeled batch of examples, such as the share of product categories in a batch of reviews. The method posts the best published scores on two multiclass quantification competition benchmarks and comes with public code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GMNet's SOTA claim is not yet robust: single training runs and untuned baselines leave open that the advantage over HistNetQ/DQN is seed variance or hyperparameter allocation.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing issue: the comparison may not isolate the bag representation due to untuned baselines and the uncertainty measure is wrong for comparing deep learning methods. I concur. The paper is otherwise methodologically sound: the architecture is clearly specified, the code is public, and the experiments are extensive. The concern does not refute the method's utility, but it does mean the 'state-of-the-art' advantage is not statistically established. Since the reader already issued CONDITIONAL, my stress-test confirms that verdict rather than changing it. The concrete check I propose—multiple seeds plus baseline hyperparameter tuning—would settle whether the observed margins are genuine or artifacts of single-run noise, and it is directly actionable with the existing public code.","tokens_in":14412,"tokens_out":4857,"duration_ms":45968,"concrete_test":"Run GMNet, HistNetQ, and DQN(med) with R=10 independent training seeds each on all three datasets, keeping the exact same data splits, augmentation procedures, and early-stopping criterion. Additionally, perform a small validation-based hyperparameter sweep for HistNetQ over bin counts (e.g., 8, 16, 32, 64) and for DQN over pooling types, choosing the best configuration per model on the validation bags. Then report the mean and standard error of the task metric across the 10 seeds, and test the GMNet-vs-HistNetQ difference with a paired comparison across seeds (e.g., count how often GMNet wins out of 10, or a Wilcoxon signed-rank test on the 10 paired means). If the confidence intervals overlap substantially or GMNet does not win a clear majority of seeds, the state-of-the-art claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Gaussian latent-space BRM is the cause of the reported state-of-the-art results. Table 1 supports this only if the comparison isolates the BRM. However, Section 4.2 concedes that 'an exhaustive hyperparameter search for deep learning methods was not feasible due to limited resources', so GMNet's own hyperparameters (K=100, latent dim 5, 9 latent spaces, lambda=0.01) are fixed by the authors while HistNetQ used a single 32-bin histogram and DQN used untuned pooling. The advantage on T1B U+APP is 0.5433 vs 0.5711 (about 5% relative), and on T2 it is 0.7062 vs 0.7455 (about 5.6%); on T3 GMNet is actually worse than HistNetQ (0.0498 vs 0.0467). The reported standard deviations are over the 5,000 test bags, not over repeated training runs, so they do not measure training-seed variability. With one seed per method, the observed margins could easily lie within seed noise or reflect a more favorable hyperparameter choice for GMNet. The paper's own admission of untuned baselines flags this as the weakest link in the otherwise clearly described and reproducible evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GMNet, an end-to-end deep quantification method whose bag representation module (BRM) models latent-space activations with learnable multivariate Gaussian distributions. Instead of pooling or histograms, the BRM computes mean Gaussian likelihoods per component and concatenates these across several latent spaces, optionally with a CKA-based diversity regularizer. The network is trained directly on bags annotated by prevalence, optimizing the task loss (RAE or NMD). Experiments on LeQua 2022/2024 datasets (T1B, T2, T3) compare GMNet against traditional quantifiers and deep baselines (DQN, HistNetQ) under two data settings (U and U+APP). The authors report state-of-the-art results on T1B and T2 and slightly worse performance than HistNetQ on T3, and they provide an additional data-availability study and a regularizer ablation.","tokens_in":14714,"tokens_out":3270,"duration_ms":28771,"significance":"If the experimental comparison were fully controlled, the paper would make a useful contribution: it introduces a novel, permutation-invariant bag representation that is continuous and captures feature correlations, and it shows that this representation improves over pooling and histogram baselines on large public multiclass quantification benchmarks. The code is publicly available, the evaluation uses standard competition datasets and official metrics (RAE, NMD), and the paper explicitly discusses a model-specific regularizer and initialization scheme. These are concrete strengths. However, the central state-of-the-art claim currently rests on a comparison in which GMNet's hyperparameters appear to have been selected more carefully than those of the deep baselines, and all deep methods are evaluated on a single training run. The reported standard deviations are over test bags only, so they do not quantify training-seed variability. The paper's own statement in Section 4.2 that 'an exhaustive hyperparameter search for deep learning methods was not feasible' flags this as the main weakness.","major_comments":[{"comment":"The claim that GMNet achieves state-of-the-art results is not yet supported because the comparison does not isolate the bag representation module. The paper states that all deep methods use the same architecture and vary only the BRM, which is good, but it also states that 'an exhaustive hyperparameter search for deep learning methods was not feasible due to limited resources.' GMNet's hyperparameters (K=100, latent dimension 5, 9 latent spaces, lambda=0.01) are fixed, while HistNetQ uses a single 32-bin histogram and DQN uses untuned pooling. The performance margins over HistNetQ on T1B U+APP (RAE 0.5433 vs 0.5711) and T2 U+APP (0.7062 vs 0.7455) are only about 5-6% relative, and the reported standard deviations are over the 5,000 test bags, not over network initializations. With one training run per method, these margins could plausibly lie within seed noise or reflect a more favorable hyperparameter allocation for GMNet. Please report results over multiple seeds and include significance tests, or tune the baselines' hyperparameters on validation data to the same degree as GMNet's.","section":"Section 4.2, Table 1"},{"comment":"The abstract claims 'Our method achieves state-of-the-art results, both against traditional quantification methods and other deep learning approaches,' but Table 1 shows that on T3 in the U+APP setting GMNet (NMD 0.0498) is worse than HistNetQ (NMD 0.0467). The conclusion appropriately limits the claim to T1B and T2, but the abstract and introduction do not. Please qualify the state-of-the-art statement to the multiclass tasks, or explain why T3 is not considered in that claim.","section":"Abstract and Section 5"},{"comment":"Table 2 does not state whether the results are for the U or U+APP setting, but it is presumably U+APP given the values in Table 1. More importantly, the lambda value used in the main experiments (0.01) is not the best on T1B: lambda=0.1 gives RAE 0.5307 and lambda=0.001 gives 0.5345, both better than 0.01's 0.5433. This indicates that the fixed lambda is a suboptimal choice for at least one dataset, which is not in itself problematic, but it should be acknowledged and the setting must be specified. Otherwise the reader cannot tell whether the main results would improve further under a tuned lambda.","section":"Section 4.4, Table 2"}],"minor_comments":[{"comment":"The formula for the covariance initialization is hard to parse in the current typesetting; the denominator 2 and the exponent are misplaced. Please rewrite it clearly, e.g., \\Sigma_{ii} = ( \\mathrm{mean}_i( \\min_{j \\neq i} |\\mu_i - \\mu_j| ) / 2 )^2, and check that the text matches the code.","section":"Equation (9)"},{"comment":"The caption refers to 'L1' as the loss being minimized at different epochs, but L1 is not defined in the text. Define it (presumably the L1 loss or mean absolute error) in the caption or in Section 3.2.","section":"Figure 3 caption"},{"comment":"The sentence 'All networks were trained using an identical procedure to ensure comparability' is followed by early stopping details, but no information is given about learning rate, optimizer, batch size, or number of epochs before early stopping. Please provide the full training configuration in the text or supplement, since reproducibility is claimed.","section":"Section 4.2"},{"comment":"In Figures 4 and 5, the x-axis is labeled 'Thousands of labeled examples' but the description refers to 'training bags' (e.g., '200k examples (200 training bags)'). Please clarify the relationship between examples and bags in these figures, and state whether the x-axis is the number of labeled examples used to generate training bags or the number of training bags.","section":"Section 4.3"},{"comment":"The text mentions 'an algorithm like Kraemer' but the reference is to Smith and Tromble (2004); the algorithm name should be spelled consistently and linked to the correct citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is plausible and the experimental setup on standard LeQua benchmarks is a strength. However, the main 'state-of-the-art' claim is currently supported only by a comparison where GMNet's hyperparameters are fixed and the deep baselines are not tuned, and all numbers come from single training runs with standard deviations over test bags rather than over seeds. This is exactly the kind of issue that should be fixed before publication. If the authors can provide multi-seed results with significance statements or tuned baselines, the paper would likely be acceptable. The overclaim in the abstract regarding T3 should also be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GMNet is a natural extension of the HistNetQ/DQN line: instead of pooling or histogramming latent features, it averages multivariate Gaussian likelihoods over multiple latent spaces and adds an optional CKA regularizer. The writing is clear, the code is public, and the experiments are on the standard LeQua benchmarks. What is genuinely new and useful is not the headline SOTA claim but the data-efficiency result in Section 4.3: GMNet becomes competitive with strong traditional quantifiers at 200k examples where HistNetQ needs 500k. That is a real finding and it is the best part of the paper.\n\nThe soft spot is the one the authors state themselves in Section 4.2: the deep baselines were not hyperparameter-tuned, and all networks ran on a single initialization. The gains over HistNetQ are about 5% relative on T1B and T2, and GMNet is slightly worse on T3. The reported standard deviations are over the 5,000 test bags, so they do not capture seed variability. With one seed per method, a 5% margin is within the range of what training randomness could plausibly produce. The CKA ablation in Table 2 shows the method is not sensitive to lambda, which somewhat weakens the concern that GMNet's numbers are a lucky choice, but it does not fix the comparison to HistNetQ's fixed 32-bin histogram.\n\nI do not see any circularity or fabrication. The method is an incremental variant of the same group's earlier work, they say so, and the benchmark results are reproducible. The novelty is moderate: same architecture family, one new BRM layer plus a regularization trick. That is fine for a niche field.\n\nWho should read it: anyone working on bag-level representations for quantification or LLP. It deserves a serious referee: the paper is coherent, honest, and reproducible, and the data-efficiency claim is worth checking. A referee should ask for multiple seeds, significance tests, and ideally matched hyperparameter search across deep baselines. With that, the paper would be solid; without it, the SOTA claim is a promise, not a result.\n\nMy recommendation: send it to review, with a clear request for seed variance and a tuned-or-matched comparison. I would cite it for the data-efficiency finding, not for the SOTA claim.","headline":"A useful but not yet decisive bag-representation paper: the architecture is clearly better than HistNetQ in data efficiency, but the SOTA claim rests on single runs and untuned baselines.","tokens_in":15208,"tokens_out":2415,"would_cite":true,"duration_ms":21039,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modeling a bag of examples by the average likelihood of its members under learnable multivariate Gaussians in latent spaces yields the best multiclass prevalence estimates on the LeQua benchmarks.","keywords":["quantification","prevalence estimation","Gaussian latent space representation","bag representation module","deep learning","multiclass quantification","ordinal quantification","CKA regularization"],"falsifier":"Re-run the LeQua T1B, T2, and T3 comparisons with every deep method given the same exhaustive hyperparameter search, including histogram bin counts, number of Gaussians, and regularization weight, and with test statistics averaged over multiple training seeds; if a tuned HistNetQ reaches or beats GMNet's RAE on T1B and T2, the claim that Gaussian latent-space representations are superior would be refuted.","tokens_in":14234,"feed_emoji":"📊","tokens_out":6372,"duration_ms":54097,"temperature":0.7,"pith_summary":"Quantification asks how many examples of each class are in a bag, given that the class mix changes after training. Most methods correct an intermediate classifier's predictions under prior-probability-shift assumptions. This paper proposes GMNet, an end-to-end network whose bag representation is built by projecting examples into latent spaces and averaging, per Gaussian component, how likely each example is under one of K learnable multivariate Gaussians. The authors claim this continuous, permutation-invariant representation is more powerful than pooling or differentiable histograms, and that it yields state-of-the-art relative absolute error on the two multiclass LeQua benchmarks while remaining competitive on an ordinal task. If correct, it means prevalence can be estimated directly from bags by optimizing the target loss, without training a classifier or relying on shift assumptions.","feed_headline":"Gaussian latent spaces top multiclass prevalence benchmarks","feed_subtitle":"End-to-end network averages Gaussian likelihoods over bags, beating histogram and pooling quantifiers on LeQua's multiclass tasks.","key_machinery":"The core object is the Gaussian bag-representation module. For each of L latent spaces of dimension d, the network places K multivariate Gaussians with learnable means $\\mu_k$ and positive-definite covariances $\\Sigma_k$, with a sigmoid bounding each latent space to the unit hypercube. For each example $z_i$ it computes Gaussian likelihoods $p(z_i|k)$, averages them over the bag to obtain $\\bar{p}(k)$, and concatenates these K mean-likelihood values per latent space into the bag vector $r$. The representation is permutation-invariant because averaging over the bag is order-free, continuous rather than binned, and the covariance lets it capture correlations between features. Training adds a Centered Kernel Alignment penalty that encourages the L latent spaces to be dissimilar, and covariances are kept positive-definite via manifold-constrained optimization; Gaussian centers are initialized uniformly with diagonal covariances derived from half the mean of minimum pairwise center distances.","core_discovery":"On its own terms, the paper's central claim is that replacing the bag-representation module matters more than the surrounding architecture: with the same feature-extraction and quantification modules and only the bag-summarization layer changed, the Gaussian-likelihood layer outperforms average, median, and max pooling and HistNetQ's differentiable histograms. Concretely, GMNet reports RAE 0.6728 in the U setting and 0.5433 in the U+APP setting on T1B versus HistNetQ's 0.6989 and 0.5711, and 0.8816 and 0.7062 on T2 versus 0.9162 and 0.7455. On the ordinal task T3 its NMD of 0.0502 and 0.0498 trails HistNetQ's 0.0489 and 0.0467. The authors also show that GMNet needs less data than HistNetQ to stay competitive, and that a CKA-based regularizer that pushes separate latent spaces apart provides a small improvement on the multiclass tasks.","pith_inferences":["The same Gaussian-likelihood representation could be dropped into any permutation-invariant set processor, such as label-proportion learning or set regression, where the LeQua results suggest it may carry information that pooling discards.","Because the averaged likelihood is a soft, continuous analogue of a histogram, a testable extension is to let the number of Gaussians K and their covariances adapt per class or per latent dimension rather than being fixed globally.","The data-scarcity trend implies a concrete prediction: on domains where bags are small and few, GMNet's gap over histogram and pooling baselines should grow, which could be checked with synthetic bags from unrelated datasets.","The CKA regularizer's benefit appearing only on the 28-class tasks suggests that diversity among latent spaces pays off when the prevalence vector is high-dimensional; an extension would be to scale the regularization weight with the number of classes."],"forward_implications":["Quantification becomes a direct optimization problem: bag prevalence can be predicted end-to-end with the chosen loss, RAE or NMD, and no intermediate classifier or prior-probability-shift correction is needed.","Prevalence-labeled bags alone are enough to train a competitive quantifier; adding synthetic bags generated under prior shift via the APP protocol improves results further.","On the multiclass benchmark tasks, the choice of bag representation is the deciding factor among deep quantifiers, with the Gaussian layer beating pooling and differentiable histograms under identical surrounding architecture.","GMNet is less data-hungry than HistNetQ, staying competitive when only a few hundred prevalence-labeled bags are available.","The advantage is not uniform: on the ordinal task T3, HistNetQ's histograms remain slightly better, so the Gaussian representation is not a universal replacement for histogram-based bag summaries."],"supporting_citations":[{"why":"Defines the DQN architecture with feature extraction, bag representation, and quantification modules, and introduces symmetric bag training that GMNet builds on; its pooling BRMs are the weaker baselines GMNet must beat.","marker":"Qi et al. [2021]"},{"why":"Introduces HistNetQ's differentiable histograms, the previous best deep quantifier on T1B, supplies the Bag Mixer augmentation, and is the main comparative baseline.","marker":"Pérez-Mon et al. [2024]"},{"why":"Establishes that permutation-invariant set representations are necessary and sufficient for set processing, the theoretical justification for averaging likelihoods across the bag.","marker":"Zaheer et al. [2017]"},{"why":"Provides the Centered Kernel Alignment score used as the regularization term that encourages latent spaces to differ.","marker":"Kornblith et al. [2019]"},{"why":"Supplies the manifold optimization used to keep each covariance matrix positive-definite during training.","marker":"Lezcano-Casado [2019]"},{"why":"Defines the LeQua 2022 tasks T1B, T2, and T3 and the official evaluation setup with RAE and NMD that anchors the experiments.","marker":"Esuli et al. [2022]"},{"why":"Provides the QuaPy library through which the traditional quantification baselines are trained and tuned.","marker":"Moreo et al. [2021]"},{"why":"The Kraemer algorithm used by the APP protocol to draw uniform prevalence vectors for synthetic shifted training bags.","marker":"Smith and Tromble [2004]"}],"fun_headline_variants":["Gaussian bag summaries beat pooling for prevalence","Latent Gaussian layers replace classifier in quantification","Gaussian bags outdo histograms and pooling in prevalence","GMNet: Gaussian latent summaries sharpen prevalence estimates","Rethinking bag pooling: Gaussian likelihood tops multiclass prevalence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result depends on the assumption that the bag-representation module, not the choice of hyperparameters, is what makes GMNet beat HistNetQ and DQN.","fun_headline_variants_meta":{"raw":{"variants":["Gaussian bag summaries beat pooling for prevalence","Latent Gaussian layers replace classifier in quantification","Gaussian bags outdo histograms and pooling in prevalence","GMNet: Gaussian latent summaries sharpen prevalence estimates","Rethinking bag pooling: Gaussian likelihood tops multiclass prevalence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001013,"raw_usage":{"total_tokens":4257,"prompt_tokens":905,"completion_tokens":3352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":3279}},"tokens_in":521,"tokens_out":3352,"duration_ms":20218,"temperature":1.0,"reasoning_tokens":3279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:44:54.615226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the LeQua T1B, T2, and T3 comparisons with every deep method given the same exhaustive hyperparameter search, including histogram bin counts, number of Gaussians, and regularization weight, and with test statistics averaged over multiple training seeds; if a tuned HistNetQ reaches or beats GMNet's RAE on T1B and T2, the claim that Gaussian latent-space representations are superior would be refuted.","supporting_citations":[],"review_version":1}