{"id":"a236ca32-fbce-4b1a-9f19-69bfe682ccc5","arxiv_id":"2411.14654","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical comparison finds that Mean pooling works best for BERT sentiment classification while Weighted Sum works best for GPT-2, though differences are small and reported without error bars.","lead":"This paper compares three pooling methods (Mean, Max, Weighted Sum) for sentence-level sentiment analysis on BERT and GPT-2 models using the IMDB dataset. It reports small performance differences, with Mean best for BERT and Weighted Sum best for GPT-2, and recommends task-specific pooling choices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The architecture-specific pooling ranking rests on single-run F1 point estimates with no variance or significance testing, so the reported ordering may be training noise.","rationale":"The reader's weakest assumption identifies the same core issue: single-run point estimates cannot support the comparative ranking that the central claim depends on. My reading of the paper confirms this. The paper is a small, plausible empirical study, and the central claim—that pooling choice can affect sentence-level sentiment performance differently across architectures—is not inherently unreasonable. However, the only evidence for it is the F1 ordering in §4.3.2, and that ordering is presented without any noise model. Without repeated runs or significance tests, the reported differences may well be within normal fine-tuning variance, especially given the modest dataset size and learning rates used. This is not an external disagreement with common practice; it is an internal evidential gap. The dataset balance contradiction in Table 1 is real and should be corrected, but it is secondary: even with a perfectly balanced dataset, unreplicated point estimates would not support the conclusion. A concrete rerun with multiple seeds and paired tests would settle the question: if the ordering persists and is significant, the claim gains support; if not, the conclusion should be weakened to an observation about this particular run. Therefore the appropriate verdict remains CONDITIONAL, matching the reader's assessment, with no change needed.","tokens_in":6019,"tokens_out":2975,"duration_ms":33626,"concrete_test":"Re-run each model×pooling configuration for at least 10 random seeds with identical hyperparameters and data splits, then report mean ± standard deviation of F1 for each configuration. For the best versus second-best comparison within each architecture, apply a paired significance test (e.g., McNemar's test on the test-set predictions, or a paired bootstrap over test sentences). If the best pooling method is not consistently ranked first across seeds, or if the difference is not significant at p<0.05, the claim that pooling choice matters differently for BERT versus GPT2 is not empirically established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that 'each pooling mechanism exhibits unique strengths and weaknesses depending on the task's specific requirements.' The operational evidence is in §4.3.2: BERT's best F1 is 87.22% with Mean pooling, and GPT2's best F1 is 88.38% with Weighted Sum pooling. These numbers are presented as isolated point estimates, with no standard deviations, no repeated seeds, and no significance tests reported anywhere in §4.3. In transformer fine-tuning on ~30k IMDB reviews with a backbone learning rate of 1e-5, run-to-run F1 variation of 1–2 points is common. The reported differences between the best and second-best pooling methods within each architecture appear to be within that range, so the observed ordering—and therefore the conclusion that pooling effectiveness is architecture-dependent—could simply reflect random seed variation. Because every practical recommendation in §4.3.3 is derived from this ranking, the missing uncertainty quantification is the load-bearing weak point. A secondary but related issue is the dataset description: §4.1 claims an equal positive/negative distribution, while Table 1 shows unequal counts in all splits; this affects metric interpretation but is not the principal threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of Mean, Max, and Weighted Sum pooling for sentence-level sentiment classification using BERT-base and GPT2 on the IMDB dataset. The methodology section formalizes the three pooling operations, and the experiments report confusion matrices, precision, recall, and F1 scores. The central claim is that pooling effectiveness is architecture- and task-dependent, with BERT peaking at 87.22% F1 under Mean pooling and GPT2 peaking at 88.38% F1 under Weighted Sum pooling, leading to the practical recommendations in §4.3.3.","tokens_in":6244,"tokens_out":4301,"duration_ms":39070,"significance":"If the reported ranking is reliable, the paper would supply a useful practical datapoint for choosing pooling layers in transformer-based sentiment classifiers, and it would challenge the common practice of defaulting to Mean or CLS pooling. The experiments are simple and transparent enough that the central claim is directly testable by other groups. The significance is, however, limited by the absence of statistical uncertainty quantification; as it stands, the empirical contribution is a set of point estimates rather than a demonstrated architecture-dependent ordering. No code or data artifacts are provided, so reproducibility rests entirely on the written description.","major_comments":[{"comment":"The central claim that pooling effectiveness is architecture-dependent rests on single-run F1 point estimates. The text reports, for example, BERT's peak F1 of 87.22% with Mean pooling and GPT2's peak of 88.38% with Weighted Sum pooling, but no standard deviations, no repeated-seed runs, and no significance tests are reported anywhere in §4.3. In fine-tuning experiments of this size, run-to-run variation of 1–2 F1 points is common, and the gaps between pooling methods visible in Figure 4 appear to be in that range. Because every recommendation in §4.3.3 is derived from this ranking, the missing uncertainty quantification is load-bearing: the observed ordering could be training noise.","section":"§4.3.2, Figure 4"},{"comment":"The text in §4.1 states that the dataset is 'meticulously balanced between positive and negative sentiments' and that the 'deliberate equal distribution' mitigates class imbalance, but Table 1 reports unequal counts in every split: 16,358 positive versus 13,642 negative in training, 2,466 versus 2,534 in validation, and 7,634 versus 7,366 in test. This is a factual inconsistency that affects the interpretation of accuracy and F1 as well as the reproducibility of the experimental setup. The authors should either provide the actual balanced split or revise the dataset description and discuss the small class imbalance.","section":"§4.1, Table 1"},{"comment":"Weighted Sum pooling is defined as a sum of token embeddings with learnable weights, but the paper does not specify how the weights are produced, whether they share parameters across sequence positions, or how they are normalized. Without this detail the equation is not a well-defined architecture for variable-length inputs, and the claim that Mean and Max pooling are 'special cases' is loose: Mean is a special case only for uniform weights, and Max can be obtained only if the weights are data-dependent and concentrated on the maximum token, which is not how the learnable weighted sum appears to be defined. This methodological gap is relevant because Weighted Sum is reported to be the best method for GPT2.","section":"§3.2.3, Eq. (4)"}],"minor_comments":[{"comment":"The related work section lists many works in a block ([4–9]) without connecting them to the argument; consider trimming or integrating these citations into the narrative.","section":"§2, Related Work"},{"comment":"The claim of an 'aggregate correct prediction rate of approximately 86.07%' is stated without showing the arithmetic or the underlying confusion matrices in the text; please state how this number is computed.","section":"§4.3.1"},{"comment":"Equation (1) has a typographical issue: the square root sign should apply to d_k, and the denominator notation should be made consistent with the softmax expression.","section":"§3.1, Eq. (1)"},{"comment":"The paper is labeled as a conference paper (ISMSI '25), but the header omits the proceedings volume and page numbers; please complete the bibliographic metadata.","section":"Header"},{"comment":"The phrase 'comprehensive experiments' overstates the scope: the evaluation uses a single dataset, one split, and two model families; a more cautious wording would better match the experiments.","section":"Abstract"},{"comment":"The statement that remaining hyperparameters use 'default configurations' is underspecified; please list the exact optimizer, epochs, batch size, warmup, and sequence length used, because the pooling comparison depends on these choices.","section":"§4.2.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is a small empirical study that appears to be aimed at a workshop or conference venue; for a journal, the absence of uncertainty quantification and the dataset/count inconsistency are the decisive issues. The novelty is modest, and the literature review contains a number of loosely connected self-citations, so I would recommend weighing the venue fit carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, readable benchmark of Mean, Max, and Weighted Sum pooling on BERT and GPT-2 for IMDB sentiment. The setup is sensible and the practical recommendations are the usual guidance, but the headline result—that the best pooling method depends on the architecture—is not backed by the evidence. The F1 gap between best and second-best pooling within each model is about a point, and no variance or significance testing is reported, so the ordering could be seed noise.\n\nWhat is genuinely here: a clear statement of the three pooling equations, a standard experimental setup on 30k IMDB reviews, and an explicit per-architecture recommendation table. The paper is straightforward and does not overclaim beyond its own results in the abstract, though the conclusion section makes broader generalizations.\n\nThe soft spots are load-bearing. Section 4.3.2 gives point estimates only. In transformer fine-tuning at lr 1e-5, run-to-run F1 spread of 1–2 points is normal, so the 87.22% vs. ~86.x% BERT differences and the 88.38% GPT2 result are not distinguishable from noise. The dataset description in §4.1 claims balanced positive/negative but Table 1 shows 16,358 vs 13,642 in training; that inconsistency needs correction. There is also no code release or details on seeds, number of runs, or early stopping, so the experiments are not reproducible as reported. The comparison with prior work is thin—Mean and Max pooling on sentence embeddings have been compared before, and the paper's novelty is limited to applying them to GPT-2 and reporting numbers.\n\nWho this is for: practitioners wanting a quick sanity check on pooling choices might skim it, but they should not change their defaults on this evidence. As a research contribution it does not clear the bar for a serious referee: the central question is reasonable, but the experimental design is not adequate to answer it. I would not send it out; I would suggest the authors add multi-seed runs, error bars, significance tests, fix the dataset description, and release code. If they do, the question might be worth a short workshop paper.\n\nRecommendation: desk reject with encouragement to resubmit after adding uncertainty quantification.","headline":"A routine pooling comparison whose architecture-dependent conclusion rests on single-run F1 differences that are likely within noise.","tokens_in":6768,"tokens_out":1988,"would_cite":false,"duration_ms":19882,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the best pooling mechanism for sentence-level sentiment analysis is model-specific: on IMDB, mean pooling gives BERT its top F1 score of 87.22%, while weighted sum pooling gives GPT-2 its top F1 score of 88.38%.","keywords":["pooling mechanisms","sentiment analysis","BERT","GPT-2","mean pooling","max pooling","weighted sum pooling","sentence embeddings"],"falsifier":"Repeat the BERT and GPT-2 experiments with multiple random seeds per pooling method and report the distribution of F1 scores. If the confidence intervals for Mean and Weighted Sum overlap for either model, the central claim that each architecture has a distinct best pooling mechanism is not supported by the data.","tokens_in":5836,"feed_emoji":"📊","tokens_out":5929,"duration_ms":51856,"temperature":0.7,"pith_summary":"The paper tries to establish that no single pooling mechanism is universally best for sentence-level sentiment analysis: the best way to compress token embeddings into one sentence vector depends on the model architecture. Using the IMDB movie-review benchmark, it compares Mean, Max, and Weighted Sum pooling on BERT and GPT-2, reporting that BERT peaks at 87.22% F1 with Mean pooling while GPT-2 peaks at 88.38% F1 with Weighted Sum pooling. The paper's practical message is that pooling selection should be tuned per architecture and application rather than assumed by default.","feed_headline":"Best pooling method depends on the LLM, study finds","feed_subtitle":"On IMDB sentiment, BERT peaks at 87.22% F1 with mean pooling while GPT-2 peaks at 88.38% with weighted sum.","key_machinery":"Three pooling operations applied to the token-level embeddings produced by the attention mechanism: Mean pooling averages all token vectors, Max pooling takes the elementwise maximum across tokens, and Weighted Sum pooling uses learnable per-token weights. The paper observes that mean pooling is weighted sum with uniform weights and max pooling is the limit where weight concentrates on one token, so the three methods form a single family of aggregation functions. These functions map the sequence of token embeddings to the sentence-level representation that feeds the sentiment classifier.","core_discovery":"The central claim is that pooling mechanism effectiveness is model-specific, not universal. On binary sentiment classification of IMDB reviews, BERT's best F1 score is 87.22% with Mean pooling, and GPT-2's best F1 score is 88.38% with Weighted Sum pooling. The authors interpret the results to mean that each pooling strategy carries distinct trade-offs: mean pooling gives a balanced, computationally cheap representation; max pooling selects the most salient features and yields the highest true-positive rate, though with a lower true-negative rate; and weighted sum pooling adds learnable token weights whose value depends on the model and task. Therefore the choice of pooling layer should be aligned with the model architecture and the application's requirements.","pith_inferences":["Since mean and max are special cases of weighted sum pooling, a fully converged weighted-sum layer should in principle match or exceed both; the fact that it fails to do so on BERT suggests optimization or overparameterization effects that the paper does not investigate.","The reported ordering comes from single-run scores, so an immediate testable extension is to repeat each configuration over multiple seeds and check whether the mean-versus-weighted-sum gap for BERT and GPT-2 persists.","Because the evidence is confined to one dataset and one task, the natural next experiment is to apply the same three pooling layers on other sentence-level benchmarks such as entailment, paraphrase, and question answering to see whether the architecture-pooling interaction generalizes."],"forward_implications":["Mean pooling is the recommended default when computational cost matters, since it is simple and gives BERT its best measured performance.","Weighted Sum pooling is preferable when the model can exploit learned token emphasis, as it does for GPT-2 in this setup.","Max pooling is the choice when maximizing detection of the positive class is the priority, because it produced the highest true-positive rate at the cost of more false positives.","Pooling recommendations should not be transferred between architectures; the same pooling method performs differently on BERT than on GPT-2."],"supporting_citations":[{"why":"Supplies the bidirectional transformer model whose pooling behavior is evaluated.","marker":"[1]"},{"why":"Supplies the autoregressive GPT formulation that defines the GPT family.","marker":"[2]"},{"why":"Defines the transformer self-attention architecture that produces the token embeddings to be pooled.","marker":"[22]"},{"why":"Provides GPT-2, the concrete model used in the experiments.","marker":"[26]"},{"why":"Supports the learnable weighting mechanism behind weighted sum pooling.","marker":"[30]"},{"why":"Additional source for the weighting and aggregation design.","marker":"[31]"},{"why":"Supplies a prior use of mean pooling for sentence classification that this paper extends.","marker":"[32]"},{"why":"Supplies a prior use of max pooling in supervised sentence representation learning.","marker":"[33]"}],"fun_headline_variants":["Pooling choice is model-specific, sentiment study shows","No universal pooling winner for LLM sentiment analysis","BERT and GPT diverge on best pooling for sentiment","Sentiment tasks favor different pooling in BERT vs GPT","Best pooling depends on LLM architecture, not task alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that single-run F1 and accuracy numbers from one 60/10/30 split of IMDB reliably rank the pooling methods; with no repeated seeds or significance testing, the reported gaps could be ordinary training noise.","fun_headline_variants_meta":{"raw":{"variants":["Pooling choice is model-specific, sentiment study shows","No universal pooling winner for LLM sentiment analysis","BERT and GPT diverge on best pooling for sentiment","Sentiment tasks favor different pooling in BERT vs GPT","Best pooling depends on LLM architecture, not task alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3651,"prompt_tokens":875,"completion_tokens":2776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2699}},"tokens_in":491,"tokens_out":2776,"duration_ms":20519,"temperature":1.0,"reasoning_tokens":2699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:02:29.469650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the BERT and GPT-2 experiments with multiple random seeds per pooling method and report the distribution of F1 scores. If the confidence intervals for Mean and Weighted Sum overlap for either model, the central claim that each architecture has a distinct best pooling mechanism is not supported by the data.","supporting_citations":[{"cited_title":"Improving language understanding by generative pre-training","cited_arxiv_id":null,"evidence_quote":"Supplies the autoregressive GPT formulation that defines the GPT family."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Provides GPT-2, the concrete model used in the experiments."},{"cited_title":"Detecting formal thought disorder by deep contextualized word representations","cited_arxiv_id":null,"evidence_quote":"Supplies a prior use of mean pooling for sentence classification that this paper extends."}],"review_version":1}