{"id":"4f1cc47a-2831-4653-a6c1-5c05bb8b28bd","arxiv_id":"1908.06263","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hyperparameter sensitivity study of AGCNN for sentence classification, with tuned settings improving accuracy by roughly 0.45 to 0.81 percentage points on the same six datasets used for tuning.","lead":"This paper measures how the accuracy of an attention-gated convolutional network for sentence classification changes when key settings are varied. It offers practical tuning ranges and a new parameter combination that yields small accuracy gains on six benchmark datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported static-model improvements in the abstract (0.47%, 0.45%) do not match Table 6; recomputing from the table yields 0.38% and 0.23%.","rationale":"The reader's verdict was CONDITIONAL, based on hyperparameter selection on the same six datasets used for evaluation. That is a legitimate overfitting/circularity concern. However, the most load-bearing problem is more concrete and internal: the paper's headline improvement numbers for the static variants do not match the data in its own Table 6. The rand averages (0.81%, 0.67%) check out exactly, but the static averages (0.47%, 0.45%) are contradicted by the printed table, where the recomputed means are 0.38% and 0.23%. The discrepancy is far too large to be explained by rounding of the two-decimal values. Since the central claim of the paper is the magnitude of the improvement, this inconsistency undermines the main contribution. The sensitivity analysis itself may still be of value, but the current manuscript's headline result is not credible. The paper would need corrected reporting and independent verification before it could be accepted; as presented, it should be rejected. The reader's selection-bias concern is valid but secondary; my concern is a direct numerical contradiction.","tokens_in":12652,"tokens_out":7354,"duration_ms":60391,"concrete_test":"Recompute the mean per-dataset accuracy difference (Ours - baseline) from the six rows in Table 6 for both static models. If the recomputed averages are 0.38% and 0.23%, the abstract's 0.47% and 0.45% claims are contradicted; request the unrounded per-dataset results and the exact protocol used to compute the averages.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim (abstract and Section IV.G) is that the proposed hyperparameters improve AGCNN by 0.81%/0.67% (rand) and 0.47%/0.45% (static). The rand averages are correctly derived from Table 6. However, for the static models, subtracting the baseline rows from the 'Ours' rows in Table 6 gives mean differences of +0.38% (six datasets: +0.52, +0.23, -0.03, +0.47, +0.04, +1.07) for AGCNN-NLReLU-static and +0.23% (+0.24, +0.21, -0.18, +0.41, -0.24, +0.95) for AGCNN-SELU-static. These are far outside the ±0.02 rounding tolerance. The claimed 0.47% and 0.45% are therefore unsupported by the paper's own data. The same-dataset selection issue noted by the reader is real but secondary; the arithmetic inconsistency is a factual error in the headline result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a sensitivity analysis of Attention-Gated Convolutional Neural Networks (AGCNNs) for sentence classification. Using six standard datasets (CR, MR, Subj, SST-1, SST-2, TREC), the authors vary one hyperparameter at a time while holding others fixed, reporting accuracy changes for the first convolutional layer's kernel window sizes, the attention-gated layer's kernel window sizes, the numbers of feature maps in both layers, the dropout keep rate, and the activation function. Based on these analyses, they select a new hyperparameter configuration (200 feature maps in the first convolutional layer, 10 in the attention-gated layer, kernel windows (1,2,3,4,5) and (1,3,5,7), keep rate 0.5) and claim in the abstract and Section IV.G that it improves average accuracy over their baselines by 0.81% and 0.67% for the rand versions and 0.47% and 0.45% for the static versions of AGCNN-NLReLU and AGCNN-SELU.","tokens_in":12837,"tokens_out":3364,"duration_ms":32052,"significance":"If the reported improvements were validated out of sample, the paper would provide useful practical guidance for tuning AGCNNs and a systematic empirical characterization of hyperparameter sensitivity. The sensitivity curves are based on 10 repeats of 10-fold cross-validation, which is a solid experimental protocol for this type of study, and the paper honestly states that its aim is analysis rather than state-of-the-art results. However, the central improvement claim is weakened by two issues: the final configuration is selected on the same six datasets used for evaluation, and the claimed static-model improvements are not supported by the paper's own Table 6. The sensitivity analysis itself remains valuable, but the headline result as stated is not reliable.","major_comments":[{"comment":"The abstract, the contributions in Section I, and Section IV.G claim average improvements of 0.47% and 0.45% for AGCNN-NLReLU-static and AGCNN-SELU-static, respectively. Recomputing from Table 6 by subtracting the baseline row from the 'Ours' row gives mean differences of +0.38% (per-dataset differences: +0.52, +0.23, -0.03, +0.47, +0.04, +1.07) and +0.23% (+0.24, +0.21, -0.18, +0.41, -0.24, +0.95), respectively. These are far outside the ±0.02 rounding tolerance. The claimed 0.47% and 0.45% are therefore unsupported by the paper's own data and must be corrected.","section":"Abstract and Section IV.G (Table 6)"},{"comment":"The final hyperparameter configuration is selected after inspecting accuracy on the same six datasets (CR, MR, Subj, SST-1, SST-2, TREC) in Sections IV.A through IV.F, and Table 6 then reports accuracy on those same datasets. This is a post hoc selection on the evaluation set, so the reported gains are in-sample fitted values rather than out-of-sample predictions. To support the improvement claim, the authors should validate the chosen configuration on a held-out dataset or use nested cross-validation; alternatively, they should explicitly reframe the claim as an in-sample description.","section":"Section IV.G"},{"comment":"No statistical significance test is reported for the average improvements. Many per-dataset differences are within one standard deviation of the baseline (for example, SELU-static differences on Subj and SST-2 are -0.18 and -0.24, and several positive differences are smaller than the reported standard deviations). The authors should provide paired tests across the 10 repeats or confidence intervals to establish that the average gains are not attributable to noise.","section":"Section IV.G and Table 6"}],"minor_comments":[{"comment":"The text says the attention-gated layer kernel window sizes are set to '(13,5,7)'; this should be '(1,3,5,7)'.","section":"Section IV.G"},{"comment":"The improvement percentages in the abstract and the contributions list should be updated to match the corrected values from Table 6.","section":"Abstract and Section I"},{"comment":"Figure 7 is referenced but its content is not described in the text beyond the sentence about SELU and ReLU; a brief summary of the observed trends would improve readability.","section":"Section IV.F"},{"comment":"There are several spacing/inconsistency issues in the table entries (for example, '85.54±0.33' in the (2,4,6,8) row of Table 4 and '47.45±0.27' in Table 3); a copyedit pass would help.","section":"Tables 2-6"}],"recommendation":"major_revision","confidential_remarks":"The sensitivity analysis appears carefully executed, with 10 repeats of 10-fold cross-validation and a reasonable range of hyperparameter values. The main problems are the arithmetic error in the static-model improvement claims and the post hoc selection on the evaluation datasets. If the authors correct the numbers, provide significance tests or confidence intervals, and temper the generalization claim, the paper could be suitable for publication. I see no grounds for questioning the integrity of the experiments; the static-model discrepancy looks like a calculation error rather than a deliberate misstatement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the sensitivity analysis itself. The authors systematically vary kernel sizes, feature-map counts, dropout, and activation functions for AGCNN across six standard datasets, using 10 repeats of 10-fold cross-validation. The curves and practical ranges (e.g., 100–400 filters in the first conv layer, 10–50 in the attention-gated layer) are a legitimate contribution for anyone stuck tuning this niche architecture. The finding that the first conv layer's window size matters more than the attention-gated layer's is plausible and supported by the figures. Credit where due: the experiments are extensive and the descriptive conclusions follow from the data.\n\nBut the headline improvement claims do not hold up. The abstract and Section IV.G state average gains of 0.47% and 0.45% for AGCNN-NLReLU-static and AGCNN-SELU-static. Recomputing from Table 6 gives 0.38% and 0.23% — not a rounding issue, a real arithmetic discrepancy. The rand-model claims (0.81%, 0.67%) check out, so the error is specific to the static rows, but it is present in the abstract and the summary, which is where readers land.\n\nBeyond the arithmetic, the central improvement is selected on the same six datasets used for evaluation. The authors chose the final hyperparameters after inspecting accuracy on CR, MR, Subj, SST-1, SST-2, and TREC, then reported gains on those same datasets. That makes the improvement claim a fitted value, not a prediction. The advice itself can still be useful because the sensitivity curves describe behavior on these datasets, but the paper should say plainly that the gains are in-sample. The standard deviations also overlap with most of the per-dataset differences, so the improvements are fragile.\n\nMinor issues: no code or data are released, and there is a typo in the settings — \"(13,5,7)\" should be \"(1,3,5,7)\".\n\nWho is this for? Someone using AGCNN and looking for reasonable starting ranges. It will not change how I think about CNN-based sentence classification, but it is a competent empirical study of a real architecture. It deserves a serious referee, not a desk reject, because the sensitivity data are worth publishing and the errors are correctable. The authors need to fix the static-model averages in the abstract, add an explicit statement that the tuned settings were selected on the same datasets, and ideally release code. If those changes are made, the paper is acceptable as a niche practitioner-oriented contribution.","headline":"A useful but flawed tuning guide: the sensitivity curves are real, but the headline static-model improvements are contradicted by the paper's own Table 6.","tokens_in":13380,"tokens_out":2256,"would_cite":false,"duration_ms":21790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuned hyperparameter settings—multi-window kernels, 200 and 10 feature maps, and keep rate 0.5—improve AGCNN sentence-classification accuracy by average gains of 0.81, 0.67, 0.47, and 0.45 percentage points across four model variants.","keywords":["sentence classification","attention-gated convolutional neural network","sensitivity analysis","hyperparameter settings","kernel window size","dropout keep rate","activation function","AGCNN"],"falsifier":"Run the four proposed configurations on held-out sentence-classification datasets that were not among the six used for selection, using the same 10-fold protocol; if the average accuracy is no better than the baselines, the central claim of consistent improvement fails. A second, sharper test would compare the proposed settings against random search over the same hyperparameter ranges; if random search matches or exceeds the gains, the specific combination is not what causes them.","tokens_in":12409,"feed_emoji":"📈","tokens_out":6599,"duration_ms":58845,"temperature":0.7,"pith_summary":"The paper sets out to map which hyperparameters control the accuracy of Attention-Gated Convolutional Neural Networks (AGCNNs) for sentence classification, and to turn that map into better default settings. It varies one hyperparameter at a time across six datasets, then combines the winning choices into a final configuration. The paper's central result is that this configuration beats the original AGCNN settings by small but consistent margins: average gains of 0.81% and 0.67% for the two random-embedding variants and 0.47% and 0.45% for the two static-embedding variants. The practical value is that someone using an AGCNN can start from these settings instead of running an expensive search. The paper is explicit that it is not claiming state-of-the-art results; the contribution is the sensitivity analysis and the improved defaults.","feed_headline":"Tuning hyperparameters lifts AGCNN accuracy by up to 0.81 percent","feed_subtitle":"It maps which hyperparameters matter and trims them for small, consistent accuracy gains.","key_machinery":"The object that carries the argument is the Attention-Gated Convolutional Neural Network (AGCNN): a CNN for sentence classification whose pooling layer is preceded by an attention-gated layer, a second convolution that produces per-feature attention weights from symmetric context windows and multiplies them into the first layer's feature maps. The analytic machinery is a controlled sensitivity study: keep every hyperparameter fixed at a baseline, vary only the component of interest, replicate each configuration 10 times with 10-fold cross-validation, and record average accuracy across six datasets. The final improved setting is then assembled from the best observed choices—multi-window kernels (1,2,3,4,5) in the first convolutional layer, (1,3,5,7) in the attention-gated layer, 200 and 10 feature maps respectively, and a dropout keep rate of 0.5. What this machinery establishes is a ranking of where the model's performance comes from: the attention-gated layer contributes most, followed by initialization method and activation function.","core_discovery":"On the paper's own terms, the discovery is that the AGCNN's accuracy is governed more by a few structural choices than by any single magic setting. The first convolutional layer's kernel window size is the most sensitive hyperparameter: large windows degrade accuracy, and combining several consecutive small windows, best at (1,2,3,4,5), improves it. For the attention-gated layer, odd symmetric window combinations, best at (1,3,5,7), outperform even ones. Accuracy rises with the number of feature maps in the first layer up to about 100 and flattens after 200, while the attention-gated layer works best with relatively few feature maps, in the 10 to 50 range. A dropout keep rate between 0.2 and 0.8 helps, with 0.5 serving as the final choice, and SELU generally outperforms NLReLU, the log-rescaled ReLU variant used in the original AGCNN, although SELU makes the model more sensitive to hyperparameter changes. Combining these observations yields the improved settings and the reported accuracy gains.","pith_inferences":["Because the final settings were selected and then evaluated on the same six datasets, the reported gains are likely optimistic; a held-out evaluation is needed to know how much transfers.","The pattern that multi-window small kernels help suggests AGCNNs benefit from capturing multiple n-gram granularities at once, a property that may also apply to other attention-based CNN text models.","In Table 6, the proposed settings slightly hurt some datasets, such as AGCNN-SELU-static on Subj and SST-2, so the average gain is not a guarantee of per-dataset improvement.","The one-factor-at-a-time method cannot capture interactions between hyperparameters; joint optimization might find better settings than the assembled best-of-each choices."],"forward_implications":["Using the proposed settings as defaults for AGCNN sentence classification should yield small accuracy gains on similar datasets without hyperparameter search.","The sensitivity ranking implies that effort spent on the first convolutional layer's kernel window sizes and on the attention-gated layer will pay off more than tuning other parts.","Odd, symmetric kernel windows for the attention-gated convolution are preferable to even ones, because they center attention weights on each target feature.","Feature-map counts beyond about 200 in the first layer give diminishing returns, so memory can be saved by staying near 200."],"supporting_citations":[{"why":"introduces the standard CNN sentence-classification baseline and the preprocessing steps used throughout the experiments.","marker":"[1]"},{"why":"defines the AGCNN architecture and the original hyperparameter settings that serve as the baseline model.","marker":"[4]"},{"why":"supplies the Subj subjectivity dataset included in the six-dataset evaluation.","marker":"[10]"},{"why":"supplies the CR customer-review dataset.","marker":"[11]"},{"why":"is the earlier sensitivity-analysis study whose one-factor-at-a-time approach this paper follows.","marker":"[19]"},{"why":"defines dropout, the regularization controlled by the keep rate under study.","marker":"[22]"},{"why":"defines the SELU activation function compared against NLReLU.","marker":"[24]"},{"why":"supplies the TREC question-classification dataset.","marker":"[25]"},{"why":"supplies the MR movie-review dataset.","marker":"[26]"},{"why":"supplies the SST-1 and SST-2 sentiment-treebank datasets.","marker":"[27]"}],"fun_headline_variants":["AGCNN Accuracy Up 0.81% With Hyperparameter Tuning","SELU Activation Beats NLReLU in AGCNN Models","Small Kernel Windows Improve Attention-Gated CNNs","Hyperparameters That Matter for AGCNN Accuracy","0.81% Accuracy Gain via AGCNN Hyperparameter Tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that the six datasets used to pick the final hyperparameters are representative enough that the gains measured on those same datasets will recur on other sentence-classification tasks; if that premise fails, the reported improvements are just in-sample tuning artifacts.","fun_headline_variants_meta":{"raw":{"variants":["AGCNN Accuracy Up 0.81% With Hyperparameter Tuning","SELU Activation Beats NLReLU in AGCNN Models","Small Kernel Windows Improve Attention-Gated CNNs","Hyperparameters That Matter for AGCNN Accuracy","0.81% Accuracy Gain via AGCNN Hyperparameter Tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000856,"raw_usage":{"total_tokens":3701,"prompt_tokens":910,"completion_tokens":2791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2708}},"tokens_in":526,"tokens_out":2791,"duration_ms":20544,"temperature":1.0,"reasoning_tokens":2708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:50:55.274775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the four proposed configurations on held-out sentence-classification datasets that were not among the six used for selection, using the same 10-fold protocol; if the average accuracy is no better than the baselines, the central claim of consistent improvement fails. A second, sharper test would compare the proposed settings against random search over the same hyperparameter ranges; if random search matches or exceeds the gains, the specific combination is not what causes them.","supporting_citations":[{"cited_title":"An Attention-Gated Convolutional Neural Network for Sentence Classification","cited_arxiv_id":"1808.07325","evidence_quote":"defines the AGCNN architecture and the original hyperparameter settings that serve as the baseline model."},{"cited_title":"A s enti mental education: S e ntiment analysis using sub jectivity summarization based on minimum cuts","cited_arxiv_id":null,"evidence_quote":"supplies the Subj subjectivity dataset included in the six-dataset evaluation."},{"cited_title":"Mining and summar izing customer revi ews","cited_arxiv_id":null,"evidence_quote":"supplies the CR customer-review dataset."},{"cited_title":"Dropout: a simple way to prevent neur al networks fro m ov erfitting","cited_arxiv_id":null,"evidence_quote":"defines dropout, the regularization controlled by the keep rate under study."},{"cited_title":"Self - normalizing neural networks","cited_arxiv_id":null,"evidence_quote":"defines the SELU activation function compared against NLReLU."},{"cited_title":"Learning question cla ssifiers","cited_arxiv_id":null,"evidence_quote":"supplies the TREC question-classification dataset."},{"cited_title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","cited_arxiv_id":null,"evidence_quote":"supplies the MR movie-review dataset."},{"cited_title":"Recursive deep models for semantic compositionality over a sen timent treebank","cited_arxiv_id":null,"evidence_quote":"supplies the SST-1 and SST-2 sentiment-treebank datasets."}],"review_version":1}