{"id":"765e70d7-e882-48db-b98d-657db590574b","arxiv_id":"1909.00080","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A hybrid network combining per-word convolution, two LSTM paths, and attention matches or beats recurrent-convolutional baselines on seven text tasks with fewer parameters.","lead":"This paper proposes SCARN, a neural network that combines word-level convolution with two LSTM branches and attention, and tests it on seven text classification datasets. The authors report accuracy gains over recurrent-convolutional baselines with fewer parameters, but without error bars or released code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'outperforms' claim rests on unverified single-run author implementations; multiple baseline scores are at or near chance, so the comparison is not checkable as written.","rationale":"Read in good faith: the paper is an architectural proposal plus diagnostic experiments, and the architecture is coherent: per-word convolution for task-specific embeddings, LSTM for sequence, second LSTM with attention, concat before softmax. The diagnostic study (ordering invariance, nth-max pooling) is interesting but itself single-run. The load-bearing point is the empirical comparison used to justify the headline. The condition that must hold is that the author-implemented baselines and the SCARN scores are comparably trained and measured. That condition is least secure because the paper provides no code, no run-to-run variability, and no parameter table for the deep CNN/LSTM baselines. The implausibly low baselines (28.999% Deep CNN on TREC; 50% Char CNN on RT) strengthen the concern that the comparison is not merely noisy but systematically weak. This is the same weakness the reader flagged; the verdict should remain CONDITIONAL until reproducible results are supplied. I do not see an internal contradiction in the model description beyond minor notation issues, so the concern is about experimental verification, not the soundness of the construction.","tokens_in":9177,"tokens_out":8809,"duration_ms":76614,"concrete_test":"Ask the authors to release code and training configurations for SCARN and all baselines. Run each model on all seven datasets with at least 10 random seeds, identical optimizer/epoch settings, and one validation check: the C-LSTM, RCNN, Deep CNN, and Char CNN implementations should first reproduce published or near-published accuracy on a canonical dataset (e.g., C-LSTM on TREC/IMDB, Deep CNN/Char CNN on their original benchmarks). If, after this validation, SCARN's mean accuracy still exceeds every baseline on the majority of datasets with non-overlapping confidence intervals, the claim stands; otherwise the comparison is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SCARN outperforms RCNN/C-LSTM with far fewer parameters and beats equally sized deep CNN/LSTM models—is supported only by Table 1's single accuracy numbers. Section 5.2's footnote states 'For lack of code, results are from our implementations,' so every baseline is an author-built approximation whose quality is unverified. This matters because several entries are implausibly low: Deep CNN scores 28.999% on TREC (a 6-way task) and Char CNN scores 50% on RT (chance for a binary task), while Table 2 lists parameter counts for only RCNN and C-LSTM, so the 'equally large' deep CNN/LSTM comparisons cannot be checked either. With no repeated runs, standard deviations, or significance tests, the sub-2-point gaps separating SCARN from RCNN/C-LSTM on Pol, TREC, AR, and IMDB could easily reverse under a correctly configured baseline or a different seed. The architecture itself is clearly described and plausible, but the empirical superiority claim is not independently testable from the paper as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an experimental critique of two CNN building blocks for text classification, fixed-window convolution and max-pooling, and uses that critique to motivate a new architecture, SCARN. SCARN first applies single-word convolution to word embeddings, feeds the transformed representations into an LSTM, and combines this branch with a second LSTM branch that uses attention over the original embeddings. The model is evaluated on seven datasets (IMDB, TREC, SO, RT, Pol, AR, SST-2) against linear models, CNNs, LSTMs, attention models, RCNN, C-LSTM, and a concat-SCARN variant. The paper claims that SCARN outperforms recurrent convolutional models with fewer parameters and matches or beats equally sized deep CNN and LSTM models.","tokens_in":9428,"tokens_out":8177,"duration_ms":68182,"significance":"The proposed architecture is simple and parameter-efficient, and the accompanying qualitative experiments (t-SNE projections, word-ordering permutations, and nth-max pooling) are thought-provoking and could inform future architecture design. If the empirical claims are substantiated, the paper would offer a useful text-classification model and a constructive analysis of convolution and pooling in NLP. However, the central quantitative claim is currently supported only by a single table of unreplicated, author-implemented baselines, with no variance or significance evidence, and several baseline scores are implausible. The paper is better treated as a proposal with a motivating study than as a benchmark result until the evaluation is strengthened.","major_comments":[{"comment":"The abstract and Section 6 claim that SCARN 'outperforms' RCNN/C-LSTM and equally sized deep CNN/LSTM models, but Table 1 is the only quantitative support and it contains one accuracy value per model/dataset. The footnote in Section 5.2 says that 'for lack of code, results are from our implementations,' yet no repeated runs, standard deviations, or significance tests are provided. Several baseline numbers are implausible for the published architectures being represented: Char CNN reaches exactly 50.000 on RT (chance for a binary task), 50.249 on Pol, and Deep CNN reaches 28.999 on TREC. Because the margins on the decisive comparisons are small (TREC: SCARN 90.799 vs C-LSTM 90.400; Pol: SCARN 78.750 vs RCNN 78.250; IMDB: SCARN 89.788 vs Bi-LSTM 88.690), the headline claim is not established as written. Please report repeated runs with variance, use significance tests or confidence intervals, and provide code or published baseline numbers so the comparisons are checkable.","section":"Table 1, Section 5.2"},{"comment":"The parameter-efficiency and 'equally large' claims cannot be checked because Table 2 reports parameter counts only for Small SCARN, Large SCARN, RCNN, and C-LSTM. The abstract's comparison with 'equally large various deep CNN and LSTM architectures' is therefore unverifiable: Deep CNN, Char CNN, LSTM, Bi-LSTM, LSTM+Attention, and concat-SCARN are omitted. Please provide parameter counts for every model and describe how the sizes were matched for the models said to be 'approximately matching.'","section":"Table 2, Section 5.2"},{"comment":"The design choices of SCARN are motivated by the ordering and nth-max pooling experiments, but these experiments also appear to be single runs per condition, with no error bars, repeats, or statistical tests. In particular, the conclusion in Section 3.2 that there is 'no apparent correlation' between max-pooling rank and task importance is a negative claim that cannot be supported by a single curve (e.g., Figure 3 on TREC and SST-2). Please add repeated runs or quantitative correlation measures; otherwise the motivation for replacing max-pooling with attention remains suggestive rather than established.","section":"Section 3.1-3.2, Figures 2-3, Appendix B"}],"minor_comments":[{"comment":"The dimension of C is written as R^{N×d×K}, but Equation (1) and the surrounding text define C_i as a vector of K filter outputs for word i, which implies C∈R^{N×K}. Please correct this notation, as it propagates to the description of the LSTM input shapes.","section":"Section 4.2, Eq. (2)"},{"comment":"Please specify the number of random seeds, whether the dev split in Table 3 was used for early stopping or model selection, and whether the scores in Table 1 are on the test split; this would make the single-run results more interpretable.","section":"Section 5.3"},{"comment":"The distribution-mismatch explanation for SCARN versus concat-SCARN is based on a visual comparison of the mean and standard deviation of one layer on the training set; please describe how these statistics were computed and provide a quantitative discrepancy measure.","section":"Section 6, Figure 7"},{"comment":"The t-SNE analysis would be easier to reproduce if the color assignment criteria for 'semantically close to negative' words and the t-SNE hyperparameters (perplexity, number of iterations) were stated.","section":"Section 3.1, Figure 1"},{"comment":"The max-pooling misclassification examples are illustrative, but the claim that certain words have 'near majority share' should be quantified; please report the actual pooled values or proportions.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe short version: the architecture is clearly described and the diagnostic study is worth reading, but the central 'outperforms' claim is not backed by the evidence as reported. The paper deserves a serious referee, but the referee should ask for code and reruns.\n\nWhat's actually new: SCARN is a specific combination—one-word-wide convolution features fed into an LSTM, parallel with an LSTM-plus-attention branch, outputs concatenated. Each block is established, but this particular configuration is not in the cited RCNN/C-LSTM work. The parameter-efficiency story is plausible: 68K–167K parameters versus 180K for RCNN and 676K for C-LSTM. The diagnostic experiments on word-order sensitivity and nth-max-pooling are well framed, and the t-SNE analysis nicely illustrates that a one-word-wide convolution can act as a task-specific embedding transformation. The math is simple enough that no derivation is needed; the model is a standard composition of known blocks. The citation pattern is unobjectionable—the self-citations are peripheral references, not load-bearing. The architecture description is complete enough to reimplement.\n\nWhere it gets shaky: Table 1 is the entire empirical case, with one run per model per dataset and no variance. The footnote in Section 5.2 says all baselines are from the authors' implementations. That alone is not disqualifying, but several baseline numbers are implausibly low—Deep CNN at 28.999% on TREC (a six-way task) and Char CNN at 50% on RT (chance for a binary task) suggest misconfiguration. Parameter counts are only given for RCNN and C-LSTM, not for the 'equally large' deep CNN/LSTM baselines, so that comparison is uncheckable. The 0.4–2 point gaps separating SCARN from RCNN/C-LSTM on several datasets could easily reverse under a properly configured baseline or a different seed. The max-pooling critique relies on misclassified examples and a few datasets; it is suggestive, not established.\n\nBottom line: this is a plausible architecture with an honest, readable write-up and a diagnostic study that has more substance than most. But the empirical superiority claim is fragile as reported. I would send it to peer review with a request for code, repeated runs, and sanity-checked baselines. The reader's conditional verdict is right.\n\nBest.","headline":"The architecture is plausible and the diagnostic study is interesting, but the 'outperforms' claim rests on single-run author-implemented baselines with several near-chance numbers, so the paper deserves review with a demand for code and reruns.","tokens_in":9915,"tokens_out":3710,"would_cite":false,"duration_ms":30931,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Convolution applied word-by-word, followed by LSTM and attention, yields a text classifier the authors report beats RCNN, C-LSTM, and equally sized deep CNN/LSTM baselines on seven datasets.","keywords":["text classification","SCARN","convolutional neural networks","LSTM","attention mechanism","max pooling","word order","sentiment analysis"],"falsifier":"Reimplement SCARN, RCNN, C-LSTM, and the parameter-matched deep CNN and LSTM baselines on SST-2 with the paper's hyperparameters, run each with at least five random seeds, and report the mean and standard deviation of accuracy; if SCARN's advantages over RCNN and C-LSTM fall within one standard deviation or reverse, the paper's central comparison is not supported. A second check is to repeat the nth-max-pooling experiment under matched hyperparameter tuning; if accuracy rises monotonically or tracks task importance for some n, the critique of max pooling weakens.","tokens_in":9001,"feed_emoji":"🧠","tokens_out":12457,"duration_ms":92224,"temperature":0.7,"pith_summary":"The paper argues that fixed-window convolution and max pooling, the two standard CNN building blocks, are mismatched to text, and it builds a classifier around that critique. The proposed network, SCARN (Sequential Convolutional Attentive Recurrent Network), applies convolution to individual words to turn embeddings into task-specific representations, then feeds that transformed sequence to an LSTM; a parallel branch runs attention over an LSTM's hidden states on the original embeddings. On seven benchmarks covering sentiment analysis and question classification, SCARN reports higher accuracy than the recurrent-convolutional baselines RCNN and C-LSTM while using about 68K to 167K parameters instead of their 181K to 677K. The paper also reports that SCARN beats equally large deep CNN, LSTM, and attention baselines on those datasets. If the comparisons are stable, the practical upshot is that a small, carefully structured recurrent-convolutional model can outperform larger standard alternatives for text classification.","feed_headline":"A 68K-parameter text classifier beats RCNN and C-LSTM on seven tasks","feed_subtitle":"Outperforms recurrent-convolutional rivals at a fraction of the parameters.","key_machinery":"The load-bearing mechanism is SCARN's split into two subnetworks. In the Convolution Recurrent subnetwork, $K$ convolution filters of size $1 \\times d$ act on each word embedding separately, producing a new feature vector per word, and the sequence of these feature vectors is learned by an LSTM. In the Recurrent Attentive subnetwork, the original embeddings are processed by a second LSTM and attention combines the hidden states into an alternative context vector. The final activations of both branches are concatenated for classification. This split is what carries the argument: the single-word convolution cannot destroy word order and is presented as a learned task-specific embedding transformation, illustrated by t-SNE plots of sentiment clustering, while attention substitutes for max pooling by weighting words by task relevance rather than by activation magnitude. The shuffle and nth-max-pooling experiments are the supporting diagnostics that motivate these choices.","core_discovery":"SCARN's central claim is that the weakness of CNNs on text is not convolution itself but how it is applied and pooled. The authors show with random- and alternate-shuffle experiments that accuracy on shuffled sentences stays close to accuracy on correctly ordered sentences and that the gap narrows as window size grows, which they read as evidence that fixed-window convolution does not fully preserve word order. They further show, by selecting the nth-largest pooled value for various n, that accuracy varies arbitrarily with n, which they read as evidence that max pooling does not reliably select task-relevant words. The proposed architecture therefore uses convolution with window size one only as a per-word embedding transformation, preserves order through an LSTM, and replaces max pooling with attention over the recurrent states of a second branch. The paper reports that this design outperforms RCNN and C-LSTM with fewer parameters and outperforms equally sized deep CNN and LSTM architectures on all seven datasets.","pith_inferences":["Editorial inference: the shuffle diagnostic could be used as a routine sanity metric for any new text encoder; a model whose accuracy barely drops under random word order is not learning order-sensitive representations, regardless of its architecture.","Editorial inference: if the t-SNE interpretation is correct, the single-word convolution layer should be replaceable by other per-word transformations, such as a learned MLP or a task-tuned embedding adapter; comparing SCARN with such variants would isolate whether the benefit comes from the convolution operator itself or from per-word task adaptation.","Editorial inference: the same two-branch structure could be tested on token-level tasks beyond classification, such as aspect extraction or named entity recognition, where per-word task-specific representations and word order both matter; the paper does not report those settings."],"forward_implications":["Fixed-window convolution can be replaced by single-word convolution plus a recurrent layer without giving up accuracy, and with fewer parameters.","Max pooling should not be the default pooling choice for text; attention over recurrent states is the alternative SCARN uses.","Recurrent-convolutional text classifiers can be competitive at 68K to 167K parameters, roughly a quarter to a tenth of C-LSTM's 677K, which changes the practical compute budget for such models.","The two diagnostic experiments, shuffled order and nth-max pooling, give future work cheap checks for whether a convolutional text encoder is actually using word order and task-relevant features."],"supporting_citations":[{"why":"Supplies the RCNN recurrent-convolutional baseline, whose max pooling over LSTM context vectors SCARN is compared against.","marker":"(Lai et al., 2015)"},{"why":"Supplies C-LSTM, the fixed-window convolution-before-LSTM recurrent-convolutional baseline that the shuffle experiments challenge.","marker":"(Zhou et al., 2015)"},{"why":"Defines the LSTM used in both SCARN subnetworks and in the LSTM baselines.","marker":"(Hochreiter and Schmidhuber, 1997)"},{"why":"Provides the text-CNN architecture whose fixed-window convolution and max pooling are the objects of the paper's diagnostic experiments.","marker":"(Kim, 2014)"},{"why":"Introduces the attention mechanism that the recurrent attentive subnetwork applies to LSTM hidden states.","marker":"(Bahdanau et al., 2014)"},{"why":"Demonstrates word-level attention for text classification, the attention formulation SCARN adapts.","marker":"(Yang et al., 2016)"},{"why":"Supplies the very deep CNN architecture used as the parameter-matched deep CNN baseline.","marker":"(Conneau et al., 2016)"},{"why":"Supplies the Rotten Tomatoes dataset used in the window-size and shuffle experiment and among the seven benchmarks.","marker":"(Pang and Lee, 2005)"},{"why":"Supplies the SST-2 dataset used for the t-SNE convolution analysis and the attention heat-map illustration.","marker":"(Socher et al., 2013)"},{"why":"Supplies the t-SNE projection used to argue that single-word convolution transforms embeddings into task-specific sentiment-aligned clusters.","marker":"(Maaten and Hinton, 2008)"}],"fun_headline_variants":["68K-param classifier outdoes RCNN and C-LSTM on 7 tasks","SCARN: 68K parameters, beats larger CNN and LSTM nets","Text classifier: conv window 1 + attention beats max-pooling","68K-parameter SCARN outperforms RCNN and C-LSTM on 7 datasets","Attention over recurrent states: compact CNN-LSTM beats larger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single accuracy scores in Table 1, produced by the authors' own reimplementations of all baselines, are stable and meaningful differences; if independent runs with different seeds reorder those scores, the claim that SCARN outperforms the other models collapses.","fun_headline_variants_meta":{"raw":{"variants":["68K-param classifier outdoes RCNN and C-LSTM on 7 tasks","SCARN: 68K parameters, beats larger CNN and LSTM nets","Text classifier: conv window 1 + attention beats max-pooling","68K-parameter SCARN outperforms RCNN and C-LSTM on 7 datasets","Attention over recurrent states: compact CNN-LSTM beats larger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3318,"prompt_tokens":905,"completion_tokens":2413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2311}},"tokens_in":521,"tokens_out":2413,"duration_ms":14780,"temperature":1.0,"reasoning_tokens":2311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:01:45.991768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reimplement SCARN, RCNN, C-LSTM, and the parameter-matched deep CNN and LSTM baselines on SST-2 with the paper's hyperparameters, run each with at least five random seeds, and report the mean and standard deviation of accuracy; if SCARN's advantages over RCNN and C-LSTM fall within one standard deviation or reverse, the paper's central comparison is not supported. A second check is to repeat the nth-max-pooling experiment under matched hyperparameter tuning; if accuracy rises monotonically or tracks task importance for some n, the critique of max pooling weakens.","supporting_citations":[],"review_version":1}