{"id":"867a9f34-cd27-440f-b5fc-181d5be72a81","arxiv_id":"2505.06607","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new feature-extraction module for NLI combines all BERT layers with element-wise sentence-pair interactions and a DenseNet, reporting average gains of roughly one point on ten benchmarks, though evaluation inconsistencies weaken the claim.","lead":"This paper proposes a small add-on to language models that reads information from all of the model's layers, not just the last one, before deciding if two sentences agree or conflict. The authors report small average accuracy gains on ten language benchmarks, though the paper's experiments contain inconsistencies that make the headline result unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 cannot support the claimed consistent gains: the RoBERTa-base RTE jump is an unexplained outlier and the reported average improvements do not match the table's own entries.","rationale":"The paper proposes an architecture and supports it with an empirical table. My focus is therefore on whether Table 1 can bear the central claim of consistent gains. It cannot, based on internal evidence alone. Averaging the ten per-dataset deltas in Table 1 gives approximately 1.0/0.8/1.6/0.4 for BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large, not the 0.8/0.7/1.5/0.5 stated in Section 5.1. More decisive is the RoBERTa-base RTE cell: an 8.9-point jump over its baseline, while the corresponding RoBERTa-large RTE improvement is only 0.3. Without code, seeds, or error bars, the most plausible explanations are a typo or a different evaluation condition; either destroys the 'consistent gains' claim. The reader's weakest assumption was the same evaluation-protocol consistency issue, and I agree with that assessment. The missing robustness experiments and placeholder citations are additional signs of incompleteness, but the core problem remains that the one table carrying the conclusion is internally inconsistent. A single targeted rerun of the RTE condition plus a recomputation of the averages would settle whether the concern lands. I recommend keeping the REJECT verdict because the load-bearing empirical evidence, not merely the presentation, is unreliable.","tokens_in":9594,"tokens_out":5426,"duration_ms":51965,"concrete_test":"Re-run RoBERTa-base and RoBERTa-base-CIRN on the RTE dev set (and test set if accessible) under the exact same truncation and evaluation script used for Table 1, with at least three random seeds, and recompute the ten-dataset average from the per-dataset deltas. If the RTE gap drops below roughly two points, or the recomputed average differs from the claimed 1.5% by more than rounding, then the central consistency claim is unsupported. Also document the STS-B scoring protocol (Pearson correlation versus binarized accuracy) in the same run.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1's central claim is that CIRN yields average gains of 0.8/0.7/1.5/0.5% over BERT-base/BERT-large/RoBERTa-base/RoBERTa-large across ten datasets. The load-bearing evidence is Table 1, but it fails an internal consistency check. First, the Avg column deltas are 1.0/0.8/1.6/0.4, not the numbers in the text; the only way to approximate the claimed 1.5% for RoBERTa-base is to lean heavily on the 82.5 RTE entry, an 8.9-point jump over the 73.6 baseline that has no counterpart in any other row (neighboring RoBERTa-large improves only 0.3 on RTE). Second, STS-B is reported as an accuracy in Table 1 although Section 4.1 includes it as a GLUE task normally scored by correlation; the paper never states a binarization or regression-to-classification protocol. Third, no code, data splits, seeds, or error bars are given, and the robustness experiments alluded to in Section 4.2 and Section 6 are absent. If the RTE entry is a typo or comes from a different evaluation subset, the average gain over RoBERTa-base drops to roughly 1.0%, and if averages are recomputed from the rows, most of the headline deltas shift by 0.1-0.2 percentage points. The empirical foundation of the 'consistent surpass' claim is therefore not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Cascaded Interactive Reasoning Network (CIRN), which takes token embeddings from all Transformer layers of a pre-trained language model (e.g., BERT or RoBERTa), computes element-wise interaction tensors between premise and hypothesis tokens at each layer, stacks these tensors along a new dimension, and processes the resulting 4D tensor with a DenseNet before classification. The central claim, stated in the Abstract and Section 5.1, is that CIRN yields consistent average accuracy gains over BERT-base (0.8%), BERT-large (0.7%), RoBERTa-base (1.5%), and RoBERTa-large (0.5%) across ten NLI and sentence-pair datasets.","tokens_in":9903,"tokens_out":5736,"duration_ms":53528,"significance":"If the empirical claims were established, the paper would show that a simple post-hoc interaction module, applied to all layer representations of a pre-trained Transformer, can consistently improve sentence-pair classification accuracy. This would be a useful and easily deployable contribution. However, the current evaluation does not support the claims: the reported averages do not match the table, one key result is an unexplained outlier, the STS-B metric is ambiguous, and no reproducibility information is provided. The architectural novelty is also modest, as element-wise interaction and dense feature extraction are known components; the main potential value lies in the claimed consistent gains, which are not reliably demonstrated.","major_comments":[{"comment":"The text claims average accuracy gains of 0.8%, 0.7%, 1.5%, and 0.5% over BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large, respectively, but the Avg column of Table 1 yields differences of 1.0, 0.8, 1.6, and 0.4 percentage points. The paper does not explain this discrepancy, so the headline quantitative claim cannot be reproduced from the table. In addition, no error bars, random seeds, or significance tests are reported, which is a serious problem given that many per-dataset gains are 0.1–0.3 points and some are ties.","section":"Section 5.1, Table 1"},{"comment":"The RoBERTa-base-CIRN RTE score of 82.5 is a 8.9-point jump over the RoBERTa-base baseline of 73.6, whereas the corresponding RoBERTa-large-CIRN gain is only 0.3 points (84.1 vs. 83.8), and no other model row shows a comparable improvement on any dataset. This single outlier largely drives the claimed 1.5% average gain over RoBERTa-base. The paper offers no explanation for this value; if it is a typo or originates from a different evaluation protocol, the central 'consistent surpass' conclusion is not supported.","section":"Table 1, RTE column"},{"comment":"STS-B is a semantic textual similarity task normally evaluated by Pearson or Spearman correlation, but Table 1 reports it as an accuracy number. The paper never specifies how STS-B scores were converted to accuracy (e.g., whether a threshold was used to binarize the continuous similarity labels or whether a regression objective was replaced by classification). Without this information, the STS-B column and the overall averages are not interpretable or comparable with the other datasets.","section":"Section 4.1, Table 1"},{"comment":"The conclusion states that CIRN shows 'significant improvement in robustness tests', and Section 4.2 refers to robustness evaluations with TextFlint, but no robustness experiments are reported anywhere in the manuscript. Moreover, Sections 4.3 and 5.2 contain placeholder citations '(?)' for the Adadelta optimizer and the MultiNLI dataset, respectively, and the paper does not provide code, data splits, or random seeds. These omissions mean the experiments cannot be reproduced or independently verified.","section":"Sections 4.2, 4.3, 5.2, and 6"},{"comment":"The DenseNet adaptation is underspecified. The interaction tensor Istack is 4D of shape n x m x d x L, but the paper only states that 'convolutional or pooling operations are first applied to reduce dimensionality' without specifying the operation, the output tensor shape, or how the reduced representation is fed into the dense blocks. This is a central architectural component, and the lack of detail prevents the work from being replicated or the design choices from being evaluated.","section":"Section 3.4"}],"minor_comments":[{"comment":"The keywords section contains the fragment 'neural language process', which appears to be a truncation of 'natural language processing'.","section":"Abstract / Keywords"},{"comment":"The caption uses numbered steps '1) PLM Encoding' through '4) Prediction Layer', but the text in Section 3 refers to the 'left panel' of the figure; the manuscript only contains this single figure, so the 'left panel' phrasing is confusing.","section":"Figure 1"},{"comment":"The ablation description in the text does not match the table. The text says removing the interaction mechanism drops accuracy from 85.1% to 84.6%, which corresponds to row 2, but row 2 is labeled 'Remove first 11 layer', while row 3 is 'Remove interaction matrix'. The mapping between the prose and the table rows needs to be corrected.","section":"Section 5.2, Table 2"},{"comment":"The choice of Adadelta with a learning rate of 0.5 and a scheduled L2 decay is very unusual for fine-tuning Transformer models and should be justified or cited; this is an important detail for reproducibility.","section":"Section 4.3"},{"comment":"Several works in the reference list are not cited in the text (e.g., references numbered 38–42), while some citations in the text are incomplete or missing (e.g., reference [3] is listed only as 'arXiv preprint.' and Sections 4.2, 4.3, and 5.2 contain placeholder '(?)' citations).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an unfinished draft: it contains placeholder citations, a missing robustness-experiments section, and a key table with internal inconsistencies. The central claim rests on the empirical evaluation, and that evaluation currently cannot be trusted because of the unexplained RTE outlier, the ambiguous STS-B metric, and the absence of reproducibility details. I recommend major revision, but if the authors cannot provide a corrected evaluation protocol, code, and an explanation of the RTE result, rejection may be the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is genuinely new. CIRN takes token-level embeddings from every BERT/RoBERTa layer, computes Hadamard products between the two sentences at each layer, stacks the resulting tensor, and runs a DenseNet over it before classification. That assembly isn't in the literature, and it's a sensible way to force the model to exploit intermediate representations rather than just the final layer. If the reported gains were trustworthy, this would be a useful incremental contribution.\n\nThey aren't trustworthy as reported. The headline numbers in Section 5.1 don't match Table 1: the Avg column shows deltas of 1.0/0.8/1.6/0.4 over BERT-base/BERT-large/RoBERTa-base/RoBERTa-large, while the text claims 0.8/0.7/1.5/0.5. The RoBERTa-base RTE entry is the load-bearing outlier: 82.5 versus a 73.6 baseline is an 8.9-point jump, while every other row gains between 0.3 and 1.9 points. If that one number is a typo or comes from a different test set, the RoBERTa-base advantage shrinks to around 1.0, which is within the noise band for these benchmarks. STS-B is reported as accuracy even though it's a similarity task normally scored by correlation; the paper gives no binarization or conversion protocol.\n\nThe problems extend beyond Table 1. There are no error bars, no seeds, no significance tests. The conclusion promises \"significant improvement in robustness tests,\" but no robustness experiments appear — Section 4.2 merely has a placeholder reference to TextFlint. The ablation study in Table 2 is undermined by the text: it says removing the interaction matrix drops accuracy from 85.1 to 84.6, but the table shows 85.1 to 84.7 for that row; the 84.6 number actually corresponds to \"remove first 11 layers.\" The citation list has placeholder markers and miscitations (e.g., CharBERT attributed to a question-answering paper). These are not optional polish items; the combination means the central claim of \"consistent gains\" is not established by the evidence in the paper.\n\nThe DenseNet-with-interactions idea itself is worth a second look, and the ablation directions are reasonable. If the authors rerun with a coherent protocol, release code and seeds, and fix the RTE and STS-B problems, there might be a workshop paper here. In its current form, I would desk-reject it: the errors are concentrated exactly where the evidence needs to be strongest.","headline":"The CIRN architecture is a plausible new assembly, but the evaluation is internally inconsistent — the RTE outlier, STS-B metric, and missing robustness experiments break the paper's central claim.","tokens_in":10445,"tokens_out":6334,"would_cite":false,"duration_ms":54366,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that comparing every token pair at every transformer layer, via a stacked interaction tensor, improves natural language inference accuracy across ten benchmarks.","keywords":["natural language inference","textual entailment","pre-trained language models","multi-layer representations","interaction tensor","DenseNet feature extraction","sentence-pair classification","GLUE benchmark"],"falsifier":"Run CIRN under the official GLUE evaluation harness, scoring STS-B by Pearson/Spearman correlation and RTE on the standard test set, and re-check whether RoBERTa-base-CIRN actually reaches 82.5 on RTE against a 73.6 baseline. If those numbers do not reproduce, the paper's consistent-improvement claim is falsified.","tokens_in":9347,"feed_emoji":"🧩","tokens_out":8565,"duration_ms":76754,"temperature":0.7,"pith_summary":"The paper argues that natural language inference misses useful signal when a model reads only the final layer of a pre-trained transformer. It proposes CIRN, which computes the element-wise product of token representations between the premise and hypothesis at every layer, stacks those interaction tensors, and runs a DenseNet over the stack before classifying. On ten sentence-pair benchmarks, CIRN claims average accuracy gains of 0.8% over BERT-base, 0.7% over BERT-large, 1.5% over RoBERTa-base, and 0.5% over RoBERTa-large. The appeal is that a relatively simple add-on could strengthen any transformer-based pair classifier without modifying the pre-trained backbone.","feed_headline":"Cross-layer token comparisons lift NLI accuracy on 10 benchmarks","feed_subtitle":"It compares every premise-hypothesis token pair at every layer and reports consistent gains over BERT and RoBERTa.","key_machinery":"The load-bearing object is the stacked interaction tensor: for each layer $l$, every token of the first sentence is multiplied element-wise with every token of the second sentence to form $I^{(l)} \\in \\mathbb{R}^{n \\times m \\times d}$, and the $L$ tensors are concatenated along a new layer axis. DenseNet, borrowed from image processing as a feature extractor with dense blocks computing $z_k = f_k(\\text{Concat}(Z_0, z_1, \\dots, z_{k-1}))$, then compresses this high-order tensor, and a single linear-softmax layer predicts the relation. This machinery replaces terminal-layer alignment with an all-depth comparison, letting the model see fine-grained local differences and global context at the same time.","core_discovery":"CIRN's central claim is that multi-layer interactive features contain information that terminal-layer representations miss. For each Transformer layer $l$, the model forms the interaction tensor $I^{(l)}_{i,j} = h^{(l,i)}_1 \\odot h^{(l,j)}_2$ for every token pair between the two sentences, stacks the $L$ tensors into $I_{\\text{stack}} \\in \\mathbb{R}^{n \\times m \\times d \\times L}$, and compresses them with DenseNet before a softmax classifier. In the paper's Table 1, this moves average accuracy from 85.8 to 86.8 over BERT-base, from 88.0 to 88.8 over BERT-large, from 87.6 to 89.2 over RoBERTa-base, and from 90.3 to 90.7 over RoBERTa-large across the ten datasets, with the largest single swing on RTE for RoBERTa-base (73.6 to 82.5).","pith_inferences":["A direct extension is to apply the same stacked-interaction head to other pair-classification tasks, such as retrieval reranking or question-answer verification, where intermediate-layer comparisons may dominate.","Because the interaction tensor scales as $n \\times m \\times d \\times L$, long-sequence applications will likely need pooling or sparse sampling over the token-pair grid; the paper does not address that cost.","The manuscript leaves unresolved citation placeholders for the robustness-evaluation tool, the MultiNLI ablation dataset, and the optimizer, and never cites the DenseNet paper, so the robustness section and the DenseNet component are not fully traceable from the text alone.","The paper reports STS-B as accuracy, which differs from GLUE's official correlation metrics; a reader should verify the quoted gains under the official STS-B protocol before transferring the numbers."],"forward_implications":["If the reported gains are real, adding CIRN to a frozen pre-trained encoder is a plug-in upgrade worth roughly one point of accuracy on sentence-pair benchmarks.","The same module transfers to paraphrase identification when paraphrase pairs are reframed as entailment, so the method is claimed to be task-general.","The ablation results attribute about 1.2 points of accuracy to the DenseNet feature extractor and smaller but consistent drops to the interaction matrix and to early-layer representations, indicating all three components contribute.","Under input perturbations, the paper reports a significant robustness improvement over BERT, so the multi-layer interaction is also claimed to stabilize predictions."],"supporting_citations":[{"why":"Supplies the BERT backbone whose per-layer representations CIRN extracts and compares.","marker":"[3]"},{"why":"Supplies the RoBERTa backbone used for the strongest baseline pair.","marker":"[4]"},{"why":"Provides the SNLI corpus, one of the two primary NLI training and evaluation sets.","marker":"[8]"},{"why":"Defines the GLUE benchmark from which six of the ten evaluation datasets are taken.","marker":"[24]"},{"why":"Formalizes the attention and transformer architecture that the multi-layer interaction tensors operate on.","marker":"[27]"},{"why":"Provides the ESIM compare-attend baseline that isolates non-pretrained architecture effects.","marker":"[2]"},{"why":"A knowledge-augmented BERT baseline that CIRN claims to outperform without external semantic labels.","marker":"[5]"},{"why":"A knowledge-guided matching baseline that CIRN surpasses, anchoring the comparison to prior knowledge-enhanced methods.","marker":"[6]"}],"fun_headline_variants":["Cascaded token-pair reasoning lifts NLI across 10 benchmarks","Multi-layer token interactions sharpen NLI accuracy","CIRN stacks pairwise token tensors to boost NLI","Interactive token-pair reasoning across layers lifts NLI","DenseNet-compressed token pairs sharpen NLI reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ten benchmark results in Table 1 were computed under one consistent evaluation protocol, with STS-B scored correctly and the RTE numbers coming from the same setup; if anything was measured differently, the average-gain claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Cascaded token-pair reasoning lifts NLI across 10 benchmarks","Multi-layer token interactions sharpen NLI accuracy","CIRN stacks pairwise token tensors to boost NLI","Interactive token-pair reasoning across layers lifts NLI","DenseNet-compressed token pairs sharpen NLI reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2958,"prompt_tokens":989,"completion_tokens":1969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1888}},"tokens_in":605,"tokens_out":1969,"duration_ms":14075,"temperature":1.0,"reasoning_tokens":1888,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:38:09.347786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CIRN under the official GLUE evaluation harness, scoring STS-B by Pearson/Spearman correlation and RTE on the standard test set, and re-check whether RoBERTa-base-CIRN actually reaches 82.5 on RTE against a 73.6 baseline. If those numbers do not reproduce, the paper's consistent-improvement claim is falsified.","supporting_citations":[{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT backbone whose per-layer representations CIRN extracts and compares."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Formalizes the attention and transformer architecture that the multi-layer interaction tensors operate on."},{"cited_title":"Semantics-aware bert for language understanding,","cited_arxiv_id":null,"evidence_quote":"A knowledge-augmented BERT baseline that CIRN claims to outperform without external semantic labels."},{"cited_title":"Using prior knowledge to guide bert’s attention in semantic textual matching tasks,","cited_arxiv_id":null,"evidence_quote":"A knowledge-guided matching baseline that CIRN surpasses, anchoring the comparison to prior knowledge-enhanced methods."}],"review_version":1}