{"id":"83d0f2ec-b20a-44ac-a137-83d364b4747d","arxiv_id":"2502.06551","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Feeding a small selected subset of sentences, rather than a full paper, improves and accelerates EICAT impact classification with both BERT and Llama models.","lead":"This paper builds a dataset of 436 invasive-species papers labeled with environmental impact categories, then tests ways to feed only the most relevant sentences to language models instead of entire papers. The authors find that selecting a few sentences, or even randomly sampling short passages, can make classification both faster and more accurate than processing full texts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains are not shown to be statistically reliable: macro F1 is computed on a small species-split test set with no confidence intervals, so the evidence selector's +0.098 over full-text input may be sampling noise.","rationale":"The reader's weakest_assumption is data provenance, and a manual audit of fuzzy evidence matching is worth doing. I do not dispute that; I see it as a dataset-quality risk. But the most load-bearing condition for the central claim is that the reported numeric differences are real. Even if every evidence label were perfect, Table 1 would not establish the claim because uncertainty is unreported on a small, species-clustered test set. The paper's own sensitivity warning about macro F1 strengthens this reading. I found no fatal internal inconsistency in the experimental pipeline: species-level splitting is appropriate, selector training does not use test labels, and random-selection results, while surprising, are internally coherent. The missing significance analysis is addressable with the released code and seeds, so the conditional verdict remains appropriate; my concern adds a specific validation requirement rather than overturning the paper.","tokens_in":11200,"tokens_out":9452,"duration_ms":91106,"concrete_test":"Run a cluster-bootstrap significance analysis on the existing data: rerun every Table 1 configuration with 20 fixed seeds (0–19), compute macro and micro F1 per seed on the same test papers, and build 95% confidence intervals by resampling test species (not individual papers) with 10,000 replicates. Also compute the paired differences Evidence minus Complete and Random minus Complete; if either interval contains zero, the corresponding headline improvement fails to reach statistical significance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 1, but every reported score is a point estimate without variance. Section 3 fixes the test split at 10% of 120 species (about 12 species; with multiple papers per species this yields only tens of test documents), and Section 4.2 itself warns that macro F1 can be strongly influenced by misclassification of a few samples. Under these conditions, the differences that carry the paper—PubMedBERT Complete 0.425 vs Evidence 0.523, and Random 0.441/0.496 vs Complete 0.425—could be seed or test-sampling variation. The authors average seven runs but do not report per-run scores, standard deviations, or any significance test, so the comparison cannot currently distinguish a real sentence-selection effect from stochasticity. This is load-bearing because the paper's central conclusion is that sentence selection improves classification; if the appropriate confidence intervals overlap zero, that conclusion is not established even with a perfectly clean dataset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EICAT, a new dataset of 436 full-text scientific papers on invasive species, annotated with IUCN EICAT impact categories and human-extracted evidence sentences. The central experiments compare full-text classification using PubMedBERT, ModernBERT, and Llama-3.1 8B against a two-step pipeline that first trains a sentence selector (trained on human evidence, LLM annotations, classifier entropy, or classifier importance scores) and then classifies using only the 15 selected sentences. The authors report that most selection strategies improve macro/micro F1 over full-text input, that repeated random sampling of short inputs at test time further boosts performance, and that this comes with efficiency gains for the LLM. They conclude that input curation plus sample voting is an effective efficiency-accuracy lever for scientific full-text classification.","tokens_in":11377,"tokens_out":4896,"duration_ms":42619,"significance":"If the reported results are statistically reliable, the paper makes a useful practical contribution: it challenges the assumption that long-context models are automatically superior, provides a new dataset for a relevant classification task, and offers a generalizable pipeline that researchers with limited compute can adopt. The study is also comparatively broad, covering five selection strategies, two classifier families, and deterministic versus randomized evaluation. The authors ship the dataset and code, which supports reproducibility. However, the main quantitative claim currently rests on point estimates from a small test set with no confidence intervals or significance tests, and the dataset's evidence labels are produced by an unvalidated fuzzy matching procedure. These two issues are load-bearing for the paper's conclusions and need to be addressed before the results can be fully credited.","major_comments":[{"comment":"No confidence intervals or significance tests are reported for any macro/micro F1 score in Table 1, despite the paper stating that seven runs were performed per model. The test split is 10% of 120 species (about 12 species, likely producing only tens of test documents), and Section 4.2 itself warns that macro F1 can be strongly influenced by misclassification of a few samples. Under these conditions, headline differences such as PubMedBERT Complete Input 0.425 vs. Evidence 0.523 macro F1 could plausibly arise from seed or test-split variation. The authors should report per-run scores, standard deviations, and results of a paired significance test (e.g., approximate randomization over the seven runs or bootstrap over test documents). Without this, the central claim that sentence selection improves classification is not statistically established.","section":"§4.2 / Table 1"},{"comment":"The evidence annotations used to train the evidence-based selector are created by a fuzzy word-overlap matching procedure, with Llama-3 validating borderline matches, but the paper provides no quantitative validation of this procedure. The authors acknowledge discrepancies between the retrieved PDF versions and the papers used by IUCN assessors, and report 2,247 matched evidence sentences versus 2,226 original annotations, yet do not measure the precision of the matches. If many matched sentences are not the ones the assessors actually read, the evidence selector is trained on contaminated labels, which directly threatens the validity of the Evidence-row results in Table 1. The paper should include a human-validated precision estimate on a sample of matches and, ideally, an analysis of how classification performance changes with matching strictness.","section":"§3"},{"comment":"Table 2 reports that the evidence-based selector achieves only 0.541 NDCG against the human evidence ground truth on the test set, a score the paper itself describes as mediocre. Given that this selector nevertheless yields the largest BERT improvement (macro F1 0.523 vs. 0.425 for full input), the paper should explain the apparent disconnect between selector agreement and classification gain. It is possible that the improvement comes not from faithfully recovering human evidence but from a side effect such as filtering out sentences about other species, as the authors themselves hypothesize. A detailed error analysis of the evidence selector's top-15 sentences would help the reader judge whether the result is driven by label quality or by a mechanism that could be achieved with a simpler rule.","section":"§5.2.1 / Table 2"}],"minor_comments":[{"comment":"The phrase \"named entity recognition and and question answering\" contains a duplicated \"and\" that should be removed.","section":"§2.1"},{"comment":"The sentence \"We hypothesize that this could be cause by two key factors\" should read \"caused by two key factors.\"","section":"§4.2"},{"comment":"The phrase \"filtering our non-relevant impacts\" should read \"filtering out non-relevant impacts.\"","section":"§5.2.3"},{"comment":"The text \"even less being classified asData Deficient\" is missing a space between \"as\" and \"Data Deficient.\"","section":"§5.2.3"},{"comment":"The repository URL is written as \"github.com/inas-argumentation/efficient full text classification\" with spaces; it should be a single URL or a properly formatted repository name.","section":"§3"},{"comment":"The seven runs per model should be described more precisely: please state whether different random seeds were used, whether the same data splits were reused across runs, and how the average scores were computed.","section":"§4.1"},{"comment":"The design choices of k=15 selected sentences and the discretization thresholds (bottom 50%, top 20%, remaining 30%) are arbitrary; a short sensitivity analysis for k and for the thresholds would make the pipeline more robust and strengthen the conclusions.","section":"§5.1"},{"comment":"When majority voting is applied to the 10 generated input samples during evaluation, the paper should specify how ties among class labels are broken.","section":"§5.2.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset and experimental pipeline are potentially valuable to the scientific-text-processing community, and the paper asks a timely question about the trade-off between input length and classification performance. However, the manuscript is not yet ready for publication: the central quantitative claims lack error bars and significance tests despite a small, macro-F1-sensitive test set, and the evidence-label construction is not validated. Both issues are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper builds a new dataset, EICAT, that links full-text invasive-species papers to IUCN impact assessments with evidence sentences, and then compares a bunch of ways to cut the input down to 15 sentences before classifying. That's genuinely useful, and the comparison is systematic: human evidence, LLM-generated annotations, entropy, importance, and random selection, on both an encoder and a decoder model. The main finding—that input curation often beats full-text input, and that repeated random sampling with voting can make even a random selector competitive—comes through clearly in Table 1 and is internally consistent.\n\nThe strongest point in its favor is that the evaluation is on a held-out species split, so the headline result is not circular. The paper is also honest about its own failures: the evidence selector's NDCG against human annotations is only 0.541, and the LLM-based evidence and LLM selectors actually hurt Llama. That kind of reporting earns trust.\n\nThe soft spots are real, though. The stress-test note is on the money: every score in Table 1 is a point estimate with no error bars, no standard deviations, no significance tests, despite seven runs. The test split is 10% of 120 species—so on the order of a dozen species and maybe tens of documents—and Section 4.2 itself warns that macro F1 swings on a few samples. Under those conditions, the +0.098 between Evidence and Complete for PubMedBERT, and the random-selector improvements, could be sampling noise. I'm not saying they are; I'm saying the paper currently gives us no way to tell.\n\nThe data provenance is a second concern. Evidence sentences were matched by fuzzy word overlap with Llama-3 adjudication, and the paper notes version discrepancies between the PDFs they retrieved and the versions assessors read. That's a plausible source of noisy evidence labels. It doesn't break the central claim that selection helps classification, because that's evaluated on held-out labels, but it does mean the evidence selector's reported advantage might be partly an artifact of matching noise.\n\nThe abstract also overstates the decoder-side gains: evidence and LLM selectors make Llama worse, only entropy/importance help, so 'improve the performance of both encoder- and decoder-based language models' is too broad. And the GitHub link in the manuscript appears to contain a space and is currently broken.\n\nNone of this is load-bearing in the sense of disproving the central hypothesis. The paper deserves a serious referee. I'd recommend sending it to review, with the expectation of a revision that adds per-run scores and confidence intervals or significance tests, fixes the code link, and tightens the abstract. If the variance is as large as it likely is on this test set, some of the point estimates will shift; the paper needs to show which differences survive. But the dataset and the systematic comparison are a solid contribution, and the randomization finding is worth taking seriously.","headline":"A useful new dataset and a systematic selection comparison whose central claim is plausible but under-evidenced because Table 1 reports no variance.","tokens_in":11908,"tokens_out":2644,"would_cite":true,"duration_ms":23227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a sentence selector to keep only the 15 most relevant sentences per paper reduces input size and improves EICAT impact classification, with the evidence-based selector raising PubMedBERT macro F1 from 0.425 to 0.523.","keywords":["EICAT","invasive species impact","scientific full-text classification","sentence selection","PubMedBERT","ModernBERT","Llama-3.1 8B","randomized sampling"],"falsifier":"Re-annotate a sample of papers using the exact PDF versions the IUCN assessors worked from and require exact or near-exact sentence matches; if the evidence-based selector no longer raises macro F1 above full-text input on that clean subset, the reported gain is an artifact of fuzzy matching rather than a property of sentence selection.","tokens_in":10970,"feed_emoji":"📄","tokens_out":5714,"duration_ms":44813,"temperature":0.7,"pith_summary":"This paper asks whether feeding an entire scientific paper to a classifier is the best way to classify it, and answers no. Using a new dataset of 436 full-text invasion-biology papers labeled with IUCN EICAT impact categories, the authors train sentence selectors that keep the 15 most relevant sentences per paper. The evidence-based selector raises PubMedBERT macro F1 from 0.425 to 0.523, above a modern long-context encoder (ModernBERT, 0.433) and far above a local Llama-3.1 8B prompted on full text (0.272). Repeatedly sampling short inputs and voting across samples further improves most models, making even a random selector competitive. The upshot is that for this kind of scientific full-text classification, input curation plus sample voting can beat simply throwing more context at the model.","feed_headline":"Picking 15 sentences beats full-text input for paper classification","feed_subtitle":"A 15-sentence selector lifted PubMedBERT macro F1 from 0.425 to 0.523, beating long-context ModernBERT.","key_machinery":"The central mechanism is the sentence selector: a PubMedBERT model trained to score each sentence's usefulness, then used to rank sentences and keep the top 15 per paper. Four selector training signals are compared: human evidence annotations, Llama-3.1-generated usefulness labels, entropy of per-sentence classifier predictions, and leave-one-out importance scores. The second mechanism is randomized sampling with majority voting: drawing a new short input per training step (drawn from the top 30 sentences for targeted selectors) and aggregating 10 sampled predictions at evaluation. Together these carry the argument that input curation, not context length, is what drives performance.","core_discovery":"The paper claims that sentence selection is an effective efficiency-accuracy lever: a two-step pipeline where a PubMedBERT-based selector ranks sentences and both the classifier and the LLM see only the top 15 sentences outperforms full-text input, and the best selector is trained on human evidence annotations from the EICAT assessments. Randomization experiments show that when a new random 15-sentence input is drawn each time a text is accessed and predictions are aggregated by majority vote over 10 samples, classification improves for most models and the random selector becomes competitive with the evidence selector. The authors interpret this as evidence that too much information hurts these classifiers, and that sample-level labels carry more usable signal than human evidence annotations or natural-language class descriptions.","pith_inferences":["Editorial inference: giving full-text baselines the species name, which the selectors implicitly use to filter out other-species content, would isolate how much of the gain comes from disambiguation rather than relevance ranking.","Editorial inference: the randomization result implies a selector-free variant: sample several short windows from the full text and majority-vote, which would remove selector training cost entirely.","Editorial inference: because fuzzy matching may misalign evidence sentences, the evidence selector's advantage should be re-tested on a subset of papers with assessor-verified PDFs before treating it as a stable gain."],"forward_implications":["A 70-times-smaller BERT classifier on selected sentences can outperform a local 8B LLM prompted with the entire text, so model scale is not the deciding factor here.","Long-context encoders are not automatically a better choice for scientific full-text classification: ModernBERT's full-text macro F1 of 0.433 trails the evidence-selected PubMedBERT score of 0.523.","Randomized short-input sampling with majority voting improves most models and makes even a random selector competitive, so costly selector training is not always needed.","Selection cuts inference cost: Llama test-set time drops from 116 to 65 seconds with importance-based selection while accuracy improves, breaking the usual accuracy-efficiency tradeoff."],"supporting_citations":[{"why":"Supplies the EICAT classification standard and category definitions used to label the dataset and to prompt the LLM.","marker":"IUCN (2020)"},{"why":"Provides PubMedBERT, the base encoder used both for classification and as the sentence selector.","marker":"Gu et al. (2021)"},{"why":"Provides ModernBERT, the long-context baseline that the selected-input models are compared against.","marker":"Warner et al. (2024)"},{"why":"Provides Llama-3.1 8B, the local LLM evaluated on full-text and selected-sentence inputs.","marker":"Grattafiori et al. (2024)"},{"why":"Motivates PubMedBERT as the encoder by demonstrating its strong prior performance on invasion-biology text.","marker":"Brinner et al. (2022)"},{"why":"Extracts raw text from PDFs, a necessary step in assembling the EICAT full-text dataset.","marker":"2008–2024. Grobid"}],"fun_headline_variants":["Top 15 sentences beat full text in EICAT classification","Random 15-sentence sampling lifts classification performance","Short inputs outperform full text for paper classification","Evidence-based sentence selection boosts EICAT classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evidence sentences matched from the retrieved PDFs are the same sentences the IUCN assessors actually read; because versions and PDF conversion differ, the paper used fuzzy word-overlap matching with Llama-3 validation, so if those alignments are wrong the training signal for the best selector is contaminated.","fun_headline_variants_meta":{"raw":{"variants":["Top 15 sentences beat full text in EICAT classification","Random 15-sentence sampling lifts classification performance","Short inputs outperform full text for paper classification","Evidence-based sentence selection boosts EICAT classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1448,"prompt_tokens":884,"completion_tokens":564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":500,"tokens_out":564,"duration_ms":5362,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:04:26.869519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a sample of papers using the exact PDF versions the IUCN assessors worked from and require exact or near-exact sentence matches; if the evidence-based selector no longer raises macro F1 above full-text input on that clean subset, the reported gain is an artifact of fuzzy matching rather than a property of sentence selection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EICAT classification standard and category definitions used to label the dataset and to prompt the LLM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates PubMedBERT as the encoder by demonstrating its strong prior performance on invasion-biology text."}],"review_version":1}