{"id":"9ab84ddb-306b-4896-bdc9-6c77b18c3add","arxiv_id":"2411.15623","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-SSC uses prompt-based Gemma-2b with a new contrastive loss to perform multi-label sequential sentence classification, and introduces the BIORC800 dataset.","lead":"This paper presents LLM-SSC, a framework that uses a small large language model (Gemma-2b) to classify the rhetorical role of each sentence in scientific abstracts, supporting both single-label and multi-label predictions. It also introduces BIORC800, a manually annotated multi-label biomedical abstract dataset, and reports modest F1 improvements over existing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BIORC800 SOTA claim is not robust: LLM-SSC uses validation-tuned per-label thresholds while baselines use a fixed 0.4 threshold, and the winning margins are tiny with no error bars.","rationale":"I read the paper in good faith: the framework is clearly described, the code and dataset are promised for release, the WeighCon loss is a plausible contribution, and the experiments cover ICL and PEFT settings. However, the central claim as stated in the abstract and Section 3.4.2 is comparative: LLM-SSC achieves the highest micro and macro F1 on BIORC800. That claim depends on the comparison being fair and the margin being meaningful. The evaluation protocol gives LLM-SSC dynamic per-label thresholds tuned on the validation set while baselines receive a single fixed threshold of 0.4; this is a concrete, easily checked asymmetry that could change the ranking. The margins are also very small, especially on micro F1 (0.907 vs 0.905), and no uncertainty estimates are provided. A single split, single-seed evaluation on 160 test abstracts cannot support a SOTA claim at this granularity. The reader's weakest assumption about annotation quality is legitimate, but it is not the most load-bearing issue for the comparative claim: label noise affects all models trained and evaluated on the same BIORC800 labels, whereas the threshold and variance issues directly affect whether the reported ranking is real. I therefore keep the reader's CONDITIONAL verdict, subject to re-evaluation with fair thresholding and uncertainty quantification.","tokens_in":18125,"tokens_out":5762,"duration_ms":54332,"concrete_test":"Re-run HSLN, SSC, SciBERT-HSLN, and LLM-SSC on BIORC800 using the identical dynamic per-label threshold selection on the development set described in Section 3.3, and report micro/macro F1 with 95% bootstrap confidence intervals or across at least 5 random seeds. If any baseline's micro F1 reaches or exceeds 0.907, the claimed highest-micro-F1 result is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4.2 reports that LLM-SSC with WeighCon achieves the highest micro/macro F1 scores (0.907 and 0.912) on BIORC800. This comparative claim requires that the evaluation protocol is fair to all systems and that the margins are larger than noise. Two conditions are not met. First, the baselines in Section 3.2 are adapted to multi-label prediction with a fixed threshold of 0.4, 'chosen empirically to balance precision and recall for each label,' whereas Section 3.3 states that LLM-SSC is evaluated with dynamic thresholding, where the optimal threshold for each label is selected on the validation set by maximizing label-specific F1. This asymmetry systematically favors LLM-SSC; the same per-label threshold tuning could plausibly improve baseline micro/macro F1. Second, the test set contains only 160 abstracts, and the reported margin is 0.907 vs 0.905 micro F1 and 0.912 vs 0.897 macro F1. No confidence intervals, significance tests, or multiple random seeds are reported. On a test set of roughly 1,600 sentences, a shift of a few sentences can change the ranking. The annotation-quality concern raised by the reader is valid, but it would affect all systems trained and tested on the same labels roughly equally; the threshold asymmetry and lack of variance estimates bear more directly on the specific claim of being 'highest.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-SSC, a Gemma-2b-based framework for sequential sentence classification (SSC) that supports both single-label and multi-label settings via in-context learning (ICL) and parameter-efficient LoRA fine-tuning. The method uses demonstration-and-query prompts, a 'space thinking' mechanism that generates extra tokens before the label, and an auto-weighting multi-label contrastive loss (WeighCon). The authors also introduce and release BIORC800, a manually annotated multi-label SSC dataset of biomedical RCT abstracts, and evaluate LLM-SSC on BIORC800, CS-ABSTRACT, PubMed 20K RCT, and ART-CORE SC. The central empirical claim is that LLM-SSC with WeighCon achieves the highest micro and macro F1 scores (0.907 and 0.912) on BIORC800, and is near state-of-the-art on single-label datasets.","tokens_in":18438,"tokens_out":3555,"duration_ms":32316,"significance":"The paper addresses a real gap by extending SSC to multi-label annotation and to LLM-based generation, and the released BIORC800 dataset plus code could be a useful community resource. The WeighCon loss is a reasonable attempt to handle the multi-label contrastive-learning challenges of sparse positive pairs and class collision, and the comparison across four datasets gives a broad picture. The paper also reports ablation studies and an analysis of the space-thinking mechanism. These are genuine strengths. However, the headline comparative claim rests on a small margin over baselines under an asymmetric thresholding protocol, and the results are single-run point estimates without uncertainty quantification; these issues must be addressed before the SOTA claim is credible.","major_comments":[{"comment":"The comparison between LLM-SSC and the baselines on BIORC800 is asymmetric in threshold selection. In Section 3.2, the baseline models are adapted to multi-label prediction with a fixed threshold of 0.4, 'chosen empirically to balance precision and recall for each label.' In Section 3.3, LLM-SSC is evaluated using dynamic thresholding, where the optimal per-label threshold is selected on the validation set by maximizing label-specific F1. This protocol gives LLM-SSC a tuning advantage that the baselines do not receive. Because the reported margins are small (Table 3: micro F1 0.907 vs. 0.905 for SSC; macro F1 0.912 vs. 0.897 for SciBERT-HSLN), applying the same per-label validation-tuned thresholding to the baselines, or a fixed threshold to LLM-SSC, could change the ranking. The paper should report both versions or otherwise justify why the asymmetric protocol is fair.","section":"Sections 3.2 and 3.3"},{"comment":"All F1 scores in the main results and ablations are single-run point estimates. No confidence intervals, multiple seeds, or significance tests are reported. The BIORC800 test set contains only 160 abstracts (about 1,600 sentences), so a shift of a few sentences can move the micro and macro F1 by the observed margin of 0.002–0.015. The ablation gaps (e.g., w/o WeighCon at 0.896 vs. 0.907, w/o Space Thinking at 0.892 vs. 0.907 in Table 4) may well be within run-to-run noise. The authors should provide repeated runs with different seeds, bootstrap confidence intervals, or paired significance tests for the key comparisons and ablations. Without such evidence, the claim that LLM-SSC with WeighCon is 'the highest' on BIORC800 is not robust.","section":"Table 3 and Table 4"},{"comment":"The reliability of the BIORC800 annotations is not established for the majority of the dataset. Inter-annotator agreement (Cohen's kappa) is reported only for the first two rounds, covering 100 abstracts. The remaining 700 abstracts were annotated by a single annotator each and then reconciled by one senior annotator, with no agreement or adjudication metrics reported for this large portion. Since BIORC800 is both a contribution and the evaluation benchmark, label noise in the 700-abstract set could affect all measured F1 scores and the conclusions drawn from them. The authors should provide a reliability assessment on a sample of the remaining abstracts, or at least an explicit description of the reconciliation procedure and its consistency.","section":"Section 3.1"}],"minor_comments":[{"comment":"There is a typo in 'We finally splited the 800 abstracts'; it should be 'split.'","section":"Section 3.1"},{"comment":"The dataset is named BIORC800 in the abstract and throughout, but Appendix B refers to aggregating annotations into the 'BIORC-1000 dataset.' This naming inconsistency should be fixed.","section":"Abstract and Appendix B"},{"comment":"In Related Works, 'CSABSTRUCT (Cohan et al., 2019)' appears to be a typo for 'CSABSTRACT' or 'CS-ABSTRACT'; please correct the citation name.","section":"Section 4"},{"comment":"The baseline threshold description ('a threshold of 0.4 chosen empirically to balance precision and recall for each label') is ambiguous: it is unclear whether a single global threshold or per-label thresholds were used. Please clarify.","section":"Section 3.2"},{"comment":"In the WeighCon loss formula, the denominator notation 'Σ_k(1 − αik)sim(hi, hk)' does not specify the index set over k or its dependence on the label c; please make the summation indices explicit.","section":"Equation (9)"},{"comment":"The phrase '10-shots' is grammatically awkward; '10-shot settings' or 'ten demonstrations' would read better.","section":"Section 3.4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution with a useful new dataset and an interesting loss function, but the headline SOTA claim is not yet supported because of the threshold asymmetry and the lack of uncertainty estimates. The dataset annotation reliability issue is also real, though it is a standard limitation for manually annotated resources. I would support acceptance after the comparison protocol is made fair (e.g., applying per-label threshold tuning to baselines or fixed thresholds to LLM-SSC), and after variance-aware reporting is added. The present form is not sufficient for the claimed 'highest' result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: read this for the dataset, not for the method's headline numbers. BIORC800 is a genuinely new resource — manually annotated, multi-label, mostly unstructured biomedical abstracts, with a detailed guideline and released code. That alone justifies a look. The paper does a decent job of showing that a 2B Gemma with LoRA can be competitive with specialized hierarchical models on SSC, and the WeighCon loss is a reasonable extension of contrastive learning for multi-label cases. The space-thinking mechanism (generate a couple of extra tokens, then classify) is a neat practical trick.\n\nThe problem is the SOTA claim. LLM-SSC is evaluated with per-label dynamic thresholds tuned on the validation set, while the baselines get a single fixed threshold of 0.4. That is not a like-for-like comparison; the same tuning would almost surely improve the baselines. On top of that, all numbers are single runs, no error bars or significance tests, and the reported margins are tiny — 0.907 vs 0.905 micro, 0.912 vs 0.906 macro. On a 160-abstract test set, that is noise territory. So the specific claim of 'highest' micro/macro F1 should not be taken at face value.\n\nLess severe but worth fixing: agreement is reported only for the first 100 abstracts; the other 700 were single-annotated and reconciled, with no second-annotation agreement. That is a limitation for the dataset's internal quality, even if it affects all systems roughly equally. And the 'first LLM-based framework' line is an overclaim without comparison to any other LLM-based system.\n\nNone of this is fatal. The dataset is the kind of resource the community needs, and the method is sensible. But the paper should be revised so the claims match the evidence: equalize threshold selection across systems, add variance estimates, and soften 'state of the art' to 'competitive.' Then it's a solid contribution.\n\nVerdict: worth a serious referee. If I were the editor, I'd send it to review, mostly to make sure the dataset and evaluation protocol are scrutinized. For a reading group, the threshold issue alone is a good discussion topic.","headline":"The new BIORC800 dataset is the real contribution; the claimed SOTA gains rest on an unfair threshold comparison and are probably within noise.","tokens_in":18963,"tokens_out":2856,"would_cite":true,"duration_ms":24763,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2-billion-parameter model with a new contrastive loss sets the top F1 scores on multi-label rhetorical tagging of biomedical abstracts.","keywords":["sequential sentence classification","multi-label classification","large language models","contrastive learning","prompt-based fine-tuning","rhetorical roles","biomedical abstracts","BIORC800"],"falsifier":"Re-annotate a random sample of the 700 singly annotated abstracts with multiple independent annotators, compute Cohen's kappa on that sample, and re-evaluate LLM-SSC and the baselines on a subset where the new annotations agree with the original labels; if the F1 ranking changes or the kappa is low, the claimed state of the art on BIORC800 is not robust to label noise.","tokens_in":17942,"feed_emoji":"🧬","tokens_out":7733,"duration_ms":62580,"temperature":0.7,"pith_summary":"The paper claims that a 2-billion-parameter language model, prompted to generate rhetorical labels sentence by sentence, can beat specialized hierarchical neural networks at sequential sentence classification, including the multi-label case where one sentence can be, say, both Objective and Methods. To support this, the authors introduce a new auto-weighting contrastive loss (WeighCon) that forms positive pairs from sentences sharing any common label and down-weights negative pairs with similar label vectors, avoiding the class-collision problem. They also release BIORC800, a manually annotated corpus of 800 biomedical abstracts, mostly unstructured, with multi-label rhetorical annotations. On that dataset, the method reports the highest micro F1 (0.907) and macro F1 (0.912) among the compared baselines. If correct, the result means that large generative models with a tailored contrastive objective can replace elaborate sequential architectures for this task.","feed_headline":"LLM with new contrastive loss tops multi-label abstract tagging","feed_subtitle":"On the new BIORC800 corpus, micro F1 reaches 0.907 and macro F1 0.912.","key_machinery":"The load-bearing component is the auto-weighting multi-label contrastive loss, WeighCon, defined by a contrastive term summed over labels with positive pairs formed whenever two sentences share a positive class, and with a learned weighting function $\\alpha_{ij} = \\sigma(\\mathrm{MLP}(y_i, y_j))$ that reduces the contribution of negative pairs whose label vectors are similar. The paper also relies on the prompt construction (demonstration plus query, with the demonstration selected by SimCSE similarity), the 'space-thinking' mechanism that lets the model emit two extra tokens before the answer and maps them through an MLP, and low-rank adaptation (LoRA) for parameter-efficient tuning. Together these convert a generic 2-billion-parameter generative model into a sequential labeler that sees the whole paragraph as context.","core_discovery":"LLM-SSC treats sequential sentence classification as conditional text generation: each sentence's label is generated by a large language model from a prompt that contains the full paragraph as context, a demonstration selected by SimCSE similarity, and a query naming the target sentence. For task-specific tuning, the model is fine-tuned with low-rank adaptation (LoRA), and the 'space-thinking' mechanism generates two extra tokens before the label, whose hidden states are mapped to the label space by a two-layer MLP. To handle multi-label data, the paper proposes WeighCon, a contrastive loss with an auto-weighting MLP that relaxes the positive-pair constraint to 'share at least one positive class' and reweights negative pairs by label-vector similarity. The paper also presents BIORC800, 800 manually annotated biomedical abstracts (700 unstructured), with 5.7% of sentences carrying multiple labels. The main empirical claim is that on BIORC800, LLM-SSC with WeighCon obtains the highest micro and macro F1 scores (0.907 and 0.912, respectively) among the compared systems, with only about 4% of the model parameters trainable; on single-label datasets it is near state of the art but not the best on micro F1.","pith_inferences":["The positive-pair relaxation in WeighCon could plausibly be applied to any multi-label classification task where exact label-vector matches are rare, not just sentence tagging; the paper does not test this beyond SSC.","The 5-shot-over-10-shot in-context learning result suggests demonstration selection and ordering matter more than quantity, but the paper does not vary retrieval methods, leaving this as a testable extension.","Because agreement was measured only on the first 100 abstracts, the labels for the remaining 700 singly annotated abstracts could contain systematic noise; re-annotating a random sample with multiple annotators would test whether the reported F1 gaps are stable."],"forward_implications":["Multi-label rhetorical tagging can be performed by a generative model prompted with the full abstract, without any sequential CRF or BIO-tagging layer.","Approximately 4% of the model's parameters need to be trainable via LoRA to reach the reported scores, lowering storage and memory requirements.","The BIORC800 dataset provides a manual multi-label benchmark on unstructured biomedical abstracts, complementing auto-generated single-label datasets like PubMed 200K RCT.","The ablation study shows that each of the three components—demonstration in the prompt, the WeighCon loss, and the space-thinking mechanism—improves F1, with a larger impact on the smaller CS-Abstract dataset."],"supporting_citations":[{"why":"Supplies the Gemma-2b backbone language model that the prompt-based framework and LoRA tuning are built on.","marker":"(Team et al., 2024)"},{"why":"HSLN baseline; a hierarchical bi-LSTM/CRF model whose multi-label adaptation serves as a comparison on BIORC800.","marker":"(Jin and Szolovits, 2018)"},{"why":"SSC/BERT baseline and the standard transformer approach for sequential sentence classification; also provides the idea of encoding surrounding context.","marker":"(Cohan et al., 2019)"},{"why":"SDT baseline using SciBERT and BIO tagging, which the paper argues is unsuitable for multi-label SSC and omits from BIORC800 comparison.","marker":"(Li et al., 2021)"},{"why":"SciBERT-HSLN baseline, a strong cross-domain hierarchical model compared on all datasets.","marker":"(Brack et al., 2022)"},{"why":"HeroCon, the contrastive-learning baseline for multi-label data that WeighCon is directly compared against.","marker":"(Zheng et al., 2022)"},{"why":"SimCSE embeddings used to select demonstration samples for the in-context learning prompts.","marker":"(Gao et al., 2021)"},{"why":"LoRA, the parameter-efficient tuning method that keeps most of the LLM frozen.","marker":"(Hu et al., 2022)"},{"why":"Provides PubMed 200K RCT, a single-label SSC benchmark, and the source of structured abstracts re-annotated in BIORC800.","marker":"(Dernoncourt and Lee, 2017)"},{"why":"Provides CS-Abstract, a single-label SSC benchmark used to test single-label performance.","marker":"(Gonçalves et al., 2020)"}],"fun_headline_variants":["LLM-SSC: LLM-based multi-label sentence classification with WeighCon","WeighCon loss boosts LLM multi-label abstract tagging to 0.907 micro-F1","New contrastive loss helps LLMs tag multi-label biomedical abstracts","LLM-SSC tops BIORC800 with auto-weighted contrastive training","LLM with WeighCon reaches 0.907 micro-F1 on multi-label SSC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores treat the manually reconciled labels of BIORC800 as ground truth, but agreement was measured only on the first 100 abstracts, so noise in the remaining 700 single-annotator labels could change the performance ranking.","fun_headline_variants_meta":{"raw":{"variants":["LLM-SSC: LLM-based multi-label sentence classification with WeighCon","WeighCon loss boosts LLM multi-label abstract tagging to 0.907 micro-F1","New contrastive loss helps LLMs tag multi-label biomedical abstracts","LLM-SSC tops BIORC800 with auto-weighted contrastive training","LLM with WeighCon reaches 0.907 micro-F1 on multi-label SSC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1388,"prompt_tokens":1005,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":621,"tokens_out":383,"duration_ms":3651,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:04:51.509653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 700 singly annotated abstracts with multiple independent annotators, compute Cohen's kappa on that sample, and re-evaluate LLM-SSC and the baselines on a subset where the new annotations agree with the original labels; if the F1 ranking changes or the kappa is low, the claimed state of the art on BIORC800 is not robust to label noise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SciBERT-HSLN baseline, a strong cross-domain hierarchical model compared on all datasets."}],"review_version":1}