Pith. sign in

REVIEW 3 major objections 5 minor 45 references

SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that feeding a language model the introduction, results, and discussion sections of a paper predicts reviewer-assigned novelty better than feeding the full text, with the best model reaching 0.682 accuracy and a 0.6515…

desk verdict A useful dataset and a plausible section-ranking result, but the headline 'IRD is optimal' rests on sub-noise margins without significance tests. read the letter →

arxiv 2505.16330 v1 pith:5UTXXCDR submitted 2025-05-22 cs.CL cs.AIcs.DL

Wenqing Wu , Chengzhi Zhang , Tong Bao , Yi Zhao This is my paper
classification cs.CLcs.AIcs.DL
keywords noveltyscorepredictionsectionstructurecombinationIMRaDpretrainedlanguagemodelslargepeerreviewscoresSciBERTautomatedassessment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the optimal way to feed an academic paper to a language model for automated novelty scoring is to use a specific combination of sections, not the whole text. Working from ICLR 2022 and 2023 peer review data, the authors segment papers into introduction, methods, results, and discussion, then fine-tune long-document pretrained models and prompt large language models with 16 different section combinations. Their central result is that the introduction, results, and discussion (IRD) combination yields the best novelty score predictions, with the full text performing markedly worse. The claim matters because automated novelty assessment could target the informative sections instead of processing entire papers, and because it suggests that current models struggle to extract novelty signals from methods-heavy or full-text input.

What carries the argument

The section-combination input scheme: each paper's paragraphs are classified into IMRaD categories by a fine-tuned SciBERT classifier whose confident predictions (probability over 0.8) are kept, while uncertain ones are checked against Llama3 and manual correction; the resulting section texts are then combined into 16 inputs (e.g., I, IM, IRD, IMRD) that drive either fine-tuned long-document PLMs or zero-shot LLM prompts. The central comparison is the accuracy, weighted F1, and correlation-coefficient ranking of these combinations, with IRD as the winner.

What would settle it

A direct test would be to rerun the same 16 section combinations on papers whose novelty labels are obtained from a rubric that explicitly separates novelty from significance (e.g., a 'novelty only' score); if IRD no longer ranks first, the paper's conclusion is an artifact of the blended label. A second test would shuffle the text blocks within each paper and check whether the true IRD combination still outperforms shuffled IRD, which would reveal whether the result depends on actual section content or merely on position and length.

Watch

Extended reading notes

Core claim

The central claim is that introduction, results, and discussion is the most appropriate section combination for assessing a paper's novelty, while using the entire text does not yield significant results. The authors find that fine-tuned PLMs consistently perform best when the input is IRD: SciBERT in its Longformer version reaches 0.682 accuracy and 0.6515 weighted F1, and IRD shows the highest correlation coefficients with the ground-truth scores across all PLM setups. They also find that combinations including the introduction perform better than those without it, and that the introduction and results are the most important sections, with discussion serving as a useful supplement.

Load-bearing premise

The ground-truth novelty scores come from ICLR reviewer ratings of 'Technical Novelty and Significance', so the labels blend novelty with significance, are averaged across reviewers, and are filtered when reviewers disagree by more than one point; if those scores do not measure novelty as defined, the section-combination ranking is about a different construct.

Editorial extensions

If this is right

  • Automated novelty scoring systems should prioritize introduction, results, and discussion content over full-text or methods-heavy input.
  • Fine-tuned PLMs outperform zero-shot LLMs on this task, so progress is more likely from adapting scientific-domain long-text models than from prompting general-purpose chatbots.
  • LLMs display a leniency bias on novelty scoring, assigning high novelty scores more often than reviewers do.
  • Section-based text input beats a citation-distance baseline (Shibayama et al.'s method) on accuracy, F1, and correlation, supporting text-based novelty assessment over reference-based measures.
  • The modest ceiling (0.682 accuracy) indicates that three-class novelty prediction from text alone remains an open problem rather than a solved one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If IRD's advantage reflects where authors explicitly claim novelty, then this method partly measures self-reported novelty claims; a testable extension would ablate sentences containing 'we propose' or 'novel' to see if the IRD edge disappears.
  • The full-text underperformance may stem from token truncation or noise in methods sections; a controlled experiment with equal token budgets per combination would separate section informativeness from length effects.
  • Because the ground-truth label blends novelty with significance, a dataset with separate novelty-only ratings could shift the optimal combination, with results and discussion possibly gaining prominence.
  • The approach should transfer to other IMRaD-structured domains, but the optimal combination may differ when novelty labels come from field-specific rubrics, so cross-domain re-benchmarking is needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SC4ANM, a task for predicting the novelty score of academic papers from different combinations of IMRaD sections. The authors collect ICLR 2022 and 2023 review reports and paper PDFs, use a fine-tuned SciBERT classifier together with Llama 3 and manual correction to identify introduction/methods/results/discussion sections, and then fine-tune five long-document PLMs on 16 section combinations. They also test GPT-3.5 and GPT-4o in a zero-shot setting on a small balanced sample, and compare the best PLM configuration with a citation-based novelty baseline. The main empirical claim is that the introduction+results+discussion (IRD) combination is the most appropriate input for automated novelty prediction, that introduction and results are the most important sections, and that using the entire text does not yield significant results. The paper reports accuracy, weighted F1, and correlation coefficients for all combinations and models, and makes code and data publicly available.

Significance. If the central claim were fully supported, the paper would provide practical guidance for selecting paper sections in automated novelty assessment, an area where current methods rely on titles, abstracts, or entity-level features. The study is also valuable for its large-scale use of real reviewer scores as labels, its comparison of five PLMs and two LLMs on a common task, and its release of code and data. The pattern that IRD ranks first for all five PLMs is a meaningful and reproducible observation. However, the headline conclusion currently rests on unquantified point estimates, and the ground-truth label is a composite of novelty and significance rather than novelty alone. These issues need to be addressed before the optimal-section claim can be accepted.

major comments (3)
  1. [§4.4.1, Table 4; Abstract] The central claim that IRD is the optimal section combination is supported only by point estimates. For SciBERT, IRD accuracy is 0.6824 versus 0.6600 for IMR and 0.6592 for IMD; on the 350-example test set, a 0.022 accuracy gap corresponds to roughly 8 examples, which is within binomial variation. No confidence intervals, standard errors, or paired tests (e.g., McNemar's test) are reported, and IRD is selected as the maximum among 16 combinations, so the chance that the winning margin is a random fluctuation is inflated. The same issue affects Table 2: the significance marks (a/b/c) test each correlation against zero, not the difference between combinations, so the statement that IRD has the highest correlation coefficient is not tested against IMR/IMD. The abstract's phrase 'the use of the entire text does not yield significant results' is also unsupported; the IMRD accuracy of 0.6182 is not shown to differ significantly from IRD. Please add paired significance tests with multiple-comparison correction (or an explicit holdout/validation protocol) and report effect sizes with confidence intervals.
  2. [§3.1, Table 1; §2.1] The ground-truth labels are ICLR 'Technical Novelty and Significance' (TNS) scores, whose rubric combines significance with novelty ('3: The contributions are significant and somewhat new...'), but Section 2.1 defines novelty as the intrinsic quality of presenting new knowledge. Calling the re-aggregated 0/1/2 labels 'basic novel', 'moderate novel', and 'highly novel' assumes that the TNS ordering and meaning are preserved under the merge of scores 1 and 2, which is not justified. This construct mismatch means the model may be predicting reviewers' perceived contribution and significance rather than novelty as defined. Please either use a label that separates novelty from significance, explicitly treat TNS as a composite construct, or rename the task and soften the novelty-specific conclusions. In any case, report sensitivity to the reviewer-disagreement exclusion threshold and the label-merge rule.
  3. [§3.5, Table 5; §4.6] The LLM experiment re-samples 40 papers independently for each section combination ('we performed random sampling of the papers again for each different combination'), so the comparisons across combinations are not paired; any observed difference could be due to the different samples rather than the input sections. Combined with n=40 per combination, this makes the LLM-based ranking in Section 4.6 (e.g., RD/IR best for GPT-4o) statistically fragile. To support the cross-model conclusion that the results section is important, use the same held-out papers for every combination or report paired analyses; otherwise, restrict the section-association conclusions to the PLM results.
minor comments (5)
  1. [Table 4; Table 2] Several table entries appear malformed: the LED/IMRD F1 value '0.592' in Table 4 is truncated, and Table 2 contains merged numerals such as '0.16030.1689' and '0.1601 a'. These formatting errors should be corrected.
  2. [§3.3] The 91% accuracy of the section-structure classifier is reported without details on the validation set or per-class performance; because misclassified sections propagate into every input combination, a brief error analysis would help readers assess the noise introduced by this step.
  3. [§4.4.1] The statement that 'input combinations including the introduction tend to perform better' is contradicted by the IMRD row, which includes the introduction but performs worse than IRD, IMR, and IMD; please qualify the claim or explain the IMRD drop.
  4. [References] Cohan et al. 2018a and 2018b appear to be the same paper with identical titles and venues; please deduplicate and ensure the intended sources are cited correctly.
  5. [§3.2] The acronym SC4ANM is defined only in Section 3.2; please define it at first use in the abstract and introduction.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the novelty labels are external ICLR reviewer scores, predictions are evaluated on a held-out test set, and the sole self-citation (Wu et al. 2024) is motivational rather than load-bearing.

full rationale

The paper's claimed result—IRD is the optimal section combination—is an empirical ranking of measured test-set accuracy and F1 across 16 combinations; it is not derived from a fitted parameter defined in terms of the same score, so the derivation chain is not circular. Ground-truth labels (Technical Novelty and Significance scores) come from ICLR 2022/2023 OpenReview reports, an external human-judgment source, and the 350-paper test set is held out from fine-tuning under an 8:1:1 split. The only self-citation is Wu et al. (2024), cited in Section 1 to note prior work on reviewer score consistency; the sentence 'We believe that the novelty scores given by reviewers can serve as a reference standard' rests on the OpenReview data itself, not on the cited paper, so the self-citation is not load-bearing. The absence of significance tests for the IRD-versus-IMR/IMD margins (Table 4: 0.6824 vs 0.6600/0.6592) is a statistical robustness concern, not a circularity, because the claim is a post-hoc selection among observed outcomes rather than a reduction of the conclusion to the inputs. No equation or definition in the paper makes the predicted label equal to the training labels by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The analysis depends on hand-set thresholds and assumptions about the trustworthiness of reviewer scores and automated section parsing. These are inputs the reader must accept to take the central finding seriously; none are derived or externally validated in the paper.

free parameters (3)
  • Reviewer disagreement exclusion threshold = max-min <= 1
    Papers are dropped when the spread across reviewer TNS scores exceeds 1, chosen to ensure label consistency; this filters the dataset and changes the label distribution (Section 3.1).
  • Label merge rule = TNS 1,2 -> 0; 3 -> 1; 4 -> 2
    Defines the three novelty classes to handle the imbalanced distribution; this mapping determines what the classifiers learn (Section 3.1).
  • Section classification confidence threshold = 0.8
    If the fine-tuned SciBERT section classifier has confidence below 0.8, LLaMA3 and manual correction are invoked; the threshold is justified qualitatively to minimize false positives, with no sensitivity analysis (Section 3.3).
assumptions (4)
  • domain assumption Reviewer-provided TNS scores are a valid reference standard for novelty.
    The entire supervised task depends on treating ICLR reviewer scores as ground truth (Sections 1 and 3.1).
  • domain assumption The fine-tuned SciBERT section classifier (trained on ArXiv and PubMed data, 91 percent accuracy) transfers to full ICLR paper text.
    Section structure identification for all ICLR papers relies on this model plus LLaMA3 agreement and manual correction (Section 3.3).
  • ad hoc to paper The three-class aggregation of TNS preserves the order and meaning of novelty levels.
    The mapping of four reviewer levels into three classes is a modeling decision made for this paper and is not independently validated (Section 3.1).
  • ad hoc to paper Zero-shot LLM outputs, after five repeated generations with majority vote, are stable enough to compare section combinations.
    GPT-3.5 and GPT-4o are evaluated on small resampled sets (40 per class) with no fixed test set, and the majority-vote procedure is assumed to remove randomness (Sections 3.5 and 4.4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers." pith.science (2026). https://pith.science/paper/5UTXXCDR

@misc{pith2026250516330,
  author       = {Pith},
  title        = {Pith review of: SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5UTXXCDR}},
  note         = {Machine review of arXiv:2505.16330}
}
read the original abstract

Novelty is a core component of academic papers, and there are multiple perspectives on the assessment of novelty. Existing methods often focus on word or entity combinations, which provide limited insights. The content related to a paper's novelty is typically distributed across different core sections, e.g., Introduction, Methodology and Results. Therefore, exploring the optimal combination of sections for evaluating the novelty of a paper is important for advancing automated novelty assessment. In this paper, we utilize different combinations of sections from academic papers as inputs to drive language models to predict novelty scores. We then analyze the results to determine the optimal section combinations for novelty score prediction. We first employ natural language processing techniques to identify the sectional structure of academic papers, categorizing them into introduction, methods, results, and discussion (IMRaD). Subsequently, we used different combinations of these sections (e.g., introduction and methods) as inputs for pretrained language models (PLMs) and large language models (LLMs), employing novelty scores provided by human expert reviewers as ground truth labels to obtain prediction results. The results indicate that using introduction, results and discussion is most appropriate for assessing the novelty of a paper, while the use of the entire text does not yield significant results. Furthermore, based on the results of the PLMs and LLMs, the introduction and results appear to be the most important section for the task of novelty score prediction. The code and dataset for this paper can be accessed at https://github.com/njust-winchy/SC4ANM.

Figures

Figures reproduced from arXiv: 2505.16330 by the authors.

Figure 1
Figure 1. An Example of review report on ICLR 2022. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Framework of this study. conference in the field of machine learning. We wrote a web crawler code to retrieve a total of 8183 ICLR papers from ICLR 2022 and 2023, each con￾taining peer review comments. The reason for selecting papers from these two years as the data source is that the review reports for these years require reviewers to provide novelty scores, as shown in [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Prompt of Llama 3 for section structure identification. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An Example of Fine-tuning PLMs for Novelty Score Prediction. Using [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: An Example used to prompt the large language model for the novelty score [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Case study on PLM (Longformer version of SciBERT) for novelty score [PITH_FULL_IMAGE:figures/full_fig_p031_6.png]
Figure 7
Figure 7. Figure 7: Case study on LLM (GPT-4o) for novelty score prediction using different [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 21 canonical work pages

  1. [5]

    doi:https://dl

    Language models are few-shot learners, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. doi:https://dl. acm.org/doi/abs/10.5555/3495724.3495883. Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., Tortora, G.,

    arXiv

  2. [7]

    The Electronic Library doi:https://doi.org/10.1108/EL-03-2024-0070

    Explor- ing the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities. The Electronic Library doi:https://doi.org/10.1108/EL-03-2024-0070. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.,

    work page doi:10.1108/el-03-2024-0070 2024

  3. [9]

    Expert Systems with Applications , 125533doi:https: //doi.org/10.1016/j.eswa.2024.125533

    Integrated sentiment analysis with bert for enhanced hybrid recommen- dation systems. Expert Systems with Applications , 125533doi:https: //doi.org/10.1016/j.eswa.2024.125533. Fagerberg, J.,

    work page arXiv 2024

  4. [12]

    URL:https://arxiv.org/abs/ 2402.10886,arXiv:2402.10886

    Reviewer2: Optimizing review generation through prompt generation. URL:https://arxiv.org/abs/ 2402.10886,arXiv:2402.10886. Guetzkow, J., Lamont, M., Mallard, G.,

    arXiv

  5. [15]

    Journal of Informetrics 16, 101306

    A new method for measuring the original- ity of academic articles based on knowledge units in semantic networks. Journal of Informetrics 16, 101306. URL:https://www.sciencedirect. com/science/article/pii/S175115772200058X, doi:https://doi.org/ 10.1016/j.joi.2022.101306. Jeon, D., Lee, J., Ahn, J.M., Lee, C.,

    arXiv

  6. [16]

    Journal of Infor- metrics 17, 101450

    Measuring the novelty of scientific publications: A fasttext and local outlier factor approach. Journal of Infor- metrics 17, 101450. doi:https://doi.org/10.1016/j.joi.2023.101450. Ji, Y., Zhang, Q., Shen, S., Wang, D., Huang, S.,

    arXiv 2023

  7. [17]

    Research on func- tional structure identification of academic text based on deep learning, in: 17TH INTERNATIONAL CONFERENCE ON SCIENTOMETRICS & INFORMETRICS (ISSI2019), VOL II, INT SOC SCIENTOMETRICS & INFORMETRICS-ISSI. pp. 2712–2713. doi:https://doi.org/10.13266/ j.issn.0252-3116.2019.13.010. Kocmi, T., Federmann, C.,

    work page arXiv

  8. [18]

    (Eds.), Proceedings of the 40 24th Annual Conference of the European Association for Machine Trans- lation, European Association for Machine Translation, Tampere, Finland

    Large language models are state-of-the-art evaluators of translation quality, in: Nurminen, M., Brenner, J., Koponen, M., Latomaa, S., Mikhailov, M., Schierl, F., Ranasinghe, T., Vanmassen- hove, E., Vidal, S.A., Aranberri, N., Nunziatini, M., Escartín, C.P., For- cada, M., Popovic, M., Scarton, C., Moniz, H. (Eds.), Proceedings of the 40 24th Annual Conf...

    work page 2023

Show all 45 references
  1. [19]

    NEJM AI 0, AIoa2400196

    Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI 0, AIoa2400196. doi:10.1056/AIoa2400196, arXiv:https://ai.nejm.org/doi/pdf/10.1056/AIoa2400196. Liu, M., Xie, Z., Yang, A.J., Yu, C., Xu, J., Ding, Y., Bu, Y.,

    doi

  2. [20]

    Information Processing & Manage- ment 61, 103743

    The prominent and heterogeneous gender disparities in scientific novelty: Evi- dence from biomedical doctoral theses. Information Processing & Manage- ment 61, 103743. doi:https://doi.org/10.1016/j.ipm.2024.103743. Liu, R., Shah, N.B.,

    2024

  3. [21]

    URL:https://arxiv.org/ abs/2306.00622,arXiv:2306.00622

    Reviewergpt? an exploratory study on using large language models for paper reviewing. URL:https://arxiv.org/ abs/2306.00622,arXiv:2306.00622. Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.,

    arXiv

  4. [22]

    (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Com- putational Linguistics, Online

    S2ORC: The semantic scholar open research corpus, in: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Com- putational Linguistics, Online. pp. 4969–4983. doi:10...

    doi

  5. [24]

    Journal of Infor- metrics 16, 101282

    Combination of research questions and methods: A new measurement of scientific novelty. Journal of Infor- metrics 16, 101282. doi:https://doi.org/10.1016/j.joi.2022.101282. Ma, B., Zhang, C., Wang, Y., Deng, S.,

    2022

  6. [25]

    Scientometrics 126, 6891–6915

    Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach. Scientometrics 126, 6891–6915. doi:https:// doi.org/10.1007/s11192-021-04049-z. Meta, A.,

    doi

  7. [27]

    URL:https://arxiv.org/abs/ 2303.08774,arXiv:2303.08774

    Gpt-4 technical report. URL:https://arxiv.org/abs/ 2303.08774,arXiv:2303.08774. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welind...

    arXiv

  8. [28]

    (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc

    Training language models to follow instructions with human feedback, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 27730–27744. URL: https://proceedings.neurips.cc...

    2022

  9. [29]

    Expert Systems with Applications 256, 124912

    Assessing llms in malicious code deobfuscation of real-world malware campaigns. Expert Systems with Applications 256, 124912. doi:https://doi.org/10.1016/j.eswa.2024. 124912. 42 Qin, C., Zhang, C.,

    2024 doi

  10. [30]

    Aslib Journal of Information Management 75, 884–916

    Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of aca- demic articles. Aslib Journal of Information Management 75, 884–916. doi:https://doi.org/10.1108/AJIM-05-2022-0244. Robertson, Z.,

    doi

  11. [31]

    URL:https://arxiv.org/abs/2307.05492, arXiv:2307.05492

    Gpt4 is slightly helpful for peer-review assis- tance: A pilot study. URL:https://arxiv.org/abs/2307.05492, arXiv:2307.05492. Rogers, M.,

    arXiv

  12. [33]

    Business Cycles: A Theoretical, Historical, and Statistical Analysis of the Capitalist Process, Martino Pub

    Business Cycles: A Theoretical, Historical, and Sta- tistical Analysis of the Capitalist Process. Business Cycles: A Theoretical, Historical, and Statistical Analysis of the Capitalist Process, Martino Pub. doi:https://doi.org/10.1086/ahr/46.1.96. Shafee, S., Bessani, A., Ferr...

    doi

  13. [34]

    Expert Systems with Applications , 125509doi:https://doi.org/10.1016/j.eswa.2024.125509

    Evaluationofllm-basedchatbots for osint-based cyber threat awareness. Expert Systems with Applications , 125509doi:https://doi.org/10.1016/j.eswa.2024.125509. Shen, C., Cheng, L., Nguyen, X.P., You, Y., Bing, L.,

    2024

  14. [35]

    (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Compu- tational Linguistics, Singapore

    Large language models are not yet human-level evaluators for abstractive summarization, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Compu- tational Linguistics, Singapore. pp. 4215–4233. doi...

    2023 doi

  15. [36]

    doi:https://doi.org/ 10.1016/j.joi.2018.07.005

    Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references? Journal of Informetrics 12, 906–930. doi:https://doi.org/ 10.1016/j.joi.2018.07.005. Thelwall, M.,

    2018 doi

  16. [37]

    doi:doi:10.2478/jdis-2024-0013

    Can chatgpt evaluate research quality? Journal of Data and Information Science 9, 1–21. doi:doi:10.2478/jdis-2024-0013. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Gra...

    2024 arXiv

  17. [39]

    Research Policy 46, 1416–1436

    Bias against novelty in science: 44 A cautionary tale for users of bibliometric indicators. Research Policy 46, 1416–1436. doi:https://doi.org/10.1016/j.respol.2017.06.006. Wang, Z., Zhang, H., Chen, J., Chen, H.,

    2017 doi

  18. [40]

    Journal of Informetrics 18, 101587

    An effective framework for measuring the novelty of scientific articles through integrated topic modeling and cloud model. Journal of Informetrics 18, 101587. doi:https: //doi.org/10.1016/j.joi.2024.101587. Wu, W., Xi, H., Zhang, C.,

    2024

  19. [41]

    Plos one 18, e0284567

    Identify novel elements of knowledge with word embedding. Plos one 18, e0284567. doi:https://doi.org/10.1371/journal.pone.0284567. Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., On- tanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., Ahmed, A.,

    doi

  20. [42]

    (Eds.), Advances in Neu- ral Information Processing Systems, Curran Associates, Inc

    Big bird: Transformers for longer sequences, in: Larochelle, H., Ran- zato, M., Hadsell, R., Balcan, M., Lin, H. (Eds.), Advances in Neu- ral Information Processing Systems, Curran Associates, Inc.. pp. 17283– 17297. URL:https://proceedings.neurips.cc/paper_files/paper/ 2020/f...

    2020

  21. [43]

    (Eds.), Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia

    Is LLM a reliable reviewer? a compre- hensive evaluation of LLM on automatic paper reviewing tasks, in: Cal- zolari, N., Kan, M.Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (Eds.), Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language ...

    2024

  22. [44]

    deep learning for pa- per section identification: Toward applications in chinese medical liter- ature

    Feature engineering vs. deep learning for pa- per section identification: Toward applications in chinese medical liter- ature. Information Processing & Management 57, 102206. doi:https: //doi.org/10.1016/j.ipm.2020.102206. Zhu, C., Yi, B., Luo, L.,

    2020

  23. [1985]

    harvard university press

    An evolutionary theory of economic change. harvard university press. doi:https://doi.org/10.2307/2232409. OpenAI,

    doi

  24. [2004]

    doi:10.1177/000312240406900203

    What is originality in the humanities and the social sciences? American Sociological Review 69, 190–212. doi:10.1177/000312240406900203. Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.H., Yang, Y.,

    doi

  25. [2006]

    Oxford University Press

    1 innovation: A guide to the literature, in: The Ox- ford Handbook of Innovation. Oxford University Press. doi:10.1093/ oxfordhb/9780199286805.003.0001. Foster, J.G., Shi, F., Evans, J.,

  26. [2012]

    Cre- ativity Research Journal 24, 92–96

    The standard definition of creativity. Cre- ativity Research Journal 24, 92–96. doi:10.1080/10400419.2012.650092. Schumpeter, J.,

    2012

  27. [2013]

    Sci- ence 342, 468–472

    Atypical combinations and scientific impact. Sci- ence 342, 468–472. doi:10.1126/science.1240474, arXiv:https://www.science.org/doi/pdf/10.1126/science.1240474. Wang, H.,

    doi

  28. [2016]

    Looking across and looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation in science. Manage. Sci. 62, 2765–2783. URL:https://doi.org/10.1287/mnsc.2015.2285, doi:10.1287/mnsc. 2015.2285. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kap...

    2015

  29. [2017]

    A dynamic network measure of techno- logical change. Manage. Sci. 63, 791–817. URL:https://doi.org/10. 1287/mnsc.2015.2366, doi:10.1287/mnsc.2015.2366. Gao, Z., Brantley, K., Joachims, T.,

    2015

  30. [2018]

    Scientometrics 115, 463–486

    Functional structure identi- fication of scientific documents in computer science. Scientometrics 115, 463–486. doi:https://doi.org/10.1007/s11192-018-2640-y. 41 Luo, Z., Lu, W., He, J., Wang, Y.,

    doi

  31. [2019]

    SciBERT: A pretrained language model for scientific text, in: Inui, K., Jiang, J., Ng, V., Wan, X. (Eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNL...

    2019 doi

  32. [2020]

    URL:https://arxiv.org/abs/2004.05150, arXiv:2004.05150

    Longformer: The long- document transformer. URL:https://arxiv.org/abs/2004.05150, arXiv:2004.05150. Boudreau, K.J., Guinan, E.C., Lakhani, K.R., Riedl, C.,

    2004 arXiv

  33. [2021]

    Re- search Policy 50, 104144

    Natural language pro- cessing to identify the creation and impact of new technolo- gies in patent text: Code, data, and new measures. Re- search Policy 50, 104144. URL:https://www.sciencedirect.com/ science/article/pii/S0048733320302195, doi:https://doi.org/10. 1016/j.respol.2...

  34. [2022]

    (Eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Association for Computational Linguistics, Seattle, United States

    LongT5: Efficient text-to-text transformer for long sequences, in: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (Eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Association for Computational Linguistics, Seattle, United States. pp. 724–736. doi...

    2022

  35. [2023]

    Journal of Machine Learning Research 24, 1–113

    Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 1–113. doi:https://dl.acm.org/doi/10.5555/ 3648699.3648939. Cohan, A., Dernoncourt, F., Kim, D.S., Bui, T., Kim, S., Chang, W., Go- harian, N., 2018a. A discourse-aware attention model for ...

  36. [2024]

    Expert Systems with Applications 235, 121186

    Can chatgpt provide intelligent diagnoses? a com- parative study between predictive models and chatgpt to define a new 38 medical diagnostic bot. Expert Systems with Applications 235, 121186. doi:https://doi.org/10.1016/j.eswa.2023.121186. Chen, Z., Zhang, C., Zhang, H., Zhao,...

  37. [2025]

    Expert Systems with Applications 259, 125295

    Aspect-based sentiment analysis via bidirec- tional variant spiking neural p systems. Expert Systems with Applications 259, 125295. doi:https://doi.org/10.1016/j.eswa.2024.125295. 45

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.