Pith. sign in

REVIEW 4 major objections 4 minor 73 references

Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining reviewer novelty sentences with ChatGPT method summaries, fused by sparse attention, predicts ICLR 2022 method novelty at 0.84 accuracy, the paper reports.

desk verdict A genuinely leaky evaluation: the human-knowledge input and the gold label come from the same reviewers, so the reported gains likely measure reviewer-text transduction, not novelty prediction. read the letter →

arxiv 2507.11330 v2 pith:MAKF5UKY submitted 2025-07-15 cs.CL cs.AIcs.DLcs.HC

Wenqing Wu , Chengzhi Zhang , Yi Zhao This is my paper
classification cs.CLcs.AIcs.DLcs.HC
keywords methodnoveltypredictionpeerreviewlargelanguagemodelsknowledgefusionsparseattentionoriginalityassessmentICLRChatGPT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automated prediction of whether a paper's method is novel works best when a pretrained language model receives two complementary text inputs: sentences from peer-review reports that discuss novelty, and a short ChatGPT-generated summary of the paper's methodology section. On 2,432 ICLR 2022 papers, with reviewer novelty scores collapsed into high and low classes, the proposed fusion model reaches 0.83 F1 and 0.84 accuracy, outperforming the best baseline (SciBERT at 0.73 and 0.74) and several large language models used directly as judges. The contribution matters because current novelty assessment is either slow expert judgment or uncertain citation-combination heuristics, and a text-based predictor would give a fast, scalable first pass. The paper further reports that a knowledge-guided sparse-attention module is the main source of improvement, and that smaller pretrained models benefit most from combining human and LLM knowledge.

What carries the argument

The central mechanism is the knowledge-guided fusion module. It encodes the two text inputs with a pretrained language model, runs self-attention over the ChatGPT summary to form method-aware features, and then applies a sparse-attention layer — attention with a sparsemax-style sparse weight distribution — so the review-text features attend selectively to method features instead of pooling all tokens. A self-attention reduction layer pools the fused sequence and feeds a two-class softmax. The load-bearing idea is that sparse attention forces the model to link each human novelty sentence to the specific few method phrases it is commenting on, which is what lets the two knowledge sources reinforce each other.

What would settle it

Keep the pipeline identical but replace each paper's extracted novelty sentences with another paper's novelty sentences, or with boilerplate review text that expresses no novelty opinion. If accuracy stays near 0.84, the human-knowledge channel contributes real signal beyond label leakage; if it drops sharply, the classifier was transcribing reviewer sentiment from the same reports that supplied the labels.

Watch

Extended reading notes

Core claim

The paper's central claim is that method novelty in academic papers can be predicted from text by fusing 'human knowledge' — sentences extracted from peer-review reports that an aspect-annotation model labels as discussing novelty — with 'LLM knowledge' — a single-pass ChatGPT summary of the paper's methodology section. The authors introduce the Method Novelty Prediction task, collapse the ICLR 2022 Technical Novelty and Significance scores into Low/High classes, and report that their knowledge-guided fusion model beats five pretrained-language-model baselines and several LLM baselines, with the SciBERT variant reaching 0.83 F1 and 0.84 accuracy. Ablations show the knowledge-guided module contributes the largest single gain: removing it drops F1 from 0.83 to 0.73, which the paper reads as evidence that human and LLM knowledge guide each other effectively and that the collaboration is most useful for smaller-parameter models.

Load-bearing premise

The load-bearing premise is that the human-knowledge sentences are independent of the scores they are predicting; because the sentences are extracted from the same review reports that contain the Technical Novelty scores, the model may be learning to reproduce reviewers' stated opinions instead of judging the paper's actual method novelty.

Editorial extensions

If this is right

  • A text-only pipeline of this kind could give authors, editors, and program committees a fast, reproducible first-pass novelty signal before full expert review.
  • The same recipe — extract aspect sentences from reviews, summarize the relevant paper section with an LLM, fuse with sparse attention — can be adapted to other novelty types such as theoretical or empirical novelty by changing which section is summarized.
  • Because smaller models like ALBERT gain the most from the fusion, the approach is feasible for deployment on limited compute.
  • Direct LLM judging stays unreliable: the paper shows ChatGPT tends to assign high novelty scores regardless of the expert score, so human text is needed to anchor the prediction.
  • Venues that do not collect explicit novelty scores could still get a unified low/high novelty judgment from free-text review comments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether its human-knowledge channel leaks the label: the novelty sentences and the TNS scores come from the same review reports, so part of the gain may be the model reading reviewer sentiment rather than judging the method. A clean test would withhold review text or pair each review with a different paper's method summary.
  • A testable extension is to feed the model pre- and post-rebuttal review texts of the same paper; if predicted novelty moves with tonal changes rather than with actual method revisions, that confirms the sentiment-reading channel.
  • The fusion design is not tied to ChatGPT; replacing the single LLM summary with retrieval-augmented or ensembled summaries from multiple LLMs is a natural next step that would test how much of the gain comes from summary quality versus the fusion architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a collaborative framework for automated method-novelty prediction of academic papers. Human knowledge is operationalized as sentences automatically extracted from peer review reports, and LLM knowledge is generated by prompting ChatGPT to summarize and evaluate the paper's method section; these two text sources are fed into a pre-trained language model and combined through a knowledge-guided sparse-attention module. The authors construct a dataset from ICLR 2022 papers whose gold labels are aggregated Technical Novelty and Significance (TNS) scores, binarized into Low/High novelty. On this dataset the best configuration (Ours-SciBERT) reaches 0.83 F1 and 0.84 accuracy (Table 2), outperforming PLM baselines and several LLMs. Ablations (Table 4) attribute parts of the gain to the knowledge-guided module, self-attention, and the self-attention reduction component.

Significance. If the reported results were valid, the paper would offer a practical pipeline for screening methodological novelty and would demonstrate a useful synergy between human review text and LLM summarization. The authors provide a public code/data commitment, evaluate a wide set of baselines including recent LLMs, and address a genuinely important problem in scholarly publishing. The central claim, however, is undermined by a design-level information leak: the "human knowledge" input is extracted from the same peer review reports whose TNS scores define the gold labels. The classifier is therefore able to predict reviewers' textual expressions of their own novelty judgments, rather than evaluating the paper's method. The decisive comparison — proposed method versus baselines on paper-only inputs — is not reported, and the paper's own Table 3 shows that human-knowledge sentences alone nearly match the strongest baseline. Consequently, the contribution to automated novelty assessment as framed in the abstract is not established.

major comments (4)
  1. [Section 3.2, Figure 4] The "human knowledge" input is derived from the same ICLR 2022 review reports whose TNS scores are aggregated into the gold labels. The review sentences are the reviewers' own written justifications for their TNS scores; the model can thus learn a text-to-score transduction (e.g., recognizing phrases like "not novel" or "well-known technique") instead of assessing the paper's method. Table 3 is consistent with this: SciBERT using HK alone gives 0.71 F1 / 0.71 accuracy, nearly matching the SciBERT baseline with HK & LLMK (0.73 / 0.74), while the method text alone gives only 0.55 / 0.60. The paper does not report results of the proposed model on method-text-only input, which is the decisive test for whether the fusion actually assesses novelty rather than reviewer sentiment.
  2. [Tables 2 and 3] All results are reported without variance, confidence intervals, significance tests, or multiple splits; Section 5.1 describes a single 8:1:1 split. The reported advantage of Ours-SciBERT (0.83 F1) over the SciBERT baseline (0.73 F1) is large, but given that the HK channel already encodes the label, the individual contributions of the sparse-attention fusion module and the self-attention reduction module could easily be within noise. A majority-class baseline (the pooled distribution in Table 1 is about 58.6% Low) would also help contextualize all accuracy figures.
  3. [Section 5.1 and Section 4.2.3] The description of the training loss is unclear. The paper states "We randomly select 5 negative samples for the NMP task during training" and "we employ negative sampling techniques (Mikolov et al., 2013) to compute the loss between predictions and true values." In a binary classification task with cross-entropy, negative sampling is not a standard procedure; if negatives are subsampled, the effective training distribution differs from the reported one, and the exact evaluation protocol is ambiguous. This hampers reproducibility.
  4. [Section 4.1 and Abstract] The task is defined as assigning a novelty label based on "the review text and feedback," and the gold label is the aggregated reviewer TNS score. Because the review text contains the reviewers' own novelty statements, the task is more accurately described as predicting reviewers' aggregate novelty judgment from their textual evaluations. The abstract's claim that "our method achieves superior performance" in automated novelty evaluation is therefore overstated; at best, the experiments demonstrate high accuracy at predicting reviewer opinions, not at independently judging the paper's methodological novelty.
minor comments (4)
  1. [Abstract, first line] There is a typo in "evaluatin g academic papers" (missing space in the rendered text).
  2. [Section 5.1] The text says "Number of output linear layers for NMP task is 2," but the task is abbreviated MNP throughout the paper; this should be corrected for consistency.
  3. [Section 4.2.2, Eq. (3)] The notation "SparseAttention(qv km / sqrt(dh))" is not defined; the sparse-attention operator and the role of the outer softmax should be specified precisely, since the paper claims this as a novel component.
  4. [Table 1 and Section 5.3] The class distribution after binarization is imbalanced (1,425 Low vs. 1,007 High). Reporting accuracy alone is not sufficient; a majority-class baseline and class-wise precision/recall would clarify whether the gains reflect the leaking HK channel or genuine signal.

Circularity Check

1 steps flagged · score 8.0 of 10

Human-knowledge sentences are extracted from the very review reports whose TNS scores define the labels, so the reported novelty prediction largely transduces reviewer text into reviewer scores.

  1. self definitional [Section 3.2 (Dataset Pre-processing), Figure 4; Section 5.3.2]
    "Hence, we utilized the model to annotate peer review texts and extracted sentences marked as aspects of novelty. ... As we need to evaluate the novelty of the methodology section of academic papers, we retained only the scores for Technical Novelty and Significance (TNS) as the gold standard for novelty. ... This is because HK consist of sentences related to novelty evaluation, which, as human knowledge, can guide the model to make relatively accurate judgments."

    The prediction target is the aggregated TNS score assigned by ICLR 2022 reviewers, and the HK input is the subset of those same reviewers' sentences that an aspect-annotation model labels as 'novelty.' The reviewers' TNS score and their novelty sentences are two expressions of the same act of evaluation; a classifier that reads HK is therefore transducing reviewer language into reviewer scores rather than measuring the paper's method. Table 3 confirms this: HK alone reaches 0.71 F1/0.71 accuracy with SciBERT, nearly matching the HK+LLMK baseline (0.73/0.74) and far exceeding method-text alone (0.55/0.60).

full rationale

The central evaluation is circular at the construct level: the gold label (TNS) and the 'human knowledge' input (HK) are extracted from the same peer-review reports. Section 3.2 states that TNS is 'the gold standard for novelty' and that novelty sentences are 'extracted' from the review texts; Section 5.3.2 admits that HK alone gives relatively good performance 'because HK consist of sentences related to novelty evaluation.' Since the reviewers' written novelty comments are the textual justification for their TNS scores, a model using HK is learning to map reviewers' words to reviewers' scores. Table 3 shows HK alone (0.71 F1 with SciBERT) nearly equals the best HK+LLMK baseline (0.73), which makes the 'superior performance' of the fusion largely a label-transduction artifact. The paper is not self-citation-circular: the citation to Zhao & Zhang (2025) is related work and is not load-bearing. The knowledge-guided module's incremental gain over the HK+LLMK baseline (0.83 vs 0.73) is a real, non-circular comparison in the narrow sense, but the headline claim that the method assesses method novelty remains compromised because the HK input contains the label's own justification. A clean evaluation would need paper-only inputs (method text, LLM summary, or external annotations) rather than reviewer novelty sentences.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; the 'knowledge-guided module' is a software component. The load-bearing assumptions are domain assumptions about the validity of ICLR TNS scores, the reliability of the aspect annotation model, the reliability of ChatGPT summaries, and the faithfulness of methodology-section extraction. The free parameters listed are hand-chosen thresholds and hyperparameters that shape the task and the model.

free parameters (6)
  • Reviewer disagreement exclusion threshold = difference > 1
    Papers where the maximum and minimum reviewer TNS scores differ by more than 1 are removed from the dataset (Section 3.2); this choice affects the label distribution and task difficulty.
  • Binary novelty split = TNS 1-2 low, 3-4 high
    Scores are collapsed into two classes to avoid imbalance (Section 3.2); this is a manual threshold that defines the task.
  • Number of negative samples = 5
    Negative sampling count for the classification loss set in Section 5.1.
  • Sparse attention dimension = 128
    Hyperparameter for the Sparse-Attention layer (Section 5.1).
  • Number of sparse attention heads = 6
    Hyperparameter for the Sparse-Attention layer (Section 5.1).
  • Dropout = 0.2
    Regularization hyperparameter (Section 5.1).
assumptions (5)
  • domain assumption ICLR 'Technical Novelty and Significance' scores are a valid gold standard for method novelty.
    Section 3.2 uses TNS as the binary label; the paper assumes this expert score reflects actual methodological novelty.
  • domain assumption Novelty-related sentences extracted from peer reviews by the Yuan et al. (2022) aspect annotation model faithfully represent human novelty judgments.
    Section 3.2 relies on the pretrained annotation model without evaluating its accuracy on ICLR reviews.
  • domain assumption ChatGPT (gpt-3.5-turbo) summaries of methodology novelty are reliable inputs.
    Section 3.1 states 'Through our manual assessment, the feedback generated by ChatGPT is relatively reliable' without reporting inter-rater agreement or validation.
  • domain assumption The methodology section extracted by GROBID/S2ORC plus manual rules is a faithful representation of the paper's method.
    Section 3.2; extraction errors would corrupt the LLM summaries used as input.
  • domain assumption Excluding papers with reviewer disagreement does not bias the evaluation.
    Section 3.2 removes cases with score spread >1; the paper does not report results on the excluded subset or justify that these cases are uninformative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge." pith.science (2026). https://pith.science/paper/MAKF5UKY

@misc{pith2026250711330,
  author       = {Pith},
  title        = {Pith review of: Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAKF5UKY}},
  note         = {Machine review of arXiv:2507.11330}
}
read the original abstract

Novelty is a crucial criterion in the peer review process for evaluating academic papers. Traditionally, it's judged by experts or measure by unique reference combinations. Both methods have limitations: experts have limited knowledge, and the effectiveness of the combination method is uncertain. Moreover, it's unclear if unique citations truly measure novelty. The large language model (LLM) possesses a wealth of knowledge, while human experts possess judgment abilities that the LLM does not possess. Therefore, our research integrates the knowledge and abilities of LLM and human experts to address the limitations of novelty assessment. One of the most common types of novelty in academic papers is the introduction of new methods. In this paper, we propose leveraging human knowledge and LLM to assist pretrained language models (PLMs, e.g. BERT etc.) in predicting the method novelty of papers. Specifically, we extract sentences related to the novelty of the academic paper from peer review reports and use LLM to summarize the methodology section of the academic paper, which are then used to fine-tune PLMs. In addition, we have designed a text-guided fusion module with novel Sparse-Attention to better integrate human and LLM knowledge. We compared the method we proposed with a large number of baselines. Extensive experiments demonstrate that our method achieves superior performance.

Figures

Figures reproduced from arXiv: 2507.11330 by the authors.

Figure 1
Figure 1. ChatGPT provides feedback and scores for the novel [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An Example of review report on ICLR 2022. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of our prompt and feedback. PDFs into JSON format. Subsequently, we manually defined numerous rules to extract the title and the methodology sections from the parsed PDFs. For example, it is necessary to determine whether the method section appears after or before the related work, as the placement of the research method section is not fixed. To meet our research requirements, we utilized the extracted ti… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of our study. 4. Methodology In this section, we will introduce the Task definition, followed by an intro￾duction to the Proposed Prediction model. Furthermore, as shown in [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Case Study of Different Input and Our Method. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 45 canonical work pages

  1. [1]

    author Amplayo, R. K. , author Hong, S. , & author Song, M. ( year 2018 ). title Network-based approach to detect novelty of scholarly literature . journal Information sciences \/ , volume 422 \/ , pages 542--557

    work page 2018

  2. [2]

    ( year 2024 )

    author Anthropic, A. ( year 2024 ). title Claude 3.5 sonnet model card addendum . journal Claude-3.5 Model Card \/ , volume 3 \/

    work page 2024

  3. [3]

    , author Hou, J

    author Arts, S. , author Hou, J. , & author Gomez, J. C. ( year 2021 ). title Natural language processing to identify the creation and impact of new technologies in patent text: Code, data, and new measures . journal Research Policy \/ , volume 50 \/ , pages 104144 . https://www.sciencedirect.com/science/article/pii/S0048733320302195. :https://doi.org/10....

    work page arXiv 2021

  4. [4]

    author Ayers, J. W. , author Poliak, A. , author Dredze, M. , author Leas, E. C. , author Zhu, Z. , author Kelley, J. B. , author Faix, D. J. , author Goodman, A. M. , author Longhurst, C. A. , author Hogarth, M. et al. ( year 2023 ). title Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social med...

    work page 2023

  5. [5]

    , author Lo, K

    author Beltagy, I. , author Lo, K. , & author Cohan, A. ( year 2019 ). title S ci BERT : A pretrained language model for scientific text . In editor K. Inui , editor J. Jiang , editor V. Ng , & editor X. Wan (Eds.), booktitle Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o...

    2019

  6. [6]

    author Boudreau, K. J. , author Guinan, E. C. , author Lakhani, K. R. , & author Riedl, C. ( year 2016 ). title Looking across and looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation in science . journal Manage. Sci. \/ , volume 62 \/ , pages 2765–2783 . https://doi.org/10.1287/mnsc.2015.2285. :10.1287/mnsc.2015.2285

    work page arXiv 2016

  7. [7]

    , author Gonz \'a lez-Geraldo, J

    author Carabantes, D. , author Gonz \'a lez-Geraldo, J. L. , & author Jover, G. ( year 2023 ). title Chatgpt could be the reviewer of your next scientific paper. evidence on the limits of ai-assisted academic reviews . journal Profesional de la informaci \'o n/Information Professional \/ , volume 32 \/

    work page 2023

  8. [8]

    author Dahlin, K. B. , & author Behrens, D. M. ( year 2005 ). title When is an invention really radical?: Defining and measuring technological radicalness . journal Research policy \/ , volume 34 \/ , pages 717--737

    work page 2005

Show all 73 references
  1. [9]

    , author Chang, M.-W

    author Devlin, J. , author Chang, M.-W. , author Lee, K. , & author Toutanova, K. ( year 2019 ). title BERT : Pre-training of deep bidirectional transformers for language understanding . In editor J. Burstein , editor C. Doran , & editor T. Solorio (Eds.), booktitle Proceeding...

    2019

  2. [10]

    , author Jauhri, A

    author Dubey, A. , author Jauhri, A. , author Pandey, A. , author Kadian, A. , author Al-Dahle, A. , author Letman, A. , author Mathur, A. , author Schelten, A. , author Yang, A. , author Fan, A. , author Goyal, A. , author Hartshorn, A. , author Yang, A. , author Mitra, A. , ...

    2024 arXiv

  3. [11]

    , author Kuznetsov, I

    author Dycke, N. , author Kuznetsov, I. , & author Gurevych, I. ( year 2023 ). title NLP eer: A unified resource for the computational study of peer review . In editor A. Rogers , editor J. Boyd-Graber , & editor N. Okazaki (Eds.), booktitle Proceedings of the 61st Annual Meet...

    2023

  4. [12]

    author Foster, J. G. , author Shi, F. , & author Evans, J. ( year 2021 ). title Surprise! Measuring Novelty as Expectation Violation \/ . type SocArXiv number 2t46f Center for Open Science. https://ideas.repec.org/p/osf/socarx/2t46f.html. :10.31219/osf.io/2t46f

    2021 doi

  5. [13]

    , author Eger, S

    author Gao, Y. , author Eger, S. , author Kuznetsov, I. , author Gurevych, I. , & author Miyao, Y. ( year 2019 ). title Does my rebuttal matter? insights from a major NLP conference . In editor J. Burstein , editor C. Doran , & editor T. Solorio (Eds.), booktitle Proceedings o...

    2019

  6. [14]

    , author Kumar, S

    author Ghosal, T. , author Kumar, S. , author Bharti, P. K. , & author Ekbal, A. ( year 2022 a ). title Peer review analyze: A novel benchmark resource for computational analysis of peer reviews . journal Plos one \/ , volume 17 \/ , pages e0259238

    2022

  7. [15]

    , author Varanasi, K

    author Ghosal, T. , author Varanasi, K. K. , & author Kordoni, V. ( year 2022 b ). title Hedgepeer: A dataset for uncertainty detection in peer reviews . In booktitle Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries \/ (pp. pages 1--5 )

    2022

  8. [16]

    , author Verma, R

    author Ghosal, T. , author Verma, R. , author Ekbal, A. , & author Bhattacharyya, P. ( year 2019 ). title D eep S enti P eer: Harnessing sentiment in review texts to recommend peer review decisions . In editor A. Korhonen , editor D. Traum , & editor L. M \`a rquez (Eds.), boo...

    2019

  9. [17]

    , author Lubis, N

    author Heck, M. , author Lubis, N. , author Ruppik, B. , author Vukovic, R. , author Feng, S. , author Geishauser, C. , author Lin, H.-c. , author van Niekerk, C. , & author Gasic, M. ( year 2023 ). title C hat GPT for zero-shot dialogue state tracking: A solution or an opport...

    2023

  10. [18]

    , & author Gimpel, K

    author Hendrycks, D. , & author Gimpel, K. ( year 2016 ). title Gaussian error linear units (gelus) . journal arXiv preprint arXiv:1606.08415 \/ ,

    2016 arXiv

  11. [19]

    , author Wang, D

    author Hou, J. , author Wang, D. , & author Li, J. ( year 2022 ). title A new method for measuring the originality of academic articles based on knowledge units in semantic networks . journal Journal of Informetrics \/ , volume 16 \/ , pages 101306 . https://www.sciencedirect....

    2022

  12. [20]

    , author Nikolov, M

    author Hua, X. , author Nikolov, M. , author Badugu, N. , & author Wang, L. ( year 2019 ). title Argument mining for understanding peer reviews . In editor J. Burstein , editor C. Doran , & editor T. Solorio (Eds.), booktitle Proceedings of the 2019 Conference of the North A m...

    2019

  13. [21]

    , author Huang, Y

    author Huang, S. , author Huang, Y. , author Liu, Y. , author Luo, Z. , & author Lu, W. ( year 2025 ). title Are large language models qualified reviewers in originality evaluation? journal Information Processing & Management \/ , volume 62 \/ , pages 103973 . https://www.scie...

    2025

  14. [22]

    author Hug, S. E. ( year 2022 ). title Towards theorizing peer review . journal Quantitative Science Studies \/ , volume 3 \/ , pages 815--831

    2022

  15. [23]

    author Jarrahi, M. H. , author Lutz, C. , author Boyd, K. , author Oesterlund, C. , & author Willis, M. ( year 2023 ). title Artificial intelligence in the work context

    2023

  16. [24]

    , author Ammar, W

    author Kang, D. , author Ammar, W. , author Dalvi, B. , author van Zuylen, M. , author Kohlmeier, S. , author Hovy, E. , & author Schwartz, R. ( year 2018 ). title A dataset of peer reviews ( P eer R ead): Collection, insights and NLP applications . In editor M. Walker , edito...

    2018

  17. [25]

    author Kennard, N. N. , author O ' Gorman, T. , author Das, R. , author Sharma, A. , author Bagchi, C. , author Clinton, M. , author Yelugam, P. K. , author Zamani, H. , & author McCallum, A. ( year 2022 ). title DISAPERE : A dataset for discourse structure in peer review disc...

    2022

  18. [26]

    author Kingma, D. P. , & author Ba, J. ( year 2014 ). title Adam: A method for stochastic optimization . journal arXiv preprint arXiv:1412.6980 \/ ,

    2014 arXiv

  19. [27]

    , author Ghosal, T

    author Kumar, S. , author Ghosal, T. , & author Ekbal, A. ( year 2023 ). title When reviewers lock horns: Finding disagreements in scientific peer reviews . In editor H. Bouamor , editor J. Pino , & editor K. Bali (Eds.), booktitle Proceedings of the 2023 Conference on Empiric...

    2023 doi

  20. [28]

    author Kung, T. H. , author Cheatham, M. , author Medenilla, A. , author Sillos, C. , author De Leon, L. , author Elepa \ n o, C. , author Madriaga, M. , author Aggabao, R. , author Diaz-Candido, G. , author Maningo, J. et al. ( year 2023 ). title Performance of chatgpt on usm...

    2023

  21. [29]

    author Kusumoto, F. M. , author Bittl, J. A. , author Creager, M. A. , author Dauerman, H. L. , author Lala, A. , author McDermott, M. M. , author Turco, J. V. , author Taqueti, V. R. , author Fuster, V. , & author of the Scientific Publications Committee, P. R. T. F. ( year 2...

    2023

  22. [30]

    , author Chen, M

    author Lan, Z. , author Chen, M. , author Goodman, S. , author Gimpel, K. , author Sharma, P. , & author Soricut, R. ( year 2019 ). title Albert: A lite bert for self-supervised learning of language representations . journal arXiv preprint arXiv:1909.11942 \/ ,

    2019 arXiv

  23. [31]

    , author Lee, J

    author Leahey, E. , author Lee, J. , & author Funk, R. J. ( year 2023 ). title What types of novelty are most disruptive? journal American Sociological Review \/ , volume 88 \/ , pages 562--597 . :10.1177/00031224231168074

    2023 doi

  24. [32]

    , author Srivastava, M

    author Lee, M. , author Srivastava, M. , author Hardy, A. , author Thickstun, J. , author Durmus, E. , author Paranjape, A. , author Gerard-Ursin, I. , author Li, X. L. , author Ladhak, F. , author Rong, F. et al. ( year 2022 ). title Evaluating human-language model interactio...

    2022 arXiv

  25. [33]

    , author Zhang, Y

    author Liang, W. , author Zhang, Y. , author Cao, H. , author Wang, B. , author Ding, D. , author Yang, X. , author Vodrahalli, K. , author He, S. , author Smith, D. , author Yin, Y. et al. ( year 2023 ). title Can large language models provide useful feedback on research pape...

    2023 arXiv

  26. [34]

    , author Song, J

    author Lin, J. , author Song, J. , author Zhou, Z. , author Chen, Y. , & author Shi, X. ( year 2023 a ). title Automated scholarly paper review: Concepts, technologies, and challenges . journal Information fusion \/ , volume 98 \/ , pages 101830

    2023

  27. [35]

    , author Song, J

    author Lin, J. , author Song, J. , author Zhou, Z. , author Chen, Y. , & author Shi, X. ( year 2023 b ). title Moprd: A multidisciplinary open peer review dataset . journal Neural Computing and Applications \/ , volume 35 \/ , pages 24191--24206

    2023

  28. [36]

    , & author Shah, N

    author Liu, R. , & author Shah, N. B. ( year 2023 ). title Reviewergpt? an exploratory study on using large language models for paper reviewing . journal arXiv preprint arXiv:2306.00622 \/ ,

    2023 arXiv

  29. [37]

    , author Ott, M

    author Liu, Y. , author Ott, M. , author Goyal, N. , author Du, J. , author Joshi, M. , author Chen, D. , author Levy, O. , author Lewis, M. , author Zettlemoyer, L. , & author Stoyanov, V. ( year 2019 ). title Roberta: A robustly optimized bert pretraining approach . journal ...

    2019 arXiv

  30. [38]

    , author Wang, L

    author Lo, K. , author Wang, L. L. , author Neumann, M. , author Kinney, R. , & author Weld, D. ( year 2020 ). title S 2 ORC : The semantic scholar open research corpus . In editor D. Jurafsky , editor J. Chai , editor N. Schluter , & editor J. Tetreault (Eds.), booktitle Proc...

    2020

  31. [39]

    , author Lu, W

    author Luo, Z. , author Lu, W. , author He, J. , & author Wang, Y. ( year 2022 ). title Combination of research questions and methods: A new measurement of scientific novelty . journal Journal of Informetrics \/ , volume 16 \/ , pages 101282

    2022

  32. [40]

    , & author Astudillo, R

    author Martins, A. , & author Astudillo, R. ( year 2016 ). title From softmax to sparsemax: A sparse model of attention and multi-label classification . In booktitle International conference on machine learning \/ (pp. pages 1614--1623 ). organization PMLR

    2016

  33. [41]

    , author Chen, E

    author Matsui, A. , author Chen, E. , author Wang, Y. , & author Ferrara, E. ( year 2021 ). title The impact of peer review on the contribution potential of scientific papers . journal PeerJ \/ , volume 9 \/ , pages e11999

    2021

  34. [42]

    , author Shibayama, S

    author Matsumoto, K. , author Shibayama, S. , author Kang, B. , & author Igami, M. ( year 2021 ). title Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach . journal Scientometrics \/ , volume 126 \/ , pages 6891--6915

    2021

  35. [43]

    , author Sutskever, I

    author Mikolov, T. , author Sutskever, I. , author Chen, K. , author Corrado, G. S. , & author Dean, J. ( year 2013 ). title Distributed representations of words and phrases and their compositionality . journal Advances in neural information processing systems \/ , volume 26 \/

    2013

  36. [44]

    , author Hall, L

    author Mulligan, A. , author Hall, L. , & author Raphael, E. ( year 2013 ). title Peer review in a changing world: An international study measuring the attitudes of researchers . journal Journal of the American Society for Information Science and Technology \/ , volume 64 \/ ,...

    2013

  37. [45]

    , & author Winter, S

    author Nelson, R. , & author Winter, S. ( year 1982 ). title An evolutionary theory of economic change harvard university press . journal Cambridge, MA and London \/ ,

    1982

  38. [46]

    title Gpt-4 technical report ,

    author OpenAI ( year 2024 ). title Gpt-4 technical report , . https://arxiv.org/abs/2303.08774. arXiv:2303.08774 http://arxiv.org/abs/2303.08774

    2024 arXiv

  39. [47]

    , author Wu, J

    author Ouyang, L. , author Wu, J. , author Jiang, X. , author Almeida, D. , author Wainwright, C. , author Mishkin, P. , author Zhang, C. , author Agarwal, S. , author Slama, K. , author Ray, A. et al. ( year 2022 ). title Training language models to follow instructions with h...

    2022

  40. [48]

    author Park, L. A. , & author Simoff, S. ( year 2014 ). title Second order probabilistic models for within-document novelty detection in academic articles . In booktitle Proceedings of the 37th international ACM SIGIR conference on Research & development in information retriev...

    2014

  41. [49]

    ( year 2023 )

    author Robertson, Z. ( year 2023 ). title Gpt4 is slightly helpful for peer-review assistance: A pilot study . journal arXiv preprint arXiv:2307.05492 \/ ,

    2023 arXiv

  42. [50]

    , author Barnett, A

    author Schulz, R. , author Barnett, A. , author Bernard, R. , author Brown, N. J. , author Byrne, J. A. , author Eckmann, P. , author Gazda, M. A. , author Kilicoglu, H. , author Prager, E. M. , author Salholz-Hillel, M. et al. ( year 2022 ). title Is the future of peer review...

    2022

  43. [51]

    author Schumpeter, J. A. ( year 1939 ). title Business cycles: a theoretical, historical and statistical analysis of the capitalist process ,

    1939

  44. [52]

    author Shah, N. B. ( year 2022 a ). title Challenges, experiments, and computational solutions in peer review . journal Communications of the ACM \/ , volume 65 \/ , pages 76--87

    2022

  45. [53]

    author Shah, N. B. ( year 2022 b ). title Improving the peer review process in a scientific manner shows promise . journal COMMUNICATIONS OF THE ACM \/ , volume 65 \/ , pages 75--87

    2022

  46. [54]

    , author Cheng, L

    author Shen, C. , author Cheng, L. , author Zhou, R. , author Bing, L. , author You, Y. , & author Si, L. ( year 2022 ). title MR e D : A meta-review dataset for structure-controllable text generation . In editor S. Muresan , editor P. Nakov , & editor A. Villavicencio (Eds.),...

    2022

  47. [55]

    , & author Wang, J

    author Shibayama, S. , & author Wang, J. ( year 2020 ). title Measuring originality in science . journal Scientometrics \/ , volume 122 \/ , pages 409--427

    2020

  48. [56]

    , author Yin, D

    author Shibayama, S. , author Yin, D. , & author Matsumoto, K. ( year 2021 ). title Measuring novelty in science with word embedding . journal PloS one \/ , volume 16 \/ , pages e0254034

    2021

  49. [57]

    , author Lee, K

    author Siler, K. , author Lee, K. , & author Bero, L. ( year 2015 ). title Measuring the effectiveness of scientific gatekeeping . journal Proceedings of the National Academy of Sciences \/ , volume 112 \/ , pages 360--365

    2015

  50. [58]

    , author Rizos, G

    author Stappen, L. , author Rizos, G. , author Hasan, M. , author Hain, T. , & author Schuller, B. W. ( year 2020 ). title Uncertainty-aware machine support for paper reviewing on the interspeech 2019 submission corpus ,

    2020

  51. [59]

    , author Barry Danfa, J

    author Sun, M. , author Barry Danfa, J. , & author Teplitskiy, M. ( year 2022 ). title Does double-blind peer review reduce bias? evidence from a top computer science conference . journal Journal of the Association for Information Science and Technology \/ , volume 73 \/ , pag...

    2022

  52. [60]

    , & author Bornmann, L

    author Tahamtan, I. , & author Bornmann, L. ( year 2018 ). title Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references? journal Journal of informetrics \/ , volume 12 \/ , pages 906--930

    2018

  53. [61]

    ( year 2023 )

    author Terwiesch, C. ( year 2023 ). title Would chat gpt3 get a wharton mba . journal A prediction based on its performance in the operations management course \/ ,

    2023

  54. [62]

    ( year 2024 )

    author Thelwall, M. ( year 2024 ). title Can chatgpt evaluate research quality? journal Journal of Data and Information Science \/ , volume 9 \/ , pages 1--21 . https://doi.org/10.2478/jdis-2024-0013. :doi:10.2478/jdis-2024-0013

    2024 doi

  55. [63]

    , author Mukherjee, S

    author Uzzi, B. , author Mukherjee, S. , author Stringer, M. , & author Jones, B. ( year 2013 ). title Atypical combinations and scientific impact . journal Science \/ , volume 342 \/ , pages 468--472

    2013

  56. [64]

    , author Yang, N

    author Wang, L. , author Yang, N. , author Huang, X. , author Yang, L. , author Majumder, R. , & author Wei, F. ( year 2023 a ). title Improving text embeddings with large language models . journal arXiv preprint arXiv:2401.00368 \/ ,

    2023 arXiv

  57. [65]

    , author Lin, X

    author Wang, X. , author Lin, X. , & author Shao, B. ( year 2023 b ). title Artificial intelligence changes the way we work: A close look at innovating with chatbots . journal Journal of the Association for Information Science and Technology \/ , volume 74 \/ , pages 339--353

    2023

  58. [66]

    , author Zhang, H

    author Wang, Z. , author Zhang, H. , author Chen, J. , & author Chen, H. ( year 2024 ). title An effective framework for measuring the novelty of scientific articles through integrated topic modeling and cloud model . journal Journal of Informetrics \/ , volume 18 \/ , pages 1...

    2024

  59. [67]

    , author Yuan, S

    author Wei, Y. , author Yuan, S. , author Yang, R. , author Shen, L. , author Li, Z. , author Wang, L. , & author Chen, M. ( year 2023 ). title Tackling modality heterogeneity with multi-view calibration network for multimodal sentiment detection . In editor A. Rogers , editor...

    2023

  60. [68]

    , author Dai, Z

    author Yang, Z. , author Dai, Z. , author Yang, Y. , author Carbonell, J. , author Salakhutdinov, R. R. , & author Le, Q. V. ( year 2019 ). title Xlnet: Generalized autoregressive pretraining for language understanding . In editor H. Wallach , editor H. Larochelle , editor A. ...

    2019

  61. [69]

    , author Wu, Z

    author Yin, D. , author Wu, Z. , author Yokota, K. , author Matsumoto, K. , & author Shibayama, S. ( year 2023 ). title Identify novel elements of knowledge with word embedding . journal Plos one \/ , volume 18 \/ , pages e0284567

    2023

  62. [70]

    , author Liu, P

    author Yuan, W. , author Liu, P. , & author Neubig, G. ( year 2022 ). title Can we automate scientific reviewing? journal Journal of Artificial Intelligence Research \/ , volume 75 \/ , pages 171--212

    2022

  63. [71]

    , author Chen, W

    author Yue, L. , author Chen, W. , author Li, X. , author Zuo, W. , & author Yin, M. ( year 2019 ). title A survey of sentiment analysis in social media . journal Knowledge and Information Systems \/ , volume 60 \/ , pages 617--663

    2019

  64. [72]

    , & author Zhang, C

    author Zhao, Y. , & author Zhang, C. ( year 2025 ). title A review on the novelty measurements of academic papers . journal Scientometrics \/ , volume 130 \/ , pages 727--753 . https://doi.org/10.1007/s11192-025-05234-0. :10.1007/s11192-025-05234-0

    2025 doi

  65. [73]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.