Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A proof-of-concept two-agent GPT system can extract microbial protein data from scientific papers, and both fine-tuning and prompt engineering reliably improve its agreement with ideal answers.

desk verdict Honest proof-of-concept for LLM literature mining in microbial protein, but the cosine-similarity metric is too weak to support the abstract's effectiveness claims. read the letter →

arxiv 2506.20598 v1 pith:4GIZXSVX submitted 2025-06-25 cs.AI cs.SYeess.SY

Alexander D. Kalian , Jaewook Lee , Stefan P. Johannesson , Lennart Otte , Christer Hogstrand , Miao Guo This is my paper
classification cs.AIcs.SYeess.SY
keywords multi-agentAIretrieval-augmentedgenerationGPTfine-tuningpromptengineeringinformationextractionmicrobialproteinsustainableproductionsentenceembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small multi-agent system built on GPT models can automate the search for, and extraction of, key data from scientific literature on microbial protein production. It compares two ways of optimising the information extraction agent: fine-tuning the model on domain-specific examples, and a structured two-stage prompt. On a held-out test set, both methods raised the mean cosine similarity between the agent's output and a curated ideal answer, with prompt engineering reaching at least 0.89 and fine-tuning at least 0.94. The paper concludes that fine-tuning delivers the higher mean score but with larger variance, making prompt engineering a lighter-weight, lower-variance alternative. If this is right, domain-specific LLM optimisation is a practical route to speeding up research into sustainable protein sources.

What carries the argument

The system has two agents. A literature search agent expands a microbial strain name with cultivation-related keywords, queries a public biomedical literature database, scores the relevance of the results, and retrieves full texts. An information extraction agent then reads the PDF text and returns four fields: reported protein % dry mass, trophic mechanism, reported substrate, and substrate class. The central mechanism under test is the optimisation of this second agent. Fine-tuning retrains GPT-4.1 on prompted examples paired with ideal outputs over ten epochs, using an 80/10/10 stratified split with balanced negative examples. Prompt engineering instead uses a two-stage prompt: a context-harvesting stage that enumerates candidate passages, followed by a constrained-generation stage that forces a schema-only answer with a NaN sentinel for missing evidence. Both variants are scored by cosine similarity between embeddings from three pre-trained transformer models of obtained versus ideal outputs, with temperature fixed at 0.0; cosine similarity here measures how close two texts are in embedding space, from 0 to 1.

What would settle it

Re-score the same held-out outputs using per-field exact-match or tolerance-based criteria, numerical protein percentage within reported precision and exact categorical matches for trophic mechanism, substrate, and substrate class, and correlate those scores with the reported cosine similarity values; if high cosine similarity routinely coincides with wrong field values, the claimed effectiveness of both optimisation routes is an artifact of the metric.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-4.1 (2025-04-14), selected as the strongest of the tested models, can be turned into an information extraction agent for microbial protein literature either by fine-tuning or by a structured two-stage prompt, and that both routes are effective. On a held-out test set, fine-tuning raised mean cosine similarity from 0.79 to 0.96, from 0.75 to 0.94, and from 0.91 to 0.98 as judged by three different sentence-transformer embedding models; prompt engineering raised the same metrics from 0.79/0.78/0.91 to 0.92/0.89/0.96. The absolute advantage of fine-tuning over prompt engineering was 0.02 to 0.05 on the three metrics, with statistical overlap, so the authors present the advantage as indicative rather than decisive. Prompt engineering produced lower standard deviations, which the paper interprets as more stable predictions. The evaluation assumed that cosine similarity between whole-output embeddings and ideal output text is a meaningful measure of extraction quality, and the paper itself flags this metric as potentially over-generalised for tasks requiring precise numerical and technical information.

Load-bearing premise

The load-bearing premise is that cosine similarity between whole-output sentence embeddings and the ideal answer text faithfully measures whether the extracted protein percentage, trophic mechanism, substrate, and substrate class are correct; if that metric does not track factual correctness, the comparison between fine-tuning and prompt engineering is unsupported.

Editorial extensions

If this is right

  • A researcher building a domain-specific extraction agent can choose prompt engineering when deployment cost and output variance matter, and fine-tuning when mean agreement with ideal answers is the priority.
  • The same two-step recipe of retrieving relevant papers and then extracting structured fields can be applied to other microbial strains and to additional output fields without changing the architecture.
  • Because both optimisation routes improved performance, the paper's proposed dual-track strategy of prompt design followed by lightweight fine-tuning is a natural next step that should yield further gains.
  • The literature search agent and extraction agent are packaged with a user interface and an automated toxicity-screening module, so the system can be used end-to-end as a proof of concept by non-specialists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: whole-output cosine similarity could hide cases where the model produces the right wording but the wrong number, so a per-field exact-match evaluation might change the reported gap between fine-tuning and prompt engineering.
  • My inference: because the metric rewards phrasing similarity, both optimisation routes may be matching the style of the ideal answers rather than the underlying biological facts; testing on newly published papers with different wording would clarify this.
  • My inference: the same two-agent architecture with a similarly constructed benchmark could transfer to other structured scientific extractions, such as chemical safety properties or process parameters, since nothing in the method is specific to protein data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript presents a proof-of-concept multi-agent AI system for sustainable microbial protein production research, consisting of a PubMed-based literature search agent and an information extraction agent built on a GPT model. The core experimental contribution is a comparison of two optimization strategies for the extraction agent: fine-tuning on a curated dataset derived from the Piercy et al. (2023) supplementary table, and a two-stage prompt-engineering design. Performance is measured as cosine similarity between sentence-transformer embeddings of the model output and curated 'ideal' strings, evaluated with three independent embedding models. The authors report that both methods improve mean cosine similarity (fine-tuning consistently ≥0.94, prompt engineering ≥0.89), with fine-tuning yielding higher means but larger variance. The paper also describes a Streamlit user interface and a BioCyc-based toxicity screening module.

Significance. If the central claim is accepted, the paper provides a useful early-stage demonstration that domain-specific optimization of a GPT-based extraction agent is feasible for extracting structured microbial-protein data from scientific literature. Concrete strengths include the use of three independent embedding models with consistent trends, strain-level stratified train/validation/test splits, balanced positive and negative examples, and an attempt to ensure methodological parity between fine-tuning and prompt engineering. The stated availability of code and data via GitHub is also a positive feature. However, the validity of the headline conclusion depends entirely on whether the cosine-similarity metric tracks the correctness of the extracted protein percentage, trophic mechanism, substrate, and substrate class; the paper itself concedes in Section 3.5 that the metric is 'potentially over-generalised and superficial' for tasks requiring precise numerical and technical information. For these reasons the significance of the empirical comparison is currently conditional on a metric-validation exercise.

major comments (3)
  1. [Section 2.4.1, Fig. 4, Section 3.5] The load-bearing evaluation metric is cosine similarity between whole-output sentence embeddings and the ideal string. Because the ideal output concatenates four fields, an output that is correct on three fields but wrong on the protein percentage (e.g., off by 10 wt%) can still receive a high cosine similarity, and outputs that merely follow the requested template without extracting the correct values may be scored favorably. The paper itself acknowledges in Section 3.5 that this metric is 'potentially over-generalised and superficial' for tasks requiring precise numerical and technical information. Since the abstract's headline numbers (fine-tuning ≥0.94, prompt engineering ≥0.89) and the conclusion that both approaches are 'highly effective' rest on this metric, the current evidence supports a claim about semantic similarity to ideal text, not a claim about extraction accuracy. Please add a field-level evaluation: parse each of the four fields separately, compute numeric error for protein % dry mass, and report exact-match accuracy or a confusion-matrix-style measure for the categorical fields (trophic mechanism, substrate, substrate class).
  2. [Section 3.4, Fig. 8, Abstract] The claimed advantage of fine-tuning over prompt engineering is reported as absolute differences of 0.04, 0.05, and 0.02 on the three embeddings, but the standard deviations overlap substantially (e.g., 0.96 ± 0.13 vs. 0.92 ± 0.075 for all-mpnet-base-v2). The text itself calls this 'an indicative finding only,' which is appropriate. The abstract and conclusions, however, state that fine-tuning improved scores 'to a greater extent' and 'consistently of ≥0.94' without carrying this statistical caveat forward. Please report confidence intervals or a paired significance test for the fine-tuning versus prompt-engineering comparison, and restrict the abstract/conclusion wording to what the statistics support.
  3. [Section 2.2 and Section 2.4.1] All negative examples are scored against a single fixed ideal sentence ('The literature provided does not contain the requested information...'). Consequently, the mean cosine similarity on negative cases largely measures whether the model reproduces that exact phrasing, not whether it correctly abstains from extraction. Because negative and positive cases are pooled in the reported aggregate means, the headline scores conflate extraction accuracy with template adherence. Please report positive-case and negative-case performance separately, or define a metric that treats any appropriate abstention as correct regardless of phrasing.
minor comments (5)
  1. [Section 2.2] The number of positive and negative examples, and the number of microbial strains represented, are not stated anywhere in the text; please report dataset sizes so the reader can gauge the scale of the fine-tuning and test sets.
  2. [Section 3.3 and Fig. 6/Fig. 8] The baseline cosine similarities for the prompt-engineering comparison (0.79, 0.78, 0.91 in Section 3.3) are not obviously the same as the baseline values shown for GPT-4.1 in Fig. 6, where all-mpnet-base-v2 scores appear to range between 0.7 and 0.85. Clarify whether Fig. 6 uses the full curated dataset and Fig. 8 uses only the test partition, and explain any resulting differences.
  3. [Conclusions] The statement that the study 'demonstrated effective creation of a research paper obtainment agent' is not supported by any quantitative evaluation of the literature search agent; the paper evaluates only the information extraction agent. Please soften this claim or add retrieval metrics.
  4. [Section 2.2] The PDF parsing tool is referred to as 'PDFMiner' in the text but as 'pdfminer.six' in reference [40]; please use one consistent name.
  5. [Supplementary Information] The GitHub repository link is described with 'shall be made available'; if the code and data are accessible at the time of publication, please state that explicitly and provide the working link.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the fine-tuning and prompt-engineering comparisons are empirical held-out evaluations, with only a minor non-load-bearing self-citation for the benchmark data.

full rationale

The central claim, that fine-tuning and prompt engineering improve the information extraction agent's cosine similarity to ideal outputs, rests on an empirical comparison against fixed reference strings. Fine-tuning uses an 80/10/10 stratified split with a reserved external testing set, and the reported scores (0.94 and above for fine-tuning, 0.89 and above for prompt engineering) are measured on that held-out set, not on training outputs. Prompt engineering is evaluated on the same partitions and with the same three sentence-embedding models, so the comparison is symmetric. The cosine-similarity metric is a validity concern rather than a circularity concern: Section 3.5 itself concedes that it may be 'a potentially over-generalised and superficial performance metric' for tasks requiring precise numerical and technical information, but this does not make the measured scores equal to the model's inputs by construction. The only self-citation is the use of the Piercy et al. (2023) supplementary table as the source of ideal outputs; although one co-author of the present paper (Hogstrand) is also an author of Piercy et al., that table is an externally published, pre-existing dataset and not a result derived from the present models. No step in the paper's derivation chain reduces to its own input by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the Piercy et al. (2023) curated dataset as ground truth, on the validity of the cosine-similarity evaluation, and on the coverage of PubMed/PMC for the search agent. The paper's chosen hyperparameters, temperature 0.0, 10 fine-tuning epochs, and balanced negative examples, influence the results but are not systematically optimized. No new physical or conceptual entities are introduced.

free parameters (3)
  • GPT temperature = 0.0
    Set to 0.0 for all final evaluations after a scan from 0.0 to 0.5 showed no significant effect; this setting controls output determinism.
  • Fine-tuning epochs = 10
    Chosen for the fine-tuning run; performance was monitored at checkpoints over the final 3 epochs, but no systematic epoch optimization is reported.
  • Number of negative examples = equal to positive examples
    The authors balanced negative and positive cases across four hand-defined categories to prevent overfitting; this balance is a design choice that affects measured precision and recall.
assumptions (3)
  • domain assumption The Piercy et al. (2023) supplementary table provides correct and complete ground-truth values for protein % dry mass, trophic mechanism, substrate, and substrate class.
    All ideal outputs for training and testing are derived from this table (Sections 2.2 and 2.4.1). Errors or omissions in this table directly bias the evaluation.
  • domain assumption Cosine similarity between sentence-transformer embeddings of whole outputs is a valid proxy for extraction correctness.
    This metric is the sole quantitative measure of agent performance (Section 2.4.1). The paper itself flags in Section 3.5 that it may be a superficial performance metric for precise numerical information.
  • domain assumption The PubMed/PMC open-access subset contains enough relevant literature for the search agent to retrieve useful papers.
    The literature search agent relies on PubMed API and PMC open-access full text (Section 2.3), but retrieval precision and recall are not evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges." pith.science (2026). https://pith.science/paper/4GIZXSVX

@misc{pith2026250620598,
  author       = {Pith},
  title        = {Pith review of: Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GIZXSVX}},
  note         = {Machine review of arXiv:2506.20598}
}
abstract

The global demand for sustainable protein sources has accelerated the need for intelligent tools that can rapidly process and synthesise domain-specific scientific knowledge. In this study, we present a proof-of-concept multi-agent Artificial Intelligence (AI) framework designed to support sustainable protein production research, with an initial focus on microbial protein sources. Our Retrieval-Augmented Generation (RAG)-oriented system consists of two GPT-based LLM agents: (1) a literature search agent that retrieves relevant scientific literature on microbial protein production for a specified microbial strain, and (2) an information extraction agent that processes the retrieved content to extract relevant biological and chemical information. Two parallel methodologies, fine-tuning and prompt engineering, were explored for agent optimisation. Both methods demonstrated effectiveness at improving the performance of the information extraction agent in terms of transformer-based cosine similarity scores between obtained and ideal outputs. Mean cosine similarity scores were increased by up to 25%, while universally reaching mean scores of $\geq 0.89$ against ideal output text. Fine-tuning overall improved the mean scores to a greater extent (consistently of $\geq 0.94$) compared to prompt engineering, although lower statistical uncertainties were observed with the latter approach. A user interface was developed and published for enabling the use of the multi-agent AI system, alongside preliminary exploration of additional chemical safety-based search capabilities

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 32 canonical work pages

  1. [2]

    and Guo, M., 2024

    - Banks, M., Taylor, M. and Guo, M., 2024. High throughput parameter estimation and uncertainty analysis applied to the production of mycoprotein from synthetic lignocellulosic hydrolysates. Current Research in Food Science, 9, p.100908

    work page 2024

  2. [3]

    and Narasagoudr, S.S., 2024

    - Dalbanjan, N.P., Eelager, M.P. and Narasagoudr, S.S., 2024. Microbial protein sources: A comprehensive review on the potential usage of fungi and cyanobacteria in sustainable food systems. Food and Humanity, p.100366

    work page 2024

  3. [4]

    and O'leary, J.A., 1976

    - Henry, D.P., Thomson, R.H., Sizemore, D.J. and O'leary, J.A., 1976. Study of Candida ingens grown on the supernatant derived from the anaerobic fermentation of monogastric animal wastes. Applied and environmental microbiology, 31(6), pp.813-818

    work page 1976

  4. [5]

    and Sambrook, I.E., 1977

    - Braude, R., Hosking, Z.D., Mitchell, K.G., Plonka, S. and Sambrook, I.E., 1977. Pruteen, a new source of protein for growing pigs. I. Metabolic experiment: utilization of nitrogen. Livestock Production Science, 4(1), pp.79-89

    work page 1977

  5. [6]

    and Dyer, P.S., 2020

    - Whittaker, J.A., Johnson, R.I., Finnigan, T.J., Avery, S.V. and Dyer, P.S., 2020. The biotechnology of quorn mycoprotein: past, present and future challenges. Grand challenges in fungal biotechnology, pp.59-79

    work page 2020

  6. [7]

    and Wall, B.T., 2021

    - Monteyne, A.J., Dunlop, M.V., Machin, D.J., Coelho, M.O., Pavis, G.F., Porter, C., Murton, A.J., Abdelrahman, D.R., Dirks, M.L., Stephens, F.B. and Wall, B.T., 2021. A mycoprotein-based high-protein vegan diet supports equivalent daily myofibrillar protein synthesis rates compared with an isonitrogenous omnivorous diet in older adults: a randomised cont...

    work page 2021

  7. [8]

    and Freedman, M.R., 2019

    - Finnigan, T.J., Wall, B.T., Wilde, P.J., Stephens, F.B., Taylor, S.L. and Freedman, M.R., 2019. Mycoprotein: the future of nutritious nonmeat protein, a symposium review. Current developments in nutrition, 3(6), p.nzz021

    work page 2019

  8. [9]

    and Ugbogu, O.C., 2016

    - Ugbogu, E.A. and Ugbogu, O.C., 2016. A review of microbial protein production: prospects and challenges. FUW Trends in Science and Technology Journal, 1(1), pp.182-185. 21

    work page 2016

Show all 62 references
  1. [10]

    and Weiss, G., 1999

    - Ferber, J. and Weiss, G., 1999. Multi-agent systems: an introduction to distributed artificial intelligence (Vol. 1). Reading: Addison-wesley

    1999

  2. [11]

    and Khoei, T.T., 2025

    - Singh, A., Ehtesham, A., Kumar, S. and Khoei, T.T., 2025. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. arXiv preprint arXiv:2501.09136

    2025 arXiv

  3. [12]

    - Lee, D., Lee, J., & Shin, D. (2024). GPT Prompt Engineering for a Large Language Model-Based Process Improvement Generation System. Korean Journal of Chemical Engineering, 41(12), 3263-3286

    2024

  4. [13]

    and Price, N.D., 2014

    - Richards, M.A., Cassen, V., Heavner, B.D., Ajami, N.E., Herrmann, A., Simeonidis, E. and Price, N.D., 2014. MediaDB: a database of microbial growth conditions in defined media. PLoS One, 9(8), p.e103548

    2014

  5. [14]

    GPT-4 Technical Report,

    - OpenAI et al., “GPT-4 Technical Report,” Mar. 04, 2024, arXiv: arXiv:2303.08774. doi: 10.48550/arXiv.2303.08774

    2024 doi arXiv

  6. [15]

    LLaMA: Open and Efficient Foundation Language Models,

    - H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” Feb. 27, 2023, arXiv: arXiv:2302.13971. doi: 10.48550/arXiv.2302.13971

    2023 doi arXiv

  7. [16]

    Gemini: A Family of Highly Capable Multimodal Models,

    - G. Team et al., “Gemini: A Family of Highly Capable Multimodal Models,” May 09, 2025, arXiv: arXiv:2312.11805. doi: 10.48550/arXiv.2312.11805

    2025 doi arXiv

  8. [17]

    Scaling Laws for Neural Language Models,

    - J. Kaplan et al., “Scaling Laws for Neural Language Models,” Jan. 23, 2020, arXiv: arXiv:2001.08361. doi: 10.48550/arXiv.2001.08361

    2020 doi arXiv

  9. [18]

    Language Models are Few-Shot Learners,

    - T. B. Brown et al., “Language Models are Few-Shot Learners,” Jul. 22, 2020, arXiv: arXiv:2005.14165. doi: 10.48550/arXiv.2005.14165

    2020 doi arXiv

  10. [19]

    Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models,

    - M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models,” Jun. 10, 2024, arXiv:

    2024

  11. [20]

    A Survey on Large Language Models for Code Generation,

    - J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A Survey on Large Language Models for Code Generation,” Nov. 10, 2024, arXiv: arXiv:2406.00515. doi: 10.48550/arXiv.2406.00515

    2024 doi arXiv

  12. [21]

    Large Language Models for Robotics: Opportunities, Challenges, and Perspectives,

    - J. Wang et al., “Large Language Models for Robotics: Opportunities, Challenges, and Perspectives,” Jan. 09, 2024, arXiv: arXiv:2401.04334. doi: 10.48550/arXiv.2401.04334

    2024 doi arXiv

  13. [22]

    The Rise and Potential of Large Language Model Based Agents: A Survey,

    - Z. Xi et al., “The Rise and Potential of Large Language Model Based Agents: A Survey,” Sep. 19, 2023, arXiv: arXiv:2309.07864. doi: 10.48550/arXiv.2309.07864

    2023 doi arXiv

  14. [23]

    What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks,

    - T. Guo et al., “What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks,” Dec. 28, 2023, arXiv: arXiv:2305.18365. doi: 10.48550/arXiv.2305.18365. 22

    2023 doi arXiv

  15. [24]

    Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation,

    - Z. Liang, W. Yu, T. Rajpurohit, P. Clark, X. Zhang, and A. Kaylan, “Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation,” May 22, 2023, arXiv: arXiv:2305.14386. doi: 10.48550/arXiv.2305.14386

    2023 doi arXiv

  16. [25]

    KoLA: Carefully Benchmarking World Knowledge of Large Language Models,

    - J. Yu et al., “KoLA: Carefully Benchmarking World Knowledge of Large Language Models,” Jul. 01, 2024, arXiv: arXiv:2306.09296. doi: 10.48550/arXiv.2306.09296

    2024 doi arXiv

  17. [26]

    AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors,

    - W. Chen et al., “AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors,” Oct. 23, 2023, arXiv: arXiv:2308.10848. doi: 10.48550/arXiv.2308.10848

    2023 doi arXiv

  18. [27]

    Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems,

    - B. Yan et al., “Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems,” Jun. 19, 2025, arXiv: arXiv:2502.14321. doi: 10.48550/arXiv.2502.14321

    2025 doi arXiv

  19. [28]

    A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges,

    - X. Li, S. Wang, S. Zeng, Y. Wu, and Y. Yang, “A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges,” Vicinagearth, vol. 1, no. 1, p. 9, Oct. 2024, doi: 10.1007/s44336-024-00009-2

    2024 doi

  20. [29]

    Large Language Models are Zero-Shot Reasoners,

    - T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” Jan. 29, 2023, arXiv: arXiv:2205.11916. doi: 10.48550/arXiv.2205.11916

    2023 doi arXiv

  21. [30]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

    - J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” Jan. 10, 2023, arXiv: arXiv:2201.11903. doi: 10.48550/arXiv.2201.11903

    2023 doi arXiv

  22. [31]

    LLM4Rec: A Comprehensive Survey on the Integration of Large Language Models in Recommender Systems—Approaches, Applications and Challenges,

    - S. Shehmir and R. Kashef, “LLM4Rec: A Comprehensive Survey on the Integration of Large Language Models in Recommender Systems—Approaches, Applications and Challenges,” Future Internet, vol. 17, no. 6, Art. no. 6, Jun. 2025, doi: 10.3390/fi17060252

    2025 doi

  23. [32]

    Improving Language Understanding by Generative Pre-Training,

    - A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” 2018

    2018

  24. [33]

    The Claude 3 Model Family: Opus, Sonnet, Haiku,

    - Anthropic, “The Claude 3 Model Family: Opus, Sonnet, Haiku,” 2024. Accessed: Jun. 23, 2025. [Online]. Available: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_ Claude_3.pdf

    2024

  25. [34]

    Closing the gap between open-source and commercial large language models for medical evidence summarization,

    - G. Zhang et al., “Closing the gap between open-source and commercial large language models for medical evidence summarization,” Jul. 25, 2024, arXiv: arXiv:2408.00588. doi: 10.48550/arXiv.2408.00588

    2024 doi arXiv

  26. [35]

    Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting,

    - Z. A. Nazi, Md. R. Hossain, and F. A. Mamun, “Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting,” Nat. Lang. Process. J., vol. 10, p. 100124, Mar. 2025, doi: 10.1016/j.nlp.2024.100124. 23

    2025

  27. [36]

    and Huang, K., 2025

    - Wu, X.K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., Meng, Q. and Huang, K., 2025. Llm fine-tuning: Concepts, opportunities, and challenges. Big Data and Cognitive Computing, 9(4), p.87

    2025

  28. [37]

    and Schmidt, D.C., 2023

    - White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J. and Schmidt, D.C., 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382

    2023 arXiv

  29. [38]

    PubMed Central: The GenBank of the published literature

    - Roberts, R.J., 2001. PubMed Central: The GenBank of the published literature. Proceedings of the National Academy of Sciences, 98(2), pp.381-382

    2001

  30. [39]

    and Li, K., 2019

    - He, J. and Li, K., 2019. How comprehensive is the PubMed Central Open Access full-text database?. IConference 2019 Proceedings

    2019

  31. [40]

    Available at: https://github.com/pdfminer/pdfminer.six (Accessed: 25 June 2025)

    - pdfminer.six (2025) pdfminer.six [online]. Available at: https://github.com/pdfminer/pdfminer.six (Accessed: 25 June 2025)

    2025

  32. [41]

    and Zhang, Z., 2024

    - Li, J., Yuan, Y. and Zhang, Z., 2024. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv preprint arXiv:2403.10446

    2024 arXiv

  33. [42]

    and Hashemi, M., 2024

    - Madhusudhan, N., Madhusudhan, S.T., Yadav, V. and Hashemi, M., 2024. Do llms know when to not answer? investigating abstention abilities of large language models. arXiv preprint arXiv:2407.16221

    2024 arXiv

  34. [43]

    and Zhou, J., 2023

    - Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., Shen, Y. and Zhou, J., 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420

    2023 arXiv

  35. [44]

    and Mądry, A., 2024

    - Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A.J., Welihinda, A., Hayes, A., Radford, A. and Mądry, A., 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  36. [45]

    Available at: https://openai.com/index/gpt-4-1/ (Accessed: 25 June 2025)

    - OpenAI (2025) Introducing GPT ‑ 4.1 and GPT ‑ 4.1 Mini/Nano [online]. Available at: https://openai.com/index/gpt-4-1/ (Accessed: 25 June 2025)

    2025

  37. [46]

    a helpful assistant

    - Zheng, M., Pei, J. and Jurgens, D., 2023. Is" a helpful assistant" the best role for large language models? a systematic evaluation of social roles in system prompts. arXiv preprint arXiv:2311.10054, 8

    2023 arXiv

  38. [47]

    and Jurgens, D., 2024, November

    - Zheng, M., Pei, J., Logeswaran, L., Lee, M. and Jurgens, D., 2024, November. When” A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP...

    2024

  39. [48]

    and Sakr, M., 2024, March

    - Agarwal, A., Mittal, K., Doyle, A., Sridhar, P., Wan, Z., Doughty, J.A., Savelka, J. and Sakr, M., 2024, March. Understanding the Role of Temperature in Diverse Question Generation by GPT-4. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. ...

    2024

  40. [49]

    and Hoque, E., 2020, May

    - Laskar, M.T.R., Huang, X. and Hoque, E., 2020, May. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In Proceedings of the twelfth language resources and evaluation conference (pp. 5505-5514)

    2020

  41. [50]

    and Gurevych, I., 2019

    - Reimers, N. and Gurevych, I., 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  42. [51]

    A survey of prompt engineering methods in large language models for different nlp tasks

    - Vatsal, Shubham, and Harsh Dubey. "A survey of prompt engineering methods in large language models for different nlp tasks." arXiv preprint arXiv:2407.12994 (2024)

    2024 arXiv

  43. [52]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    - Liu, Pengfei, et al. "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing." ACM computing surveys 55.9 (2023): 1-35

    2023

  44. [53]

    Summarization for Generative Relation Extraction in the Microbiome Domain

    - Khettari, Oumaima El, Solen Quiniou, and Samuel Chaffron. "Summarization for Generative Relation Extraction in the Microbiome Domain." arXiv preprint arXiv:2506.08647 (2025)

    2025 arXiv

  45. [54]

    A Study of Biomedical Relation Extraction Using GPT Models

    - Zhang, Jeffrey, et al. "A Study of Biomedical Relation Extraction Using GPT Models." AMIA Summits on Translational Science Proceedings 2024 (2024): 391

    2024

  46. [55]

    Learning to route llms with confidence tokens

    - Chuang, Yu-Neng, et al. "Learning to route llms with confidence tokens." arXiv preprint arXiv:2410.13284 3 (2025)

    2025 arXiv

  47. [56]

    and Fernández, J.H., 2022

    - Khorasani, M., Abdou, M. and Fernández, J.H., 2022. Web application development with streamlit. Software Development, pp.498-507

    2022

  48. [57]

    - Kanehisa, M., & Goto, S. (2000). KEGG: Kyoto Encyclopedia of Genes and Genomes. Nucleic Acids Research, 28(1), 27–30

    2000

  49. [58]

    and Ong, W.K., 2019

    - Karp, P.D., Billington, R., Caspi, R., Fulcher, C.A., Latendresse, M., Kothari, A., Keseler, I.M., Krummenacker, M., Midford, P.E., Ong, Q. and Ong, W.K., 2019. The BioCyc collection of microbial genomes and metabolic pathways. Briefings in bioinformatics, 20(4), pp.1085-1093

    2019

  50. [59]

    and Petryszak, R., 2024

    - Milacic, M., Beavers, D., Conley, P., Gong, C., Gillespie, M., Griss, J., Haw, R., Jassal, B., Matthews, L., May, B. and Petryszak, R., 2024. The reactome pathway knowledgebase 2024. Nucleic acids research, 52(D1), pp.D672-D678. 25

    2024

  51. [60]

    and Muller, K.R., 2009

    - Hansen, K., Mika, S., Schroeter, T., Sutter, A., Ter Laak, A., Steger-Hartmann, T., Heinrich, N. and Muller, K.R., 2009. Benchmark data set for in silico prediction of Ames mutagenicity. Journal of chemical information and modeling, 49(9), pp.2077-2081

    2009

  52. [61]

    and Kakade, S., 2023

    - Zhang, H., Zhang, Y.F., Yu, Y., Madeka, D., Foster, D., Xing, E., Lakkaraju, H. and Kakade, S., 2023. A study on the calibration of in-context learning. arXiv preprint arXiv:2312.04021

    2023 arXiv

  53. [62]

    and Bikel, D., 2023

    - Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. and Bikel, D., 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  54. [63]

    and Dai, D., 2024

    - Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C. and Dai, D., 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Supplementary Information Python scripts, other software developed and data used in this study, sha...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.