Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs

Bianca Raimondi; Maurizio Gabbrielli; Saverio Giallorenzo

REVIEW 5 major objections 6 minor 29 references

Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small fine-tuned LLaMA-2 models match a 70B model on course MCQs, at a fraction of the hardware cost.

desk verdict Useful dataset and thorough fine-tuning sweep, but the lenient scoring rule and abstract's overclaim make the central result unproven. read the letter →

arxiv 2501.05891 v2 pith:XCYTYQCC submitted 2025-01-10 cs.CL cs.AI

Bianca Raimondi , Saverio Giallorenzo , Maurizio Gabbrielli This is my paper · ORCID

classification cs.CLcs.AI

keywords LargeLanguageModelsmultiple-choicequestionsfine-tuningquantisationLLaMA-2programminglanguageseducationaffordableLLMsLoRA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether educators and students can affordably use LLMs to answer course-specific multiple-choice questions, and answers yes: fine-tuning small open-source LLaMA-2 variants (7B and 13B) on the course textbook makes them about as accurate as the much larger 70B pre-trained model on a 162-question Programming Languages MCQ test. The fine-tuned 13B quantised variant, in particular, matches the 70B's 59% accuracy while using only about 21GB of GPU memory for fine-tuning, within reach of consumer-grade hardware. The paper also contributes the MCQ dataset and a systematic study of how fine-tuning hyperparameters and textbook material selection affect accuracy. The central consequence is that domain-specific fine-tuning can substitute for model scale in this educational task, making LLM-based assessment tools materially affordable.

What carries the argument

The mechanism is parameter-efficient fine-tuning with LoRA and QLoRA, applied to LLaMA-2 7B and 13B variants using a dataset derived from the course textbook (Gabbrielli and Martini), where each paragraph becomes an instruction–output training sample. The MCQ evaluation dataset of 162 questions, with topic labels aligned to the textbook chapters, lets the authors separate questions covered by the fine-tuning material from those outside it, serving as a litmus test for catastrophic forgetting and for whether fine-tuning teaches general domain vocabulary rather than rote answers. The inference prompt is a fixed template asking the model to output the correct alternatives, scored under the rule that providing at least one correct answer counts as correct.

What would settle it

Re-run the evaluation using a stricter rule that requires all correct alternatives to be present (and no incorrect ones) for the 19 multi-answer questions, then compare the accuracy rankings; if the fine-tuned small models no longer match the 70B baseline, the central claim would be substantially weakened.

Watch

Extended reading notes

Core claim

The paper's central claim is that smaller, textbook-based fine-tuned models outperform generic larger ones when answering course-specific MCQs. Concretely, on a 162-question undergraduate Programming Languages MCQ set, the pre-trained 70B quantised LLaMA-2 reaches 59% accuracy, while fine-tuned variants of the 13B quantised model—trained with QLoRA on paragraphs from the course textbook—reach comparable or higher accuracy. Among the 240 fine-tuned 13B quantised configurations, 78% beat the pre-trained 13B base, and the best of them track the 70B baseline. The authors further show that the choice of textbook chapters used for fine-tuning is the single most influential hyperparameter, and that quantised fine-tuning (QLoRA) preserves accuracy while cutting memory requirements enough to run on consumer GPUs.

Load-bearing premise

The evaluation counts a response as correct if it contains at least one of the listed correct answers, so a model that includes the right letter among several guesses gets credit; on the 19 multi-answer questions this can inflate accuracy and possibly change which model wins.

Editorial extensions

If this is right

If the claim holds, educational institutions can deploy course-specific MCQ answering assistants on affordable consumer GPUs instead of relying on large proprietary models.
The result implies that textbook material, which educators already have, is sufficient fine-tuning data for at least this STEM domain, reducing the data-engineering barrier.
The finding that a single well-chosen textbook chapter yields stable accuracy gains suggests that targeted fine-tuning can be more effective than broad fine-tuning on a full book.
Quantised fine-tuning (QLoRA) is shown to be effective enough to nearly close the gap between small quantised models and larger unquantised ones, lowering the entry cost further.
The public MCQ dataset provides a reproducible benchmark for future work on LLM accuracy in programming-languages education.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's evidence suggests a general recipe for other technical courses: fine-tune a 7B or 13B open model on the assigned textbook, then evaluate on past exam MCQs, to obtain an in-domain assistant at consumer hardware cost.
The scoring rule that rewards any one correct alternative may systematically inflate accuracy, especially for the 19 multi-answer questions; a stricter all-correct rule could change the ranking between small fine-tuned models and the 70B baseline.
The observed catastrophic forgetting in roughly 15% of fine-tuned 7B variants points to a practical safety check: institutions should validate fine-tuned models on questions outside the training material before deployment.
Because the paper focuses on a single course and a single model family, the transferability of 'textbook fine-tuning beats scale' to other subjects, question formats, and newer open models remains an open question worth testing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The new thing here is a public dataset of 162 programming-languages MCQs plus a systematic comparison of LoRA/qLoRA fine-tuned LLaMA-2 7B/13B models against the pre-trained 70B on that dataset. They actually run 720 fine-tuned variants, release code and data, and report memory footprints. That is reproducible empirical work, and the resource-constrained framing is legitimate: a 13B quantized model that matches a 70B on course-specific questions is interesting for educators.

The soft spots are real and load-bearing. The scoring rule accepts an answer if it contains at least one correct letter, and the prompt literally tells the model the question has multiple correct answers. So a single-answer question keyed 'a' is scored correct for the output 'a, b, c, d'. The paper gives no statistics on how many letters each model emits, and no robustness check with a strict all-correct rule. If the fine-tuned small models are more verbose than the 70B baseline, their reported accuracy advantage could be a scoring artifact. That is the first thing I would check.

Second, the abstract says 'outperform generic larger ones,' but Section 3 says 'some fine-tuned alternatives can achieve the same accuracy.' Those are different claims. There are no confidence intervals or significance tests; the best-of-720 selection is presented as evidence. The provenance overlap (fine-tuning on a textbook co-authored by a paper author, with test items from a course using it) is a fairness concern, though the PL partition not covered by the textbook gives some independent support—under the same lenient scoring.

The paper is not sloppy in its experiments; the hyperparameter analysis is thorough, and the limitations section covers scope. It just doesn't address the evaluation-metric problem, which is central.

Who is this for? People building local LLM assistants for education, and anyone working on MCQ evaluation methodology. As a referee, I would ask for a strict re-scoring, an answer-length table, and a corrected abstract before accepting. The dataset and fine-tuning comparison deserve a serious look, so I'd send it to review, not desk reject.

Referee Report

5 major / 6 minor

Summary. The paper investigates whether fine-tuning smaller LLaMA-2 variants (7B and 13B) on a course-specific textbook can make them competitive with, or even superior to, the much larger pre-trained 70B model for answering multiple-choice questions (MCQs) in Programming Languages. The authors introduce a new public dataset of 162 MCQs, run inference on pre-trained and quantized models, fine-tune 720 variants with LoRA/QLoRA using different textbook partitions and hyperparameters, and analyze the effects of quantization, learning rate, batch size, and epochs on accuracy. The abstract claims that smaller textbook-based fine-tuned models outperform generic larger ones, while the body reports more cautiously that some fine-tuned alternatives can achieve the same accuracy as the larger pre-trained versions.

Significance. If the findings are robust, the paper makes a useful contribution to the emerging literature on domain-specific fine-tuning for education, showing that affordable consumer-grade GPUs can support models that approach the accuracy of much larger general-purpose LLMs on a specialized MCQ task. The public dataset and the broad hyperparameter sweep (720 fine-tuned models) are valuable assets for the community. However, the central comparison is currently threatened by a permissive scoring rule, a potential overlap between fine-tuning data and evaluation questions, and the absence of statistical significance testing, so the headline claim as stated is not yet supported.

major comments (5)

[Section 2, MCQ Dataset and Inference with Prompt] The evaluation rule states: "To check the LLM's answer, it must provide at least one of these correct answers." For the 143 single-answer MCQs, a response that contains the correct letter along with any number of incorrect letters is scored correct, and for the 19 multi-answer MCQs one correct letter suffices even if other letters are wrong. The prompt also explicitly tells the model that "The question is a multiple-choice question with multiple correct answers," which plausibly encourages multi-letter outputs. All reported accuracies (Table 1, Table 5, Figs. 3–6) are computed under this rule. If fine-tuned small models tend to emit more letters per response than the pre-trained 70B baseline, their accuracy is inflated relative to the larger model, so the central claim that small fine-tuned models match 70B could be an artifact of the scoring rule. The paper provides no answer-length statistics and no robustness check under a stricter rule that requires all correct answers to be present. This is a load-bearing issue because the main comparison in the abstract and Section 4 rests on these accuracy numbers.
[Abstract, Section 3, Section 4] The abstract claims that "smaller textbook-based fine-tuned models outperform generic larger ones," but Section 3 states only that "some fine-tuned alternatives can achieve the same accuracy as the larger/base versions" and Section 4 says the smaller models achieve "an accuracy that is similar to the larger pre-trained ones." Table 5 reports the percentage of fine-tuned variants that beat their own pre-trained version (e.g., 78% for 13B quantized, whose pre-trained baseline is 45%), not the percentage that exceed the 70B quantized model's 59% accuracy on the full dataset. The paper never directly reports the accuracy of the best (or median) fine-tuned model against the 70Bq baseline, so the "outperform" wording in the abstract is unsupported by the presented evidence. The authors should either provide direct comparisons with confidence intervals or temper the abstract to match the body's more cautious phrasing.
[Section 2, MCQ Dataset and On Fine-Tuning] The fine-tuning material is the textbook "Programming Languages: Principles and Paradigms" by Gabbrielli and Martini, and one of the co-authors of this paper (Gabbrielli) is also a co-author of that textbook. In addition, 15% of the MCQs come from an undergraduate PL course that likely uses this textbook (the paper describes the textbook as "widely adopted" and the course is at the same institution). This creates a risk of data leakage or privileged access: the fine-tuned models may benefit from memorizing textbook content that directly overlaps with the exam questions, inflating their apparent advantage over a generic 70B model. The "Threats to Validity" paragraph does not acknowledge this overlap or the authorial relationship. The authors should disclose this conflict, quantify the overlap between the fine-tuning data and the evaluation questions, and report the central comparison on the subset of questions that do not overlap with the textbook (e.g., the PL partition) as the primary evidence rather than as a secondary analysis.
[Section 3, Tables 1 and 5 and Figures 3–6] The reported accuracies are point estimates on a single test set of 162 questions, with no confidence intervals, bootstrap estimates, or significance tests. Differences such as 13B 50% vs. 13Bq 45% in Table 1 are within the sampling noise of a 162-item test (standard error of roughly 4 percentage points). The claim that "some fine-tuned alternatives can achieve the same accuracy" as 70Bq requires a statistical non-inferiority test or at least a bootstrap interval; otherwise the main comparison is not supported. The authors should add these analyses or explicitly label the findings as exploratory and refrain from making strong comparative claims without them.
[Section 2, Inference with Prompt] The fixed prompt includes the sentence "The question is a multiple-choice question with multiple correct answers" for every question, including the 143 questions that have a single correct answer. This instruction is factually incorrect for the majority of the dataset and likely biases models toward producing multi-letter answers, which interacts with the permissive scoring rule discussed above. The authors should either tailor the prompt to the actual number of correct answers per question or perform a sensitivity analysis to determine how much this instruction affects the results. Without such an analysis, the prompt design itself is a confound in the comparison between fine-tuned and pre-trained models.

minor comments (6)

[Section 2, Threats to Validity] The heading "Threats to Validity" is misspelled as "Treats to Validity" in the manuscript; consider correcting the typo and moving this paragraph to the Limitations section for better visibility.
[Figure 3] The violin plots do not show the number of fine-tuned models at each accuracy level, and the density scale is not labeled. Consider adding a legend that identifies the horizontal lines (pre-trained accuracies) and reporting the sample size per violin.
[Table 2] The memory consumption values for fine-tuning 13B base and 70B base are listed as ">80GB" without a measured peak; consider reporting actual measurements or a more precise upper bound, since the exact memory footprint is relevant to the affordability claims.
[Section 3.2.1] The statement that "only one chapter of the book can generally lead to more stable results" is not quantified; consider reporting the interquartile range or variance of accuracies for each fine-tuning dataset partition.
[Section 3.3] The mapping of the fine-tuning dataset hyperparameter (1 = whole book, 2 = three chapters, 3 = single chapter) is defined only in the text; consider adding it to the figure caption or to Table 4 so that Figure 8 is self-contained.
[References] Reference [9] is for Mixtral, but the text in Section 2 compares with Claude-2.1, Gemini Pro, GPT-3.5, Mistral, and Turbo; please verify that the reference supports the claim about performance similarity or add the appropriate citations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the fine-tuning input and MCQ accuracy output are independent; self-citation and scoring-rule caveats are validity issues, not circularity.

full rationale

Walked the claimed chain: baseline accuracy on 162 MCQs (Table 1), LoRA/qLoRA fine-tuning on textbook paragraphs (Section 2), then accuracy re-measurement (Figs. 3-6). The training input and the evaluation output are distinct objects: accuracy is measured from generated answers, not algebraically implied by the fine-tuning text. The PL partition (80 MCQs) is explicitly included as a litmus test for topics absent from the textbook, which is the opposite of a circular construction. The only self-citation is the Gabbrielli-Martini textbook [6], co-authored by a paper author; it supplies fine-tuning material rather than an unverified premise, the book is publicly available, and the dataset and code are released, so the finding is externally checkable. This is a conflict-of-interest/contamination caveat, not circularity. The 'at least one correct answer' scoring rule (Section 2) is a measurement-validity threat -- it may credit multi-letter guesses and change model ordering -- but it is not a fit renamed as a prediction and does not make any result true by construction. Section 4.1 limitations acknowledge dataset and model scope but not these two caveats; that is a completeness weakness, not circularity. No specific equation or claim reduces to its inputs, so score 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central empirical claim rests primarily on the choice and provenance of data (course textbook and MCQ collection), an under-specified LoRA configuration, and a lenient scoring rule.

free parameters (1)

LoRA rank and alpha
The fine-tuning method depends on LoRA parameters that are not reported in the paper; lit-gpt defaults are implied. These choices can materially affect accuracy and are needed to reproduce the fine-tuned models.

assumptions (3)

domain assumption LLaMA-2 is representative of current open-source LLMs.
Used in Section 2 'Threats to Validity' to generalize conclusions beyond LLaMA-2 to other models like Claude-2.1, Gemini Pro, GPT-3.5, Mistral.
domain assumption The 162 MCQs are a valid and representative test set for PL knowledge.
Section 2 'Threats to Validity' argues validity because questions come from exams, a textbook, and web sources validated by two PL teachers, but no inter-rater reliability or coverage analysis is given.
domain assumption The Gabbrielli and Martini textbook is an appropriate and sufficient source of domain knowledge for fine-tuning.
Section 2 'On Fine-Tuning: Technique and Dataset' uses this textbook as the sole fine-tuning corpus; its authority is asserted because it is widely adopted, not demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs." pith.science (2026). https://pith.science/paper/XCYTYQCC

@misc{pith2026250105891,
  author       = {Pith},
  title        = {Pith review of: Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XCYTYQCC}},
  note         = {Machine review of arXiv:2501.05891}
}

read the original abstract

In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL) -- the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material -- (parts of) the course's textbook -- for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable.

Figures

Figures reproduced from arXiv: 2501.05891 by the authors.

**Figure 2.** Number of MCQs per partition 2 Methodology MCQ Dataset. We first present the MCQ dataset. Both dataset 1 and code 2 are publicly available. Data include 162 questions where 15% are from an undergraduate-level university course on Programming Languages of the degree in Computer Science, 25% from a book on undergraduate-level questions on PL by Williams et al. [23], and 60% come from publicly available MCQs from the w… view at source ↗

**Figure 3.** Distribution of the accuracy of the fine-tuned mod [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Others MCQs Acc [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 8.** Figure 8: Correlation Matrix of the hyperparameters with [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 7.** Figure 7: Batch sizes accuracy with respect to epochs. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 13 canonical work pages

[1]

Lightning AI. 2023. LitGPT. https://github.com/Lightning-AI/litgpt

work page 2023
[2]

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit Optimizers via Block-wise Quantization. arXiv:2110.02861 [cs.LG]

arXiv 2022
[3]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 (2023)

arXiv 2023
[4]

Ramon Dijkstra, Zülküf Genç, Subhradeep Kayal, Jaap Kamps, et al. 2022. Reading Comprehension Quiz Generation using Generative Pre-trained Transformers. 3https://www.cineca.it

work page 2022
[5]

A. Extance. 2023. CHATGPT has entered the classroom: How llms could transform education. Nature News. A vailable at: https://www.nature.com/articles/d41586-023- 03507-3 623 (2023), 474–477

work page 2023
[6]

Maurizio Gabbrielli and Simone Martini. 2023. Programming languages: principles and paradigms. Springer Nature

work page 2023
[7]

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 (2017)

arXiv 2017
[8]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

arXiv 2021

Show all 29 references

[9]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[10]

Martin Juan José Bucher and Marco Martini. 2024. Fine-Tuned’Small’LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification. arXiv e-prints (2024), arXiv–2406

2024
[11]

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. 2019. A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.12322 (2019)

2019 arXiv
[12]

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

2017
[13]

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft

2022
[14]

Philip M Newton and Maira Xiromeriti. 2023. ChatGPT Performance on MCQ Exams in Higher Education. A Pragmatic Scoping Review. https://doi.org/10. 35542/osf.io/sytu3

2023
[15]

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems.arXiv preprint arXiv:2303.13375 (2023)

2023 arXiv
[16]

Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. 2024. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv p...

2024 arXiv
[17]

Pouya Pezeshkpour and Estevam Hruschka. 2023. Large Language Mod- els Sensitivity to The Order of Options in Multiple-Choice Questions. arXiv:2308.11483 [cs.CL]

2023 arXiv
[18]

Tareq Rasul, Sumesh Nair, Diane Kalendra, Mulyadi Robin, Fernando de Oliveira Santini, Wagner Junior Ladeira, Mingwei Sun, Ingrid Day, Raouf Ahmad Rather, and Liz Heathcote. 2023. The role of ChatGPT in higher education: Bene- fits, challenges, and future research directions. ...

2023
[19]

Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2023. Leveraging Large Language Models for Multiple Choice Question Answering. arXiv:2210.12353 [cs.CL]

2023 arXiv
[20]

Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023. Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code. arXiv:2303.08033 [cs.CL]

2023 arXiv
[21]

Bruno Silva, Leonardo Nunes, Roberto Estevão, Vijay Aski, and Ranveer Chandra
[22]

Andrew Tran, Kenneth Angelikas, Egi Rama, Chiku Okechukwu, David H Smith IV, and Stephen MacNeil. 2023. Generating Multiple Choice Questions for Computing Courses using Large Language Models. (2023)

2023
[23]

Williams

T.J. Williams. 2014. MCQs in Computer Science: Second Edition . McGraw-Hill Ed- ucation (India) Pvt Limited. https://books.google.it/books?id=CFO8AwAAQBAJ

2014
[24]

Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. A Survey on Language Models for Code. arXiv:2311.07989 [cs.CL]

2023 arXiv
[25]

Zheyuan Zhang, Jifan Yu, Juanzi Li, and Lei Hou. 2023. Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach. arXiv preprint arXiv:2310.08172 (2023)

2023 arXiv
[26]

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

2023 arXiv
[27]

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang
[29]

arXiv:2309.03882 [cs.CL]

Large Language Models Are Not Robust Multiple Choice Selectors. arXiv:2309.03882 [cs.CL]

arXiv
[2023]

arXiv:2310.06225 [cs.AI]

GPT-4 as an Agronomist Assistant? Answering Agriculture Exams Using Large Language Models. arXiv:2310.06225 [cs.AI]

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Lightning AI. 2023. LitGPT. https://github.com/Lightning-AI/litgpt

work page 2023

[2] [2]

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit Optimizers via Block-wise Quantization. arXiv:2110.02861 [cs.LG]

arXiv 2022

[3] [3]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 (2023)

arXiv 2023

[4] [4]

Ramon Dijkstra, Zülküf Genç, Subhradeep Kayal, Jaap Kamps, et al. 2022. Reading Comprehension Quiz Generation using Generative Pre-trained Transformers. 3https://www.cineca.it

work page 2022

[5] [5]

A. Extance. 2023. CHATGPT has entered the classroom: How llms could transform education. Nature News. A vailable at: https://www.nature.com/articles/d41586-023- 03507-3 623 (2023), 474–477

work page 2023

[6] [6]

Maurizio Gabbrielli and Simone Martini. 2023. Programming languages: principles and paradigms. Springer Nature

work page 2023

[7] [7]

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 (2017)

arXiv 2017

[8] [8]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

arXiv 2021