{"id":"c90c4bc3-51d8-434a-9332-c4eb9f17c7c1","arxiv_id":"2501.05891","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuned 7B and 13B LLaMA-2 models answered course-specific programming language MCQs about as accurately as the much larger 70B model, at a lower hardware cost.","lead":"This study fine-tuned small LLaMA-2 models on a programming languages textbook and tested them on 162 course multiple-choice questions. It found that some small fine-tuned models can match the accuracy of a much larger pre-trained model while running on affordable hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'at least one correct answer' scoring rule credits multi-letter guesses, so the fine-tuned small models' reported advantage over 70B may be an artifact; re-scoring with a strict rule is needed.","rationale":"The central claim is an accuracy comparison, so any property of the accuracy metric that can systematically favor one model family over another is load-bearing. Section 2 defines correctness as 'provide at least one of these correct answers.' The paper applies this to all 162 MCQs, so extra wrong options are free. The prompt explicitly states that questions have multiple correct answers, encouraging the model to over-generate letters; fine-tuned models may do this more often than the 70B baseline. This could create a spurious advantage for the small fine-tuned models and invalidate the abstract's 'outperform' claim. The reader's weakest_assumption is the same issue, and I agree. A re-scoring experiment with exact-match/all-correct rules would settle it. If the rankings survive, the central claim has much stronger support; if not, the headline result is an artifact. I therefore leave the CONDITIONAL verdict unchanged: the paper should not be accepted as-is without this robustness check.","tokens_in":11589,"tokens_out":5882,"duration_ms":57859,"concrete_test":"Obtain the raw generated answers for all 720 fine-tuned variants and the pre-trained baselines from the authors' repository. Re-score every answer under a strict rule: for single-answer MCQs the predicted answer must contain exactly the correct letter; for multi-answer MCQs the predicted set must contain all correct letters and no incorrect ones (or, alternatively, equal the key set). Recompute Table 1, Table 5, and the 13B fine-tuned distribution in Fig. 3, and compare the best and median fine-tuned 13Bq accuracy against the 59% of 70Bq. Also report the distribution of the number of letters emitted per model. If the best or median fine-tuned 13Bq no longer reaches or exceeds 59%, or the ordering in Table 1 changes materially, the central claim is an artifact of the lenient scoring rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the evaluation rule in Section 2 ('MCQ Dataset'): 'To check the LLM's answer, it must provide at least one of these correct answers.' This rule is applied to all 162 questions, including the 143 single-answer MCQs. Consequently, a response such as 'a, b, c, d' to a single-answer question keyed 'a' is scored correct, and for the 19 multi-answer questions any one correct letter suffices even if wrong letters are included. The prompt itself instructs the model that 'The question is a multiple-choice question with multiple correct answers' (Section 2, 'Inference with Prompt'), which likely encourages multi-letter outputs. All reported accuracies (Table 1, Table 5, Figs. 3–6) use this permissive scoring. If the fine-tuned 7B/13B models emit more letters per answer than the pre-trained 70B baseline, their accuracy is inflated relative to the larger model, so the central claim that small fine-tuned models match 70B could be an artifact of the scoring rule. The paper reports no answer-length statistics and no stricter-scoring robustness check in Section 4.1's limitations. The PL partition does not rescue this issue, because it is scored under the same rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether fine-tuning smaller LLaMA-2 variants (7B and 13B) on a course-specific textbook can make them competitive with, or even superior to, the much larger pre-trained 70B model for answering multiple-choice questions (MCQs) in Programming Languages. The authors introduce a new public dataset of 162 MCQs, run inference on pre-trained and quantized models, fine-tune 720 variants with LoRA/QLoRA using different textbook partitions and hyperparameters, and analyze the effects of quantization, learning rate, batch size, and epochs on accuracy. The abstract claims that smaller textbook-based fine-tuned models outperform generic larger ones, while the body reports more cautiously that some fine-tuned alternatives can achieve the same accuracy as the larger pre-trained versions.","tokens_in":11775,"tokens_out":8056,"duration_ms":69500,"significance":"If the findings are robust, the paper makes a useful contribution to the emerging literature on domain-specific fine-tuning for education, showing that affordable consumer-grade GPUs can support models that approach the accuracy of much larger general-purpose LLMs on a specialized MCQ task. The public dataset and the broad hyperparameter sweep (720 fine-tuned models) are valuable assets for the community. However, the central comparison is currently threatened by a permissive scoring rule, a potential overlap between fine-tuning data and evaluation questions, and the absence of statistical significance testing, so the headline claim as stated is not yet supported.","major_comments":[{"comment":"The evaluation rule states: \"To check the LLM's answer, it must provide at least one of these correct answers.\" For the 143 single-answer MCQs, a response that contains the correct letter along with any number of incorrect letters is scored correct, and for the 19 multi-answer MCQs one correct letter suffices even if other letters are wrong. The prompt also explicitly tells the model that \"The question is a multiple-choice question with multiple correct answers,\" which plausibly encourages multi-letter outputs. All reported accuracies (Table 1, Table 5, Figs. 3–6) are computed under this rule. If fine-tuned small models tend to emit more letters per response than the pre-trained 70B baseline, their accuracy is inflated relative to the larger model, so the central claim that small fine-tuned models match 70B could be an artifact of the scoring rule. The paper provides no answer-length statistics and no robustness check under a stricter rule that requires all correct answers to be present. This is a load-bearing issue because the main comparison in the abstract and Section 4 rests on these accuracy numbers.","section":"Section 2, MCQ Dataset and Inference with Prompt"},{"comment":"The abstract claims that \"smaller textbook-based fine-tuned models outperform generic larger ones,\" but Section 3 states only that \"some fine-tuned alternatives can achieve the same accuracy as the larger/base versions\" and Section 4 says the smaller models achieve \"an accuracy that is similar to the larger pre-trained ones.\" Table 5 reports the percentage of fine-tuned variants that beat their own pre-trained version (e.g., 78% for 13B quantized, whose pre-trained baseline is 45%), not the percentage that exceed the 70B quantized model's 59% accuracy on the full dataset. The paper never directly reports the accuracy of the best (or median) fine-tuned model against the 70Bq baseline, so the \"outperform\" wording in the abstract is unsupported by the presented evidence. The authors should either provide direct comparisons with confidence intervals or temper the abstract to match the body's more cautious phrasing.","section":"Abstract, Section 3, Section 4"},{"comment":"The fine-tuning material is the textbook \"Programming Languages: Principles and Paradigms\" by Gabbrielli and Martini, and one of the co-authors of this paper (Gabbrielli) is also a co-author of that textbook. In addition, 15% of the MCQs come from an undergraduate PL course that likely uses this textbook (the paper describes the textbook as \"widely adopted\" and the course is at the same institution). This creates a risk of data leakage or privileged access: the fine-tuned models may benefit from memorizing textbook content that directly overlaps with the exam questions, inflating their apparent advantage over a generic 70B model. The \"Threats to Validity\" paragraph does not acknowledge this overlap or the authorial relationship. The authors should disclose this conflict, quantify the overlap between the fine-tuning data and the evaluation questions, and report the central comparison on the subset of questions that do not overlap with the textbook (e.g., the PL partition) as the primary evidence rather than as a secondary analysis.","section":"Section 2, MCQ Dataset and On Fine-Tuning"},{"comment":"The reported accuracies are point estimates on a single test set of 162 questions, with no confidence intervals, bootstrap estimates, or significance tests. Differences such as 13B 50% vs. 13Bq 45% in Table 1 are within the sampling noise of a 162-item test (standard error of roughly 4 percentage points). The claim that \"some fine-tuned alternatives can achieve the same accuracy\" as 70Bq requires a statistical non-inferiority test or at least a bootstrap interval; otherwise the main comparison is not supported. The authors should add these analyses or explicitly label the findings as exploratory and refrain from making strong comparative claims without them.","section":"Section 3, Tables 1 and 5 and Figures 3–6"},{"comment":"The fixed prompt includes the sentence \"The question is a multiple-choice question with multiple correct answers\" for every question, including the 143 questions that have a single correct answer. This instruction is factually incorrect for the majority of the dataset and likely biases models toward producing multi-letter answers, which interacts with the permissive scoring rule discussed above. The authors should either tailor the prompt to the actual number of correct answers per question or perform a sensitivity analysis to determine how much this instruction affects the results. Without such an analysis, the prompt design itself is a confound in the comparison between fine-tuned and pre-trained models.","section":"Section 2, Inference with Prompt"}],"minor_comments":[{"comment":"The heading \"Threats to Validity\" is misspelled as \"Treats to Validity\" in the manuscript; consider correcting the typo and moving this paragraph to the Limitations section for better visibility.","section":"Section 2, Threats to Validity"},{"comment":"The violin plots do not show the number of fine-tuned models at each accuracy level, and the density scale is not labeled. Consider adding a legend that identifies the horizontal lines (pre-trained accuracies) and reporting the sample size per violin.","section":"Figure 3"},{"comment":"The memory consumption values for fine-tuning 13B base and 70B base are listed as \">80GB\" without a measured peak; consider reporting actual measurements or a more precise upper bound, since the exact memory footprint is relevant to the affordability claims.","section":"Table 2"},{"comment":"The statement that \"only one chapter of the book can generally lead to more stable results\" is not quantified; consider reporting the interquartile range or variance of accuracies for each fine-tuning dataset partition.","section":"Section 3.2.1"},{"comment":"The mapping of the fine-tuning dataset hyperparameter (1 = whole book, 2 = three chapters, 3 = single chapter) is defined only in the text; consider adding it to the figure caption or to Table 4 so that Figure 8 is self-contained.","section":"Section 3.3"},{"comment":"Reference [9] is for Mixtral, but the text in Section 2 compares with Claude-2.1, Gemini Pro, GPT-3.5, Mistral, and Turbo; please verify that the reference supports the claim about performance similarity or add the appropriate citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially serious undisclosed conflict of interest: one author (Gabbrielli) is a co-author of the textbook used for fine-tuning, and the evaluation set includes a subset of questions from a course that likely uses that textbook. This should be disclosed and addressed in the revision. The permissive evaluation rule (\"at least one correct answer\") is a further threat to validity that could change the ranking of models, particularly given the prompt instructing that questions have multiple correct answers. The abstract overstates what the body actually supports. I recommend major revision with a request for re-scoring under a stricter rule, statistical significance testing, direct comparison of fine-tuned models against the 70B baseline, and a revised abstract that matches the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is a public dataset of 162 programming-languages MCQs plus a systematic comparison of LoRA/qLoRA fine-tuned LLaMA-2 7B/13B models against the pre-trained 70B on that dataset. They actually run 720 fine-tuned variants, release code and data, and report memory footprints. That is reproducible empirical work, and the resource-constrained framing is legitimate: a 13B quantized model that matches a 70B on course-specific questions is interesting for educators.\n\nThe soft spots are real and load-bearing. The scoring rule accepts an answer if it contains at least one correct letter, and the prompt literally tells the model the question has multiple correct answers. So a single-answer question keyed 'a' is scored correct for the output 'a, b, c, d'. The paper gives no statistics on how many letters each model emits, and no robustness check with a strict all-correct rule. If the fine-tuned small models are more verbose than the 70B baseline, their reported accuracy advantage could be a scoring artifact. That is the first thing I would check.\n\nSecond, the abstract says 'outperform generic larger ones,' but Section 3 says 'some fine-tuned alternatives can achieve the same accuracy.' Those are different claims. There are no confidence intervals or significance tests; the best-of-720 selection is presented as evidence. The provenance overlap (fine-tuning on a textbook co-authored by a paper author, with test items from a course using it) is a fairness concern, though the PL partition not covered by the textbook gives some independent support—under the same lenient scoring.\n\nThe paper is not sloppy in its experiments; the hyperparameter analysis is thorough, and the limitations section covers scope. It just doesn't address the evaluation-metric problem, which is central.\n\nWho is this for? People building local LLM assistants for education, and anyone working on MCQ evaluation methodology. As a referee, I would ask for a strict re-scoring, an answer-length table, and a corrected abstract before accepting. The dataset and fine-tuning comparison deserve a serious look, so I'd send it to review, not desk reject.","headline":"Useful dataset and thorough fine-tuning sweep, but the lenient scoring rule and abstract's overclaim make the central result unproven.","tokens_in":12392,"tokens_out":2065,"would_cite":false,"duration_ms":19997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small fine-tuned LLaMA-2 models match a 70B model on course MCQs, at a fraction of the hardware cost.","keywords":["Large Language Models","multiple-choice questions","fine-tuning","quantisation","LLaMA-2","programming languages education","affordable LLMs","LoRA"],"falsifier":"Re-run the evaluation using a stricter rule that requires all correct alternatives to be present (and no incorrect ones) for the 19 multi-answer questions, then compare the accuracy rankings; if the fine-tuned small models no longer match the 70B baseline, the central claim would be substantially weakened.","tokens_in":11340,"feed_emoji":"🎓","tokens_out":2841,"duration_ms":27838,"temperature":0.7,"pith_summary":"This paper asks whether educators and students can affordably use LLMs to answer course-specific multiple-choice questions, and answers yes: fine-tuning small open-source LLaMA-2 variants (7B and 13B) on the course textbook makes them about as accurate as the much larger 70B pre-trained model on a 162-question Programming Languages MCQ test. The fine-tuned 13B quantised variant, in particular, matches the 70B's 59% accuracy while using only about 21GB of GPU memory for fine-tuning, within reach of consumer-grade hardware. The paper also contributes the MCQ dataset and a systematic study of how fine-tuning hyperparameters and textbook material selection affect accuracy. The central consequence is that domain-specific fine-tuning can substitute for model scale in this educational task, making LLM-based assessment tools materially affordable.","feed_headline":"Small fine-tuned LLMs match a 70B model on course MCQs","feed_subtitle":"A 13B LLaMA-2 fine-tuned on a programming-languages textbook reaches the larger model's accuracy using consumer-grade GPUs.","key_machinery":"The mechanism is parameter-efficient fine-tuning with LoRA and QLoRA, applied to LLaMA-2 7B and 13B variants using a dataset derived from the course textbook (Gabbrielli and Martini), where each paragraph becomes an instruction–output training sample. The MCQ evaluation dataset of 162 questions, with topic labels aligned to the textbook chapters, lets the authors separate questions covered by the fine-tuning material from those outside it, serving as a litmus test for catastrophic forgetting and for whether fine-tuning teaches general domain vocabulary rather than rote answers. The inference prompt is a fixed template asking the model to output the correct alternatives, scored under the rule that providing at least one correct answer counts as correct.","core_discovery":"The paper's central claim is that smaller, textbook-based fine-tuned models outperform generic larger ones when answering course-specific MCQs. Concretely, on a 162-question undergraduate Programming Languages MCQ set, the pre-trained 70B quantised LLaMA-2 reaches 59% accuracy, while fine-tuned variants of the 13B quantised model—trained with QLoRA on paragraphs from the course textbook—reach comparable or higher accuracy. Among the 240 fine-tuned 13B quantised configurations, 78% beat the pre-trained 13B base, and the best of them track the 70B baseline. The authors further show that the choice of textbook chapters used for fine-tuning is the single most influential hyperparameter, and that quantised fine-tuning (QLoRA) preserves accuracy while cutting memory requirements enough to run on consumer GPUs.","pith_inferences":["The paper's evidence suggests a general recipe for other technical courses: fine-tune a 7B or 13B open model on the assigned textbook, then evaluate on past exam MCQs, to obtain an in-domain assistant at consumer hardware cost.","The scoring rule that rewards any one correct alternative may systematically inflate accuracy, especially for the 19 multi-answer questions; a stricter all-correct rule could change the ranking between small fine-tuned models and the 70B baseline.","The observed catastrophic forgetting in roughly 15% of fine-tuned 7B variants points to a practical safety check: institutions should validate fine-tuned models on questions outside the training material before deployment.","Because the paper focuses on a single course and a single model family, the transferability of 'textbook fine-tuning beats scale' to other subjects, question formats, and newer open models remains an open question worth testing."],"forward_implications":["If the claim holds, educational institutions can deploy course-specific MCQ answering assistants on affordable consumer GPUs instead of relying on large proprietary models.","The result implies that textbook material, which educators already have, is sufficient fine-tuning data for at least this STEM domain, reducing the data-engineering barrier.","The finding that a single well-chosen textbook chapter yields stable accuracy gains suggests that targeted fine-tuning can be more effective than broad fine-tuning on a full book.","Quantised fine-tuning (QLoRA) is shown to be effective enough to nearly close the gap between small quantised models and larger unquantised ones, lowering the entry cost further.","The public MCQ dataset provides a reproducible benchmark for future work on LLM accuracy in programming-languages education."],"supporting_citations":[{"why":"Supplies the LLaMA-2 model variants (7B, 13B, 70B) that are the subjects of the study.","marker":"[11]"},{"why":"The course textbook by Gabbrielli and Martini is the source of the fine-tuning dataset.","marker":"[6]"},{"why":"LoRA is the low-rank adaptation technique used for fine-tuning, reducing memory and parameter updates.","marker":"[8]"},{"why":"QLoRA is the quantised fine-tuning method that makes fine-tuning feasible on consumer GPUs.","marker":"[3]"},{"why":"Prior work by Bucher and Martini showing small fine-tuned models can outperform larger generic ones, which this paper extends to MCQs.","marker":"[10]"},{"why":"Shows LLMs need ad-hoc prompting for MCQ tasks, motivating the prompt template used in the experiments.","marker":"[19]"},{"why":"The Williams book supplies 25% of the MCQ dataset questions.","marker":"[23]"}],"fun_headline_variants":["13B LLM fine-tuned on a textbook beats 70B on course MCQs","Textbook-tuned 13B tops generic 70B on course MCQs","Small fine-tuned LLM outdoes 70B on course quizzes","Cheap fine-tuning lets 13B beat 70B on course MCQs","Affordable 13B fine-tune tops 70B on course MCQs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation counts a response as correct if it contains at least one of the listed correct answers, so a model that includes the right letter among several guesses gets credit; on the 19 multi-answer questions this can inflate accuracy and possibly change which model wins.","fun_headline_variants_meta":{"raw":{"variants":["13B LLM fine-tuned on a textbook beats 70B on course MCQs","Textbook-tuned 13B tops generic 70B on course MCQs","Small fine-tuned LLM outdoes 70B on course quizzes","Cheap fine-tuning lets 13B beat 70B on course MCQs","Affordable 13B fine-tune tops 70B on course MCQs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2707,"prompt_tokens":925,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":541,"tokens_out":1782,"duration_ms":11544,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:06:10.170934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation using a stricter rule that requires all correct alternatives to be present (and no incorrect ones) for the 19 multi-answer questions, then compare the accuracy rankings; if the fine-tuned small models no longer match the 70B baseline, the central claim would be substantially weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The course textbook by Gabbrielli and Martini is the source of the fine-tuning dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work by Bucher and Martini showing small fine-tuned models can outperform larger generic ones, which this paper extends to MCQs."},{"cited_title":"Williams","cited_arxiv_id":null,"evidence_quote":"The Williams book supplies 25% of the MCQ dataset questions."}],"review_version":1}