(WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

Mohamed Hisham Abdellatif

REVIEW 3 major objections 7 minor 13 references

(WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Paper claims fine-tuning PHI-3 on TruthfulQA lifts multiple-choice accuracy to 90.8%.

desk verdict A candid but methodologically unsupported fine-tuning report; the 90.8% accuracy claim cannot be trusted without a held-out split, but the positional-bias admission is genuinely instructive. read the letter →

arxiv 2501.01588 v1 pith:AV7JYU66 submitted 2025-01-03 cs.CL cs.AI

Mohamed Hisham Abdellatif This is my paper · ORCID

classification cs.CLcs.AI

keywords LLMfine-tuningmultiple-choicequestionsTruthfulQAPHI-3promptdesignperplexity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a compact language model, PHI-3 at 1.3 billion parameters, can be turned into an accurate multiple-choice question answerer by supervised fine-tuning on the TruthfulQA dataset and by redesigning the prompt. The author reports perplexity dropping from 4.68 to 2.27 and accuracy rising from 62% to 90.8% after only 60 training steps on a standardized 1,000-question set. The motivation is practical: if small models can reach this level, automated tutoring and assessment systems can run on modest hardware. The paper presents this as evidence that prompt design and fine-tuning matter as much as model scale for structured tasks like MCQ answering.

What carries the argument

The machinery is the combination of dataset standardization and prompt engineering before supervised fine-tuning. TruthfulQA is a factual multiple-choice benchmark designed to expose hallucination, and the paper standardizes it by keeping the best correct answer and equalizing the option count. The prompt is the sensitive part: a simple completion prompt caused the model to exploit answer position, always picking the last option, while an Alpaca-style instruction prompt (a style of instruction-following prompt template) plus fine-tuning made the model answer by content. This prompt-plus-fine-tuning pipeline carries the claim, because the reported gains depend on replacing positional guessing with content-based selection.

What would settle it

Hold out a random 20% of the TruthfulQA questions, fine-tune only on the remaining 80% with the paper's stated hyperparameters, and report accuracy on the held-out set; if held-out accuracy is well below 90.8% while training-set accuracy is near 90.8%, the reported gain is memorization rather than learning to answer MCQs.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that PHI-3, initially a general text generator, becomes a reliable MCQ answerer after supervised fine-tuning and prompt redesign. The best prompt combines the basic completion format with an Alpaca-style instruction format, and the dataset is preprocessed so each question has one best answer and a consistent number of options. The author reports that with this setup the fine-tuned PHI-3.5 reaches 90.8% accuracy, an F1 score of 90.6, and a perplexity of 2.27, outperforming the GPT-3 baseline in the comparison table despite using roughly 1% of the parameters. The paper attributes the gain to eliminating the positional bias that made the initial prompt always choose the last option.

Load-bearing premise

The result assumes that the reported accuracy of 90.8% was earned on TruthfulQA questions that were not part of the 1,000-item training set, but the paper never states such a split.

Editorial extensions

If this is right

If the result holds, a 1.3-billion-parameter model can outperform a reported GPT-3 baseline on TruthfulQA-style MCQs while being deployable on consumer hardware.
The design implies that prompt formatting is as important as model scale for MCQ tasks; a poorly structured prompt leads to positional overfitting rather than comprehension.
Resource-constrained educational platforms could use fine-tuned compact models for automated assessment and tutoring without large GPU clusters.
The perplexity drop from 4.68 to 2.27 suggests the fine-tuned model is substantially more confident in its answer choices, which would matter for real-time feedback systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never states a train/test split, so an immediate extension is to report held-out accuracy separately from training-set accuracy to determine whether the 90.8% figure reflects generalization or memorization.
Varying the order of answer options during evaluation would measure how much of the gain is genuine content understanding rather than residual positional bias, which the paper itself identifies as a failure of its initial prompt.
Testing the same prompt design on other MCQ benchmarks such as MMLU or ARC would show whether the method transfers beyond TruthfulQA's specific format.
The abstract's 62% baseline and Table I's 78.3% baseline differ, so the size of the reported improvement depends on which baseline the reader uses.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper reports a fine-tuning run of PHI-3 on TruthfulQA for MCQ answering, with a striking accuracy jump to 90.8% and a perplexity drop from 4.68 to 2.27. But the evaluation is not described as held-out, the baseline number changes from 62% in the abstract to 78.3% in Table I, and the paper itself admits the model learned to pick the last option. So the central claim is unsupported as written. That said, there is something honest and useful here: the author clearly documents a positional-overfitting failure, which is a real pitfall in MCQ fine-tuning and a more candid admission than most papers give.

The concrete training setup—60 steps, batch size 8, seed 3407, on a GTX 1650—is clearly stated, and the goal of adapting a 1.3B model for low-resource education is sensible. Unfortunately, the dataset and code links are placeholders ('here' with no URLs), so rerunning the experiment is not possible as submitted.

The load-bearing flaws are the ones the stress-test note flags. No train/test split is stated anywhere. With 1,000 examples and 60 training steps, the reported accuracy could easily be memorization. The abstract says baseline accuracy was 62%, Table I says 78.3%—that is a large unexplained discrepancy. The prompt section is vague ('best-performing prompt combined elements from both approaches'), so there is no reproducible prompt-engineering insight. And the introduction's claim of a 'novel approach to prompt design' is not backed by any specific innovation.

These are not minor blemishes; they undermine the headline result. The underlying experiment could be valid if rerun with a proper held-out split and corrected baselines, but this paper does not support the 90.8% claim. I would send it to peer review for the sole purpose of giving the author a clear path to fix the evaluation—the flaws are fundamental but fixable. I would not cite it, and I would not bring it to reading group except as a short example of how not to report fine-tuning results. The honest admission about positional bias is the only part I'd remember.

Referee Report

3 major / 7 minor

Summary. The paper reports an empirical study of fine-tuning Microsoft PHI-3 (referred to in places as PHI-3.5) on the TruthfulQA dataset for multiple-choice question answering. The authors describe a pipeline of dataset preprocessing, prompt design, supervised fine-tuning with an 8-bit optimizer, and evaluation with perplexity, accuracy, F1, and recall. The central claim, stated in the abstract and Section V, is that fine-tuning improves accuracy from 62% to 90.8%, lowers perplexity from 4.68 to 2.27, and raises F1 from 66 to 90.6, with Table I reporting a PHI-3 baseline accuracy of 78.3%. The paper also acknowledges that an initial prompt format induced a positional bias where the model consistently selected the last option.

Significance. If the central claim were supported, the paper would provide a notable data point: a 1.3B-parameter model reaching 90.8% MCQ accuracy after only 60 training steps on a 1,000-item dataset would be remarkable for resource-constrained educational applications. The work also transparently reports training hyperparameters and points to code and data, which are strengths. However, the claimed results are not backed by a described evaluation protocol: no held-out split is stated, the abstract and Table I disagree on the baseline accuracy, and the paper itself admits a positional-overfitting failure mode. As a result, the main quantitative claims are currently unsupported.

major comments (3)

[Sections III, IV, V] The paper never states that evaluation was performed on items held out from fine-tuning. Section IV-B describes training for up to 60 steps with an effective batch size of 8 on a 1,000-item dataset, and Section V reports perplexity and accuracy improvements, but no train/test split, cross-validation, or evaluation-set construction is described anywhere in Sections III–V. Without a held-out split, the reported drop in perplexity from 4.68 to 2.27 and the accuracy gain from 62% (abstract) or 78.3% (Table I) to 90.8% are compatible with memorization of the training data, which is exactly the risk raised by the paper's own admission of positional overfitting in Section IV-C. This is load-bearing for the central claim and cannot be verified from the manuscript.
[Abstract and Table I] The reported baseline accuracy is internally inconsistent: the abstract states that accuracy rose from 62% to 90.8%, while Table I lists the PHI-3 baseline as 78.3%. Similarly, the abstract and Section V state that F1 increased from 66 to 90.6, but Table I lists baseline F1 as 0.75 and fine-tuned F1 as 0.90. These mismatches mean a reader cannot determine the actual improvement, and the central claim of a 62% to 90.8% jump is contradicted by the paper's own table.
[Section IV-C] The paper states that with the initial prompt format the model 'consistently chose the last option' and 'learned to exploit the options' positions rather than engage in reasoning.' The revised prompt is said to fix this, but no evaluation on held-out data or ablation is provided to show that the final prompt does not still induce a positional or superficial bias. Given that the reported accuracy is the only evidence for the fine-tuning benefit, this acknowledged failure mode directly undermines the claim that the final results reflect understanding rather than format exploitation.

minor comments (7)

[Section IV-C] The text refers to 'Section 4.2' for the revised prompt details, but no numbered subsection 4.2 exists; the prompt revision is not actually described in detail anywhere.
[Section III-A.1] The paper says TruthfulQA contains '1,000 MCQs,' but the original TruthfulQA release contains 817 questions; the preprocessing that yields 1,000 items, if any, should be described and cited precisely.
[Throughout] The model is called both PHI-3 and PHI-3.5 in the abstract, Section V, and Table I; the authors should clarify which model version was actually fine-tuned and evaluated.
[References] Reference [5] lacks year, venue, and arXiv identifier, and the citation [12] in the Conclusion appears to be the TruthfulQA reference placed after 'future work' rather than where it is needed in the methodology section.
[Introduction and Figure 1] The introduction says 'as it is clear in I' without a clear figure reference, and Figure 1 has a caption but no visible image content; the figure should be either included or removed.
[Section IV-B] The paper says training was run for 'a maximum of 60 steps' but does not report the actual number of steps used or the number of training examples per step, which makes it impossible to compute how many epochs the model saw.
[Abstract and Section V] The code and dataset are referred to as 'in here' placeholders; the actual URLs are not printed in the manuscript, preventing reproducibility.

Circularity Check

1 steps flagged · score 7.0 of 10

No held-out evaluation is described; the reported fine-tuning gains on TruthfulQA may reflect training-set memorization rather than generalization.

fitted input called prediction [Section III-A, Section IV-B, Section V, Table I]
"We use the TruthfulQA dataset, which contains 1,000 MCQs across various categories... This preprocessing step ensured the model had a consistent input format, which was crucial for fine-tuning and evaluation. ... Training was conducted for a maximum of 60 steps. ... We observed a sharp decrease in perplexity from 4.68 to 2.27 post-fine-tuning... Accuracy improved from 62% to 90.8%."

The paper never specifies a train/test split. The same TruthfulQA dataset is described as serving both fine-tuning and evaluation, with no held-out portion identified. With 1,000 total items and only 60 training steps, the reported perplexity drop and accuracy rise are computed on items that the model has seen during training (or at least on the same dataset distribution without any stated separation). The claimed improvement is therefore a measure of training-set fit, not a prediction on unseen data. The paper's own admission of positional overfitting in Section IV-C further indicates that the model memorized prompt-position cues rather than learning a generalizable answer strategy.

full rationale

The derivation chain is short: preprocess TruthfulQA, fine-tune PHI-3.5 on it, then report perplexity and accuracy on TruthfulQA. Nowhere does the paper state a train/test split, so the central result (perplexity 4.68 to 2.27, accuracy 62% to 90.8%) is not demonstrably an out-of-sample prediction; it is consistent with evaluating on the training set. This is a fitted input called prediction: the model is fitted to the dataset, and the headline metrics are taken from the same dataset. The abstract's 62% baseline also conflicts with Table I's 78.3% baseline, further undermining the claimed gain. The paper does not rely on self-citations for load-bearing arguments, and the prompt-design discussion is independent, but the evaluation itself is circular because no held-out split or external benchmark is used. This warrants a score of 7, indicating the central claim reduces to a training-set fit rather than a demonstrated generalization.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the dataset being valid ground truth, on an unstated held-out evaluation split, and on a handful of hand-chosen settings such as the prompt template, training steps, and seed. No new theoretical entities are introduced.

free parameters (3)

Best-performing prompt template = Not disclosed
The paper selected one prompt format from several variants based on accuracy and perplexity (Section IV-C and Section V). The selected prompt determines the reported 90.8% accuracy, and no held-out prompt selection is described.
Training step count (max 60) = 60
Chosen by hand as sufficient for the small dataset; the result depends on stopping early, so it is a tuned hyperparameter rather than a derived quantity.
Random seed = 3407
Set for reproducibility, but a single seed means no variance estimate; the reported numbers are conditional on this seed.

assumptions (3)

domain assumption TruthfulQA labels are correct and provide a fair target for MCQ accuracy.
The evaluation equates selecting the dataset's best answer with correctness; the paper does not audit label quality for the preprocessed subset.
domain assumption Reported metrics are computed on questions not used for fine-tuning.
No train/test split is stated anywhere in Sections III to V, yet the improvement claim presupposes held-out evaluation.
domain assumption PHI-3.5 refers to the same model family as the PHI-3 technical report.
The paper cites the PHI-3 report [1] but reports results for PHI-3.5 without explaining the relationship.

how reviews work

0 comments

Cite this review

Pith. "Pith review of (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges." pith.science (2026). https://pith.science/paper/AV7JYU66

@misc{pith2026250101588,
  author       = {Pith},
  title        = {Pith review of: (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AV7JYU66}},
  note         = {Machine review of arXiv:2501.01588}
}

read the original abstract

Large Language Models (LLMs) have become essential tools across various domains due to their impressive capabilities in understanding and generating human-like text. The ability to accurately answer multiple-choice questions (MCQs) holds significant value in education, particularly in automated tutoring systems and assessment platforms. However, adapting LLMs to handle MCQ tasks effectively remains challenging due to the hallucinations and unclear prompts. This work explores the potential of Microsoft's PHI-3\cite{Abdin2024}, a compact yet efficient LLM, for MCQ answering. Our contributions include fine-tuning the model on the TruthfulQA dataset, designing optimized prompts to enhance model performance, and evaluating using perplexity and traditional metrics like accuracy and F1 score. Results show a remarkable improvement in PHI-3.5's MCQ handling post-fine-tuning, with perplexity decreasing from 4.68 to 2.27, and accuracy rising from 62\% to 90.8\%. This research underlines the importance of efficient models in adaptive learning systems and educational assessments, paving the way for broader integration into the classroom, particularly in fields like test preparation, student feedback, and personalized learning.

Figures

Figures reproduced from arXiv: 2501.01588 by the authors.

**Figure 2.** Proposed Methodology Pipeline had a consistent input format, which was crucial for fine-tuning and evaluation. 2) Prompt Design: We experimented with different prompts to guide PHI-3 in answering MCQs accurately. Initially, we used a basic text completion prompt, which led to hallucinations and irrelevant answers. We then modified the prompt structure using Alpaca-style prompts, allowing more precise control over t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages

[1]

Phi-3 technical report: A highly capable language model locally on your phone,

M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V . Chaudhary, D. Chen, D. Chen, W. Chen, Y .-C. Chen, Y .-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V . Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswa...

arXiv 2024
[2]

Multiple-choice questions are efficient and robust llm evaluators,

Z. Zhang, Z. Jiang, L. Xu, H. Hao, and R. Wang, “Multiple-choice questions are efficient and robust llm evaluators,” 5 2024. [Online]. Available: http://arxiv.org/abs/2405.11966

arXiv 2024
[3]

Generating multiple choice questions from a textbook: Llms match human performance on most metrics,

A. M. Olney, “Generating multiple choice questions from a textbook: Llms match human performance on most metrics,” 2023. [Online]. Available: http://ceur-ws.org

work page 2023
[4]

Can large language models be an alternative to human evaluations?

C.-H. Chiang and H. yi Lee, “Can large language models be an alternative to human evaluations?” 5 2023. [Online]. Available: http://arxiv.org/abs/2305.01937

arXiv 2023
[5]

Can multiple-choice questions really be useful in detecting the abilities of llms?

W. Li, L. Li, T. Xiang, X. Liu, W. Deng, N. Garcia, and M. A. Lab, “Can multiple-choice questions really be useful in detecting the abilities of llms?” [Online]. Available: https: //github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs

work page
[6]

Hellaswag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” 5 2019. [Online]. Available: http://arxiv.org/abs/1905.07830

arXiv 2019
[7]

Winogrande: An adversarial winograd schema challenge at scale,

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” 7 2019. [Online]. Available: http://arxiv.org/abs/1907.10641

arXiv 2019
[8]

Training verifiers to solve math word problems,

K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 10

work page

Show all 13 references

[9]

Measuring massive multitask language understanding,

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” 9 2020. [Online]. Available: http://arxiv.org/abs/2009.03300

2020 arXiv
[10]

Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena,

A. Myrzakhan, S. M. Bsharat, and Z. Shen, “Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena,” 6 2024. [Online]. Available: http://arxiv.org/ abs/2406.07545

2024 arXiv
[11]

Think you have solved question answering? try arc, the ai2 reasoning challenge,

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 3 2018. [Online]. Available: http://arxiv.org/abs/1803.05457

2018 arXiv
[12]

Truthfulqa: Measuring how models mimic human falsehoods,

S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” 9 2021. [Online]. Available: http://arxiv.org/abs/2109.07958

2021 arXiv
[2021]

Available: http://arxiv.org/abs/2110.14168

[Online]. Available: http://arxiv.org/abs/2110.14168

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Phi-3 technical report: A highly capable language model locally on your phone,

M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V . Chaudhary, D. Chen, D. Chen, W. Chen, Y .-C. Chen, Y .-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V . Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswa...

arXiv 2024

[2] [2]

Multiple-choice questions are efficient and robust llm evaluators,

Z. Zhang, Z. Jiang, L. Xu, H. Hao, and R. Wang, “Multiple-choice questions are efficient and robust llm evaluators,” 5 2024. [Online]. Available: http://arxiv.org/abs/2405.11966

arXiv 2024

[3] [3]

Generating multiple choice questions from a textbook: Llms match human performance on most metrics,

A. M. Olney, “Generating multiple choice questions from a textbook: Llms match human performance on most metrics,” 2023. [Online]. Available: http://ceur-ws.org

work page 2023

[4] [4]

Can large language models be an alternative to human evaluations?

C.-H. Chiang and H. yi Lee, “Can large language models be an alternative to human evaluations?” 5 2023. [Online]. Available: http://arxiv.org/abs/2305.01937

arXiv 2023

[5] [5]

Can multiple-choice questions really be useful in detecting the abilities of llms?

W. Li, L. Li, T. Xiang, X. Liu, W. Deng, N. Garcia, and M. A. Lab, “Can multiple-choice questions really be useful in detecting the abilities of llms?” [Online]. Available: https: //github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs

work page

[6] [6]

Hellaswag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” 5 2019. [Online]. Available: http://arxiv.org/abs/1905.07830

arXiv 2019

[7] [7]

Winogrande: An adversarial winograd schema challenge at scale,

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” 7 2019. [Online]. Available: http://arxiv.org/abs/1907.10641

arXiv 2019

[8] [8]

Training verifiers to solve math word problems,

K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 10

work page