{"id":"28fbda2f-50bc-463d-9f0b-230d63da8586","arxiv_id":"2412.08430","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A small pilot study claims GPT-4o is more personalized than LLaMA 3 and PaLM 2 for computing-career mentoring, but the paper's own similarity data point the other way.","lead":"This study compared three large language models on career mentoring advice for computing students and concluded that GPT-4o is the most personalized. The evidence is a small pilot with three synthetic profiles, 15 questions, descriptive word clouds, and two human raters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own semantic-similarity metric contradicts its headline claim: LLaMA 3 produces the most distinct answers by the stated operationalization, so 'GPT-4 more personalized' rests on an unvalidated word-cloud override.","rationale":"Reading the paper as a study comparing three LLMs for personalized mentoring, the central claim requires showing that GPT-4o's responses are more tailored to each student profile than LLaMA 3's or PaLM 2's. The only objective measure offered for this is the semantic-similarity pipeline in Section III.C, whose own criterion (lower fraction of high-overlap sentences = more distinct) makes LLaMA 3 the winner on every profile. The paper then switches to word clouds and a small human survey. Word clouds show what tokens are frequent but do not establish that those tokens reflect the profile or that they make answers more personalized; without a baseline, a model that repeats 'Hispanic' or 'woman' is not necessarily more personalized. The human evaluation appears only as a figure, with no numeric table, no statistical test, and no reliability measure for the two raters. Because the strongest available quantitative result contradicts the headline and no validated alternative is provided, the verdict REJECT stands. I partially agree with the reader: the confounded profiles in Table I are a real additional threat to RQ1/RQ2, but the decisive failure for the central claim is the mismatch between Table III and the reported conclusion.","tokens_in":9388,"tokens_out":3354,"duration_ms":34488,"concrete_test":"Release the raw LLM outputs for all 15 questions × 3 profiles × 3 models, then independently recompute Table III using the stated sentence-transformer method and threshold θ=0.7. Apply the paper's own rule that lower overlap means more distinct answers. If the recomputed ordering matches Table III, the ordering is LLaMA 3 > GPT-4o > PaLM 2, directly contradicting the abstract; if the word-cloud conclusion is to survive, it needs a separate quantitative distinctiveness metric with a defined validation procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.C defines personalization operationally: 'the percentage of sentences in the answers with high semantic similarity (>= theta) need to be low to have distinct answers.' Table III reports these percentages. Averaging across the PR, GR, and Overall rows for each profile yields GPT-4o: 0.352/0.197/0.421, LLaMA 3: 0.327/0.142/0.328, PaLM 2: 0.542/0.246/0.415. By the paper's own criterion, LLaMA 3 has the lowest overlap for every profile, i.e., the most distinct answers. The abstract and conclusion nevertheless claim GPT-4o 'offers more personalized mentoring.' The word-cloud analysis used to override Table III is purely descriptive: it reports prominent tokens without a baseline or a profile-specific expectation, and it is not scored against the paper's stated threshold. The human evaluation also has no numeric data table (Figure 3 only), only two raters, and no inter-rater reliability statistic. Thus the central comparative claim is not derivable from the paper's quantitative evidence; the internal evidence points in the opposite direction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript compares GPT-4o, LLaMA 3, and PaLM 2 as zero-shot personalized career mentors for computing students. Three hand-constructed student profiles (Table I) and 15 mentoring questions (Table II) are used. A semantic-similarity NLP pipeline (Section III.C) measures the percentage of sentences with similarity ≥0.7 between answers across profiles, and word clouds identify profile-specific vocabulary. A human evaluation (Section IV.B) uses 13 Likert items rated by two experts. The paper claims GPT-4 offers the most personalized mentoring.","tokens_in":9542,"tokens_out":4728,"duration_ms":47509,"significance":"The research question is timely, and the study is a useful template for evaluating LLM-based mentoring: it compares three LLMs under a fixed zero-shot prompt, makes the operationalization of \"personalized response\" explicit, and combines quantitative and human assessment. If the central claim were supported, the work would inform choices of LLM backends for career-mentoring tools. However, the current evidence does not support the headline claim, because the paper's own quantitative criterion ranks LLaMA 3 as more distinct/personalized than GPT-4o, and the human evaluation lacks statistical grounding. The study therefore currently serves more as a cautionary example than as a basis for selecting GPT-4.","major_comments":[{"comment":"The paper's stated operationalization of personalization is the low percentage of sentences with semantic similarity ≥ θ between responses to different profiles (Section III.C). Under this definition, the Overall Average row of Table III shows LLaMA 3 has the lowest overlap for every profile (0.327, 0.142, 0.328 versus GPT-4o's 0.352, 0.197, 0.421), meaning LLaMA 3 produces the most distinct answers. The abstract and conclusion nevertheless attribute the most personalized mentoring to GPT-4. This is a direct internal contradiction, and the quantitative basis for the central claim is missing.","section":"Section III.C and Table III"},{"comment":"The word-cloud analysis that is used to override Table III is purely descriptive. It reports salient tokens for each profile without a baseline corpus, without frequency counts, and without any statistical or threshold-based criterion for what counts as \"more personalized\". The qualitative observations about \"African American\", \"Hispanic\", and \"woman\" cannot by themselves establish a comparative ranking of GPT-4o over LLaMA 3.","section":"Section IV.A, Figures 4–6"},{"comment":"The human evaluation is reported without numeric results: Figure 3 shows questionnaire scores only as an image, no table of per-item means is given, and no inferential statistics are computed. Only two raters are mentioned, with no inter-rater reliability (e.g., Cohen's κ or Cronbach's α). Consequently, the claim that \"GPT-4 outperformed the other two LLMs\" in the survey is not statistically supported.","section":"Section IV.B and Figure 3"},{"comment":"The three student profiles differ simultaneously in gender, race/ethnicity, college year, and major (Table I). RQ1 asks whether LLMs account for social background (race/ethnicity), and RQ2 asks whether they account for educational background (year/major). With this profile set, any observed difference in responses between profiles could be due to any of the four attributes, or their interaction; it is impossible to attribute differences to race/ethnicity or educational level separately. This confound undermines both research questions.","section":"Section III.A and Table I"},{"comment":"The similarity threshold θ=0.7 is introduced without justification or sensitivity analysis. Since the comparative ranking of models depends on this threshold, and the threshold is not validated against human judgments of \"same answer,\" the main quantitative result is not robustly established.","section":"Section III.C, Eq. (1)"}],"minor_comments":[{"comment":"The sentence \"Table IV show the results\" should refer to Table III, which is the table that actually appears.","section":"Section IV.A"},{"comment":"The model name is written inconsistently as \"PaLM 2\" in Section III.B and \"Palm 2\" throughout Section IV; please standardize.","section":"Throughout"},{"comment":"Reference [5] is an anonymized placeholder (\"Anonymous. Anonymous. In 2024 ASEE Annual Conference & Exposition, 2024\") and must be completed or removed.","section":"References"},{"comment":"The caption of Figure 3 does not identify whether the plotted values are mean scores or individual ratings, and the image is too small to read the survey items; a companion table would improve clarity.","section":"Figure 3"},{"comment":"The word clouds use different visual scales and fonts across models, making cross-model comparison unreliable; adding frequency tables would improve clarity.","section":"Figures 4–6"},{"comment":"The phrase \"does not provider personalized answers\" should be \"does not provide personalized answers.\"","section":"Section III.C"}],"recommendation":"reject","confidential_remarks":"For the editor: the discrepancy between the paper's own quantitative metric and its headline claim is not a matter of interpretation—Table III directly contradicts the abstract. The human evaluation is too underpowered to rescue the claim, and the profile design confounds the attributes the research questions set out to isolate. A revision that addresses these issues would require a new experimental design and re-running the evaluation, which goes beyond the scope of a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's own semantic-similarity metric contradicts its headline: LLaMA 3 produces the most distinct answers by the stated operationalization, so the claim that GPT-4o is more personalized rests on an unvalidated word-cloud override. That is the first thing to know.\n\nWhat is new: this is a routine three-model comparison for AI career mentoring in computing, with 15 questions and three student profiles. The prompt design is transparent and the zero-shot setup is a reasonable baseline. The quantitative pipeline using sentence embeddings and a similarity threshold is a concrete, if crude, operationalization of personalization.\n\nThe soft spots are load-bearing. Section III.C defines personalization as low high-similarity sentence overlap (theta=0.7). Table III shows LLaMA 3 with the lowest overlap for every profile, so by the paper's own measure LLaMA 3 is most personalized. The abstract and conclusion instead credit GPT-4o, overriding the numeric result with word clouds that have no baseline or validation. The human evaluation adds only two raters with no reliability statistic and no numeric table. The three profiles change gender, race, year, and major simultaneously, so the RQs about isolating social or educational background cannot be answered. No data or code is provided, so nothing can be independently checked.\n\nDespite all this, the paper is not a waste. The experimental skeleton is salvageable, and the topic is practical. But the central claim fails on its own evidence. A revision that either reinterprets the semantic similarity findings or drops the word-cloud override could change my mind.\n\nFor a referee: this deserves a serious referee because there is a real evaluation question and the design is transparent. I would still recommend reject in its current form. The paper is for researchers in AI-in-education who are deciding between LLM backends for mentoring tools; they may find the table numbers useful after a careful reinterpretation.","headline":"The paper's own semantic-similarity metric says LLaMA 3 is more personalized, so the headline claim for GPT-4o rests on an unvalidated word-cloud override.","tokens_in":670,"tokens_out":2353,"would_cite":false,"duration_ms":42351,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4 gives more personalized AI mentoring than LLaMA 3 or PaLM 2","keywords":["large language models","AI mentoring","personalized mentoring","computing education","career planning","semantic similarity","human evaluation","zero-shot prompting"],"falsifier":"A controlled version of the same experiment—keeping the 15 questions and prompt format, but changing only one attribute at a time between otherwise identical profiles—would settle the claim: if GPT-4's overlap scores and word signatures barely move when race alone changes, then the personalization attributed to social background is actually driven by the other co-varying attributes, or by prompt noise.","tokens_in":1528,"feed_emoji":"🎓","tokens_out":2025,"duration_ms":90105,"temperature":0.7,"pith_summary":"The paper asks whether large language models can act as personalized career mentors for computing students, tailoring advice to who a student is—their race, gender, major, and year in college. It evaluates GPT-4, LLaMA 3, and PaLM 2 by giving each model 15 career questions from three fictional student profiles, then measures how much the answers differ across profiles. Both an automatic word-and-similarity analysis and human expert ratings point the same way: GPT-4's answers are more personalized than the other two models' answers. The authors conclude that LLM-based mentoring tools are feasible but should keep human mentors involved, and that fine-tuning on educational data could improve personalization further.","feed_headline":"GPT-4 gives more personalized AI mentoring than LLaMA 3 or PaLM 2","feed_subtitle":"Word-frequency and expert ratings show GPT-4 tailors career advice to a student's background; human mentors still belong in the loop.","key_machinery":"The load-bearing setup is a paired comparison: three student profiles (African-American male junior computer-science major; white male freshman undecided major; Hispanic female freshman computer-science major) are combined with 15 career-planning questions, and each LLM answers every question in zero-shot mode through a system prompt that instructs it to act as an AI mentor and a user message carrying the student's background. Personalization is measured in two ways: the NLP pipeline converts sentences to embeddings and flags sentence pairs whose cosine similarity $\\cos(a,b) = \\frac{a \\cdot b}{\\|a\\| \\, \\|b\\|}$ exceeds 0.7 as shared content, so a lower overlap percentage indicates more profile-specific answers; word clouds then show which words each model uses repeatedly for each profile. The human survey, built from earlier academic-advising and mentor-effectiveness instruments, provides the second, independent read on whether the profile-specific differences are actually better mentoring.","core_discovery":"The paper's central claim is that GPT-4 outperforms LLaMA 3 and PaLM 2 at personalized career mentoring for computing students: its responses vary more distinctly with a student's social and educational background, and human evaluators rate them as more accurate and useful. The quantitative evidence is that GPT-4's answers to the same question for different profiles share fewer semantically similar sentences (measured by cosine similarity above a 0.7 threshold in sentence-embedding space) and display profile-specific vocabulary such as 'Hispanic,' 'woman,' or 'professional network' in word clouds. The qualitative evidence is a 13-question Likert survey on which GPT-4 received the highest scores for most items, while PaLM 2 scored lowest. The paper also reports that all three models give weak help on setting concrete goals, and it attributes that to the absence of goal-oriented questions in the test set rather than to a general model limitation.","pith_inferences":["The paper does not test which background dimension drives the personalization: because profile changes are bundled, the GPT-4 advantage cannot yet be assigned specifically to race, gender, or experience level.","A natural extension is to run the same pipeline in interactive chat mode, where the model discovers the student's background through conversation; the current one-shot setup may understate how personalized a dialogue-based mentor could be.","The word-frequency and overlap method could transfer to other mentoring domains, such as engineering or business, and to non-English settings by swapping the sentence-embedding model.","A larger and more diverse evaluator panel, with results split by evaluator background, would test whether the human preference for GPT-4 is stable."],"forward_implications":["If GPT-4's advantage is stable across new questions and profiles, it is the natural default backend for AI career-mentoring assistants in computing education.","Because overlap measurement is automatic, the same pipeline can serve as a cheap screening test for personalization before a human evaluation is run.","Since all three models scored low on goal-setting, a practical mentoring tool would need a dedicated goal-setting module or additional prompt structure, not just a stronger LLM.","The authors' conclusion that human involvement is essential means near-term deployments should be human-in-the-loop, with the LLM drafting advice and a mentor checking it."],"supporting_citations":[{"why":"Defines GPT-4, the model whose zero-shot mentoring responses are compared.","marker":"[2]"},{"why":"Defines PaLM 2, the model used as a second comparison point.","marker":"[4]"},{"why":"Defines LLaMA 3, the model used as the third comparison point.","marker":"[12]"},{"why":"The authors' earlier computing-student survey that produced the 15 mentoring questions and their themes.","marker":"[5]"},{"why":"An academic-advising study with ChatGPT that supplies the question-driven evaluation approach and part of the human-evaluation metrics.","marker":"[3]"},{"why":"A study of chatbot response accuracy and reliability whose evaluation dimensions feed the human survey.","marker":"[16]"},{"why":"A faculty-mentoring effectiveness scale used to add mentor-quality criteria to the human evaluation.","marker":"[8]"}],"fun_headline_variants":["GPT-4 tops LLaMA 3 and PaLM 2 for personalized AI mentoring","Study: GPT-4 delivers more tailored AI career advice than rivals","GPT-4 best at adjusting mentoring to student background in computing","AI mentoring test: GPT-4 beats LLaMA 3 and PaLM 2","GPT-4 leads in personalized mentoring for computing students"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The load-bearing assumption is that comparing three profiles that differ at once in gender, race, college year, and major allows the study to attribute differences in the LLMs' answers to those background factors; because the profiles vary on several dimensions together, the contribution of any single dimension is not isolated.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 tops LLaMA 3 and PaLM 2 for personalized AI mentoring","Study: GPT-4 delivers more tailored AI career advice than rivals","GPT-4 best at adjusting mentoring to student background in computing","AI mentoring test: GPT-4 beats LLaMA 3 and PaLM 2","GPT-4 leads in personalized mentoring for computing students"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1507,"prompt_tokens":933,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":549,"tokens_out":574,"duration_ms":6228,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:50:09.312268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled version of the same experiment—keeping the 15 questions and prompt format, but changing only one attribute at a time between otherwise identical profiles—would settle the claim: if GPT-4's overlap scores and word signatures barely move when race alone changes, then the personalization attributed to social background is actually driven by the other co-varying attributes, or by prompt noise.","supporting_citations":[{"cited_title":"Anonymous","cited_arxiv_id":null,"evidence_quote":"The authors' earlier computing-student survey that produced the 15 mentoring questions and their themes."},{"cited_title":"Ai-supported academic advis- ing: Exploring chatgpt’s current state and future potential toward student empowerment","cited_arxiv_id":null,"evidence_quote":"An academic-advising study with ChatGPT that supplies the question-driven evaluation approach and part of the human-evaluation metrics."},{"cited_title":"Accuracy and reliability of chatbot responses to physician questions","cited_arxiv_id":null,"evidence_quote":"A study of chatbot response accuracy and reliability whose evaluation dimensions feed the human survey."},{"cited_title":"Measuring the effectiveness of faculty mentoring relationships","cited_arxiv_id":null,"evidence_quote":"A faculty-mentoring effectiveness scale used to add mentor-quality criteria to the human evaluation."}],"review_version":1}