{"id":"6eff5032-ee32-4592-8f20-465ba2ff92a2","arxiv_id":"2607.17022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs measurably vary code-explanation wording when prompted with different problem-solving styles, yielding a 13-category taxonomy and a model ranking.","lead":"Researchers asked six open-source AI models to explain COBOL code to people with different problem-solving styles, then compared the wording of 1,072 explanations. They found 13 ways the models changed their language—like giving step-by-step instructions or offering reassurance—and ranked the models by how often they adapted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 13-category taxonomy is derived and evaluated on the same 1,072 explanations with an odds-midpoint filter rather than a significance rule, no holdout, no multiple-comparison correction, and an unmatched control; the style-specific adaptation claim is therefore not yet established.","rationale":"The reader's conditional verdict already captures the main risk. My reading sharpens it: the most load-bearing point is not just that the control is unmatched, but that the taxonomy is derived and tested on the same data with a non-statistical selection rule, so the p-values in Tables 4–8 cannot serve as independent evidence for the 13 categories. This affects the central claim directly—if the categories were selected because they separated style values, re-observing that separation in the same corpus is expected, not confirmatory. The model ranking is downstream of the same categories and is additionally confounded by output length, as the paper's own concise-explanation analysis illustrates. A holdout-based derivation-and-test would settle whether the taxonomy generalizes. Because the basic phenomenon—models vary wording when given different style descriptions—is plausible and visually supported by many quoted examples, the verdict should remain conditional rather than rejected; the authors should add validation before the taxonomy is used in design claims.","tokens_in":24056,"tokens_out":8119,"duration_ms":80041,"concrete_test":"Hold out one model (preferably LLaMa 3, which drives several exclusive categories) before any category construction. On the remaining five models' outputs, re-derive candidate adaptations using a true significance rule (e.g., Fisher exact test with Benjamini-Hochberg correction across n-gram×comparison pairs, not the odds-midpoint threshold), freeze the regular expressions, then apply them to the held-out model's explanations and re-run the odds-ratio analyses of Tables 4–8. If the held-out model's categories largely fail to replicate—or if the matched-control comparison (replacing the 'Your user...' block with a neutral user description) removes the 'vs. control' differences—the taxonomy and model ranking should be treated as pipeline artifacts rather than evidence of adaptation to problem-solving style.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 (Analysis Methods) defines adaptation with a 'significantly more likely' condition, but the discovery pipeline does not apply any significance test. It filters 442,216 n-grams by (1) an odds-midpoint heuristic—'more likely than not' within the observed odds distribution—and (2) exclusivity with a 10% floor; the survivors are clustered into 13 categories. Regular expressions are then written for those same categories and applied back to the same 1,072 explanations to produce the odds ratios and p-values in Tables 4–8. Because the regexes were constructed after observing which n-grams separated the style values, those p-values are not confirmatory; they are at least partly a restatement of the selection criterion. The 'vs. control' leg is also structurally unmatched: the 11th prompt lacks the entire 'Your user...' agreement/alignment block, so any response to being given a user description counts as adaptation even if it is not style-specific. The model ranking in Figure 2 inherits both issues and is not normalized for output length; LLaMa 3's higher adaptation count could be a verbosity effect—indeed, the paper's own 'Give Concise Explanations' analysis shows LLaMa 3 claiming conciseness without delivering it. The central claim is plausible, but the current evidence does not separate style-cue effects from prompt-structure effects or from discovery overfitting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether LLM-generated code explanations adapt to GenderMag problem-solving style types. It constructs 10 style-value prompts from GenderMag survey statements plus a no-user control, generates 1,072 COBOL explanations from six open-weight LLMs, and discovers a taxonomy of 13 linguistic adaptation categories via n-gram odds, exclusivity filtering, clustering, and regex matching. It reports odds ratios and p-values for each category and ranks models by adaptations per explanation, concluding that LLMs adapt to 8 of the 10 style values and that LLaMa 3 adapts most frequently.","tokens_in":24406,"tokens_out":9980,"duration_ms":93276,"significance":"As a first mapping of LLM code-explanation behavior onto an established individual-differences framework, the paper addresses a real gap and offers a useful starting point. Its strengths are the use of GenderMag's validated items, the full disclosure of prompts, the multi-model corpus, and a self-checking analysis of the 'Give Concise Explanations' adaptation in which a null result is honestly reported. If the taxonomy were validated, it would be a valuable resource for personalized code explanation. However, the central claim is not yet established: the taxonomy is derived and evaluated on the same data with data-dependent thresholds, the control condition does not isolate style cues, and some classifications violate the paper's own statistical definition. These issues are fixable within the manuscript's scope, but the current evidence does not separate style-cue effects from prompt-structure effects or from discovery overfitting.","major_comments":[{"comment":"The discovery pipeline does not implement the paper's significance requirement. Of the 442,216 n-grams, the 'more likely than not' filter is an odds-midpoint cut, not a significance test; the exclusivity filter uses a 10% floor; and the thresholds (n=3–15, midpoint, 10%) are data-dependent. No multiple-comparison correction is applied. The regular expressions are then built after inspecting the same 1,072 explanations and are applied back to those explanations, so the p-values in Tables 4–8 are not confirmatory. A holdout, preregistered thresholds, or a permutation-based FDR procedure is needed.","section":"Section 3, 'Analysis Methods' (Definition of Adaptation)"},{"comment":"The control prompt is structurally unmatched. The 11th prompt is only the introductory block; all 10 style prompts append an agreement block and an alignment statement. Consequently, 'vs. Control' / 'vs. Unad.' odds ratios reflect any effect of providing a user description, not style-specific adaptation. Add a neutral user-description control with the same block structure, ideally one that contains the survey items without the style label.","section":"Section 3, 'Problem-Solving Style Value Prompts' and Appendix B"},{"comment":"'Address Code Complexity' is classified as risk-averse although the risk-averse vs. risk-tolerant comparison is p=.076 (OR=1.97), not significant at the .05 level the paper uses elsewhere. The text's 'directional consistency' does not satisfy the stated definition. Other rows (e.g., 'Normalize Knowledge Gaps') omit required control comparisons, indicated by dashes. Either restrict the taxonomy to rows meeting the stated definition or revise and apply the definition uniformly.","section":"Section 3 Definition vs. Table 6"},{"comment":"The corpus size 1,072 is unexplained: 6 models × 11 prompts × 3 programs yields 198 unique outputs. If there were repetitions, the paper must say how many and how they were sampled; if not, the count is inconsistent. This also matters because the statistical tests assume independent observations, whereas multiple outputs from the same model/program would be correlated.","section":"Section 3, 'LLM Selection & Explanation Generation' and Table 2 caption"},{"comment":"The model ranking counts adaptations per explanation without normalizing for output length. Since LLaMa 3 tends to produce longer explanations, its higher count may be a verbosity artifact. The paper's own 'Give Concise Explanations' analysis demonstrates that claimed adaptations need not correspond to behavioral change. Report rates per token or use length-matched comparisons after the taxonomy is corrected.","section":"Section 4, 'How Frequently Did These LLMs Adapt?' and Figure 2"},{"comment":"Some categories appear to echo prompt wording. For instance, 'Invite Follow-Up Questions' examples include 'how it works' and 'experimenting' from the tinkering prompt. If the n-gram/regex categories largely overlap with survey statements, the observed frequency differences may be lexical echoing rather than style-based adaptation. Quantify the overlap between category regexes and prompt n-grams and demonstrate that the taxonomy survives after removing prompt-derived phrases.","section":"Section 4, learning-style examples and Table 3"}],"minor_comments":[{"comment":"The caption says 'motivations adaptations'; the table reports information processing style adaptations.","section":"Table 8 caption"},{"comment":"The colors are mentioned in the caption but no legend is provided. Add a legend mapping colors to the 10 style values.","section":"Table 3"},{"comment":"'Over on third' should be 'Over one third.'","section":"Section 4, paragraph after Table 4"},{"comment":"The x-axis label is word count, but the plot style is not described. State whether the curves are densities or boxplots and add sample sizes.","section":"Figure 1"},{"comment":"Abbreviations (Unad., Proc., Comp., Sele., Ave., Tol.) are used without a single consolidated legend; define them in each caption or add a common legend.","section":"Tables 4–8"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible core idea and the authors are transparent about some limitations, but the central claims rest on a non-confirmatory discovery pipeline and an unmatched control. These are fixable with re-analysis, so I would not reject. I recommend major revision rather than acceptance as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new empirical exploration — GenderMag's five problem-solving styles as prompt conditions for LLM code explanations — and the authors are honest enough to show where their own result fails. But the 13-category taxonomy and the model ranking are discovery products, not confirmatory findings. The paper deserves a serious referee, but it needs a statistical overhaul before the specific claims can be trusted.\n\nWhat's new and good: the prompt design comes from a validated survey, the three COBOL programs are a sensible test bed, and the taxonomy is concrete and usable — a designer can walk away with 13 language-level adaptations and per-style prompts. The authors also triangulate with prior GenderMag literature and include a nice self-check: LLaMa 3 says it will be concise for selective users but actually writes longer comprehensive explanations. That kind of falsifiable check makes the exploratory part worth reading.\n\nThe soft spots are real, though. Section 3's adaptation definition says \"significantly more likely,\" but the discovery pipeline filters 442,216 n-grams with an odds-midpoint heuristic and a 10% exclusivity cut, both chosen from the data. The regexes were written after clustering and applied back to the same 1,072 explanations, so the p-values in Tables 4–8 are at least partly restatements of the selection criterion. No holdout, no multiple-comparison correction, and the control prompt lacks the \"Your user...\" block, so the \"vs control\" leg can't separate style-specific adaptation from generic user modeling. To be fair, the opposing-value comparisons are structurally matched, so the central direction isn't dead. The model ranking also isn't normalized for output length, and LLaMa 3's higher adaptation count could partly be verbosity. On top of that, 1,072 explanations doesn't obviously follow from 11 prompts × 6 models × 3 programs, and no data or code are provided.\n\nWho it's for: HCI, SE, and LLM personalization researchers who want a starting menu and prompts for personalized code explanations. I'd take the taxonomy as a hypothesis generator, not a validated instrument. If I were the editor, I'd send it to peer review — the novelty and honesty justify referee time — but I'd expect major revision: validate on a holdout, match the control, correct for multiple comparisons, release artifacts, and label the taxonomy as exploratory.","headline":"A genuinely novel and honestly reported first exploration of LLM code explanations adapting to GenderMag problem-solving styles, but the taxonomy and model ranking are discovery results, not confirmatory ones.","tokens_in":24873,"tokens_out":4180,"would_cite":true,"duration_ms":42230,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models measurably adapt their code explanations to a reader's problem-solving style, yielding a taxonomy of 13 linguistic shifts.","keywords":["code explanation","problem-solving styles","linguistic adaptation","prompting","LLM personalization","COBOL","n-gram analysis","GenderMag"],"falsifier":"A matched-pair experiment in which the agreement statements for opposing style values are swapped (e.g., presenting risk-averse statements but labeling the user risk-tolerant) would settle it: true style-driven adaptation should track the labeled value, while prompt echo would track the statement text; if the n-gram shifts follow the statements, the 'adaptation' is not about the user.","tokens_in":23952,"feed_emoji":"🧠","tokens_out":5367,"duration_ms":43404,"temperature":0.7,"pith_summary":"The paper asks whether LLMs adjust code explanations to fit different ways people solve problems. It generates 1,072 COBOL explanations from six open-weight LLMs, prompting each with one of ten problem-solving style values drawn from a validated inclusive design method. Using n-gram analysis and odds ratios, it finds 13 recurring linguistic adaptations—for example, step-by-step structure for process-oriented learners, reassurance and risk identification for risk-averse users, and reduced hedging for lower self-efficacy users. Models differ sharply: LLaMa 3 adapts far more often than CodeLLaMa. The implication is that code explanations need not be one-size-fits-all, and that model choice affects whether personalization happens.","feed_headline":"LLMs tailor code explanations to problem-solving styles","feed_subtitle":"Across 1,072 explanations, six open-weight LLMs shift step-by-step detail, hedging, and reassurance for 10 style values.","key_machinery":"The mechanism that carries the argument is the problem-solving style value prompt: each of the 10 prompts combines an introductory block (constant across conditions), an agreement block quoting survey statements the target value agrees or disagrees with, and a single alignment statement. These prompts are fed zero-shot to six open-weight LLMs. The detection pipeline applies CountVectorizer n-grams (3–15 tokens) and an odds-ratio definition: a phrase counts as an adaptation if it is significantly more likely in one style value than in its opposing value and the control, or appears exclusively in one value in at least 10% of its explanations. This renders 'adaptation' as a measurable linguisti","core_discovery":"The paper's central claim is that LLMs can and do adapt code explanations to a reader's problem-solving style, without fine-tuning, when given a prompt built from a validated problem-solving style survey. Across 1,072 explanations of three obfuscated COBOL programs, six open-weight LLMs produced 13 recurring linguistic adaptations: step-by-step explanations and follow-up question invitations for learning style; knowledge-gap normalization, hope-of-clarity, and reduced hedging for self-efficacy; reassurance, risk identification, and complexity acknowledgment for risk attitude; purpose-overview suppression for motivations; and conciseness plus ending summaries for information processing style.","pith_inferences":["The same prompting and n-gram pipeline could be applied to other explanation domains—medical, legal, tutorial—to see whether the taxonomy generalizes; the paper leaves this open.","The frequency-based definition may conflate prompt echo with genuine adaptation; a human evaluation of whether the shifted explanations actually improve comprehension would be a necessary next test.","The fact that one model accounts for nearly all 'Infer User's Interest' instances suggests that adaptation can be an artifact of a model's training data rather than the prompt—worth probing with more models.","If the taxonomy is real, then default code explanations are likely biased toward one style (the control already had 30% Step-by-Step and 44% Summarize), meaning some users are systematically underserved."],"forward_implications":["If correct, code explanation systems can personalize by prompting for a user's problem-solving style, without fine-tuning or model retraining.","The 13-category taxonomy gives UI designers a concrete checklist—step-by-step, reassurance, risk identification, summaries, and so on—for building explanation controls.","Model selection matters: LLaMa 3 produces roughly four adaptations per explanation, CodeLLaMa about one, so a system's personalization capacity depends on the backbone model.","The adaptation definition offers a reusable method for detecting style-driven linguistic shifts in any LLM output, not just code explanations."],"fun_headline_variants":["LLMs adapt code explanations to five problem-solving styles","Prompt-only: LLMs tailor code help to thinking styles","1,072 explanations show LLMs fit code advice to styles","Six open LLMs tweak code explanations for thinker types"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the frequency differences in n-grams are caused by the problem-solving style cue, not by the leftover wording of the prompt statements or the model echoing survey language; if that premise fails, the taxonomy describes prompt artifacts rather than genuine adaptation.","fun_headline_variants_meta":{"raw":{"variants":["LLMs adapt code explanations to five problem-solving styles","Prompt-only: LLMs tailor code help to thinking styles","1,072 explanations show LLMs fit code advice to styles","Six open LLMs tweak code explanations for thinker types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1086,"prompt_tokens":719,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":463,"tokens_out":367,"duration_ms":3574,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:14:38.577142+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched-pair experiment in which the agreement statements for opposing style values are swapped (e.g., presenting risk-averse statements but labeling the user risk-tolerant) would settle it: true style-driven adaptation should track the labeled value, while prompt echo would track the statement text; if the n-gram shifts follow the statements, the 'adaptation' is not about the user.","supporting_citations":[],"review_version":1}