{"id":"8e5a025c-dc3d-40b6-a272-1cb9a2f8d35f","arxiv_id":"2501.11935","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In a 32-participant within-subjects study, CS2 students scored significantly better on a conceptual quiz for currying when learning via web search than via ChatGPT, while query behavior differed markedly between the two tools.","lead":"Researchers compared 32 computer science students learning two JavaScript concepts, one with ChatGPT and one with Google and YouTube. Students scored significantly higher on the conceptual quiz for the harder topic when using traditional web search, and they used different querying strategies in each setting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The currying effect rests on an under-verified between-subjects comparison; without evidence that the 16 AI and 16 NoAI participants were exchangeable, the causal claim is not secure.","rationale":"The reader's weakest_assumption identifies exactly the assumption I would attack first. The paper's central quantitative claim is the currying quiz difference between AI and NoAI, but for that specific task the comparison is between two disjoint groups of 16 participants. The within-subject design helps with task-level comparisons (currying vs IIFE) but does nothing to equalize the two treatment groups for a given task. The manuscript reports only aggregate participant characteristics (Tables 1 and 2), not a treatment-by-characteristic balance table for the currying cell. The phrase 'balanced' in Section 4.1 describes order and task assignment, not random assignment or covariate balance. With N=16 per cell, even a modest imbalance in prior JavaScript exposure, general programming skill, or motivation could produce the observed mean difference. The Threats to Validity section mentions small sample and gender imbalance but does not address this exchangeability issue, so the manuscript itself does not close the gap. I also considered whether incorrect ChatGPT responses during the study could be more load-bearing; that would affect the interpretation about the 'interactive, direct-answer format' but would not overturn the measured claim that ChatGPT was less effective for this task. Exchangeability is upstream: if the groups are not comparable, the measured claim itself is unsupported. I would keep the CONDITIONAL verdict because the qualitative results, the transparent reporting, and the plausible mechanism still make the study valuable, but the central causal claim should be treated as provisional pending replication or access to the raw data for the proposed balance check.","tokens_in":13643,"tokens_out":7757,"duration_ms":91257,"concrete_test":"Ask the authors for the per-participant raw data for the currying task (treatment, task order, gender, pre-study programming experience, CS1 grade/course performance, prior generative-AI usage) and compute standardized mean differences (SMD) between the AI and NoAI groups. If any |SMD|>0.3, re-estimate the quiz-score gap with an ANCOVA/mixed model adjusting for that covariate (and task order); if the adjusted gap shrinks below significance (p>.05), the central claim fails. If all SMDs are small and the gap survives adjustment, the exchangeability concern is settled and the conditional verdict can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing risk is that the headline currying effect (AI M=-0.69 vs NoAI M=2.0, p=.005, Section 5.2.2) is not protected by the within-subject design. For this comparison, the design is effectively between-subjects: each participant learned currying under exactly one treatment, so N=16 per cell. The paper states task/treatment assignment was 'balanced' (Section 4.1) but does not report random assignment, nor does it report any balance table for programming ability, CS1 performance, prior AI familiarity, or motivation across the two currying groups. Tables 1 and 2 report only aggregate demographics. If the NoAI-currying cell happened to contain stronger or more motivated students, the -2.69 gap could be selection, not tool effect. Section 6 acknowledges small sample and gender imbalance but not this exchangeability threat. The result is also a single uncorrected p among several comparisons, which makes it more fragile; however, the more fundamental issue is that causal attribution requires exchangeability, and that assumption is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a within-subjects lab study with 32 CS2 students at IIT Kanpur, in which each participant learned two JavaScript concepts—currying and IIFE—one with ChatGPT and one with traditional web-based resources (Google/YouTube). The authors analyze screen recordings, prompts/queries, quiz scores, and debugging performance to compare information-gathering strategies and learning outcomes across the two conditions. The headline finding is that participants scored significantly lower on a conceptual quiz for the more difficult concept (currying) after learning with ChatGPT (M = -0.69) than with web search (M = 2.0, p = .005), which the authors interpret as evidence that LLMs may be less effective for learning difficult concepts. Secondary findings include that LLM prompts are more likely to be follow-ups and that search-engine queries are more keyword-based.","tokens_in":13773,"tokens_out":6422,"duration_ms":62825,"significance":"If the headline result holds, the paper makes a meaningful contribution to the computing-education literature by providing a controlled comparison of LLM-based and web-based self-learning, with a mixed-methods design that includes both behavioral traces and outcome measures. The paper also contributes a prompt/query taxonomy and evidence on strategic differences (e.g., more follow-ups in AI, more keyword queries in web). However, the significance is currently limited by the lack of evidence for causal exchangeability in the key between-subjects currying comparison, and by the overstatement of 'ease' in the abstract. The paper is exploratory and would benefit from replication with larger samples and better-controlled assignment.","major_comments":[{"comment":"The currying comparison is an effectively between-subjects comparison with N=16 per group, yet the paper treats it as evidence for a causal effect of the tool. The manuscript states (Section 4.1) that assignment was 'balanced' but does not report random assignment or any balance table comparing the AI and NoAI currying groups on relevant covariates such as prior programming experience, CS1 performance, prior generative-AI familiarity, or motivation. Section 6 does not acknowledge this threat. Please provide a balance table and/or explicitly limit the claim to an associational finding; if causal language is intended, justify exchangeability or adjust for observed covariates.","section":"Section 5.2.2, 'AI vs. NoAI in Currying'"},{"comment":"The claim that students 'found it easier to learn a more difficult concept using traditional methods than using ChatGPT' is not directly supported by the measured variables. The study assessed quiz scores and debugging performance; it did not measure perceived ease or difficulty. The 'more difficult' characterization of currying is inferred from overall lower quiz scores and longer completion times, but no treatment-by-difficulty interaction was tested. Please rephrase to describe the actual outcome (e.g., 'participants performed better on a conceptual quiz for the more difficult concept after web search than after ChatGPT'), and soften the causal interpretation.","section":"Abstract and Section 1"},{"comment":"The headline p-value (p = .005) is one of many significance tests performed in the paper (paired t-tests, chi-square tests, GLM, Mann-Whitney U tests) without any multiple-comparison correction. With roughly a dozen tests, a conservative correction would push the threshold to about p < .004, making this result borderline or non-significant. Report the total number of tests, or provide corrected p-values or false-discovery-rate procedures, and include confidence intervals for the effect sizes (e.g., r = 0.50 for the currying comparison).","section":"Section 5.2.2 and overall statistical reporting"},{"comment":"The chi-square test for follow-up vs. base queries treats all 237 prompts and 148 queries as independent observations, even though multiple prompts/queries come from the same participant, inducing clustering. The Poisson GLM likewise models per-observation counts without accounting for within-participant correlation. Please use participant-level analyses (e.g., mixed-effects models or per-participant summaries) or explicitly justify the assumption of independence; otherwise these strategy differences may be spurious.","section":"Section 5.1.2"}],"minor_comments":[{"comment":"Section 6 says 'We randomized the order of tasks and treatments,' but Section 4.1 only says assignment was 'balanced.' Please clarify the exact assignment procedure (randomization vs. counterbalancing) and keep terminology consistent.","section":"Section 6"},{"comment":"The same reference appears twice: [40] and [41] are both Sun et al. (2024), and [47] and [48] are both Younas et al. (2025). Please consolidate duplicate citations.","section":"References"},{"comment":"The description of the multiple-choice question scoring is ambiguous: 'scored as +1, -1 based on all correct and incorrect answers respectively' could mean all-or-none scoring; please specify the exact scoring rule (e.g., full credit only if all correct options selected and no incorrect options).","section":"Section 4.4.2"},{"comment":"The phrase 'byte-sized answers' appears to be a pun, but it may distract; consider spelling as 'bite-sized' if that is the intended meaning, or use a more standard term.","section":"Section 5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid exploratory design but the headline causal claim rests on an under-verified between-subjects comparison. I recommend major revision: the authors should add a balance table for the currying groups, reframe the abstract to match the measured outcomes, and address the clustering issue in the query-level analyses. If the data cannot support these revisions, rejection or a strictly associational framing would be more appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a modest but useful exploratory comparison of CS2 students learning two JavaScript concepts with ChatGPT vs. traditional web search. The strongest part is the behavioral analysis: participants used more keyword queries with Google and more explicit question-style prompts with ChatGPT, and they followed up more in the LLM condition. Those findings are well-coded, with reported interrater reliability, and they make sense.\n\nThe headline quantitative claim—that students scored worse on the currying quiz in the AI condition (M=-0.69 vs. 2.0, p=.005)—is interesting but less solid than the abstract implies. That comparison is effectively between-subjects: each student learned currying under one treatment, so N=16 per cell. The paper says assignment was 'balanced' but does not report randomization or a balance table for prior programming ability, motivation, or AI familiarity across the two currying groups. Without exchangeability, the 2.69-point gap could be selection rather than tool effect. This is the main soft spot. Also, this is one significant p among several uncorrected tests, and the abstract's 'found it easier to learn' overstates what was measured: quiz scores, not perceived ease.\n\nThe IIFE comparison showed no significant difference, and the debugging task showed none either, so the currying result is a single isolated finding. That is acceptable for an exploratory study, but it needs replication with a larger, preregistered design.\n\nWhat the paper does well: clear research questions, a within-subjects protocol for the overall task/treatment pairing, balanced order, transparent instruments, and an honest threats section that mentions small sample and gender imbalance but misses the exchangeability issue. The query coding taxonomy is a useful contribution.\n\nBottom line: this is a legitimate exploratory study that belongs in the literature, but the central causal claim should be framed as preliminary. For peer review, I would send it out; I would ask the authors to add a balance table or explicitly label the currying result as hypothesis-generating, and to soften the abstract.\n\nRecommendation: accept with revision along those lines.","headline":"A useful exploratory comparison of ChatGPT vs. web search for CS2 learning, with solid behavioral findings but a headline quiz result that is less secure than the abstract suggests.","tokens_in":14347,"tokens_out":2108,"would_cite":true,"duration_ms":21402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CS2 students learned a harder programming concept less effectively with ChatGPT than with web search, with a large quiz-score gap on currying.","keywords":["ChatGPT","LLM in education","CS2","currying","IIFE","information seeking","self-directed learning","conceptual understanding"],"falsifier":"A larger preregistered study that randomly assigns students to ChatGPT or web search for currying, records prior JavaScript and programming experience, and finds no quiz-score gap (or a gap favoring ChatGPT) would overturn the central claim.","tokens_in":13396,"feed_emoji":"🤖","tokens_out":8415,"duration_ms":80669,"temperature":0.7,"pith_summary":"This paper argues that when CS2 students teach themselves a new programming concept, their information-gathering tool changes both how they search and how well they learn. In a within-subject lab study with 32 students, the harder of two JavaScript concepts, currying, was learned significantly less effectively with ChatGPT than with traditional web search: the average conceptual quiz score was $M=-0.69$ in the AI condition versus $M=2.0$ in the NoAI condition ($p=.005$). For the easier concept, immediately invoked function expressions, quiz scores did not differ significantly between the two conditions. The paper also claims that students adapt their questioning style to the tool, writing more follow-up and question-style prompts to ChatGPT and more keyword-based queries to search engines. If the result holds, the direct-answer interactivity of an LLM can be a disadvantage for initial learning exactly when the topic is difficult.","feed_headline":"Web search beat ChatGPT on a hard programming concept quiz","feed_subtitle":"CS2 students scored far lower learning currying with ChatGPT than with Google and videos.","key_machinery":"The argument is carried by a within-subject lab protocol in which each of 32 CS2 students learned two JavaScript concepts, currying and immediately invoked function expressions, in 15-minute sessions, one with ChatGPT (GPT-3.5 Turbo) and one with web search and video, with task-treatment order balanced. Learning outcomes were measured with a five-question conceptual quiz scored from $-8$ to $+8$ with a \"don't know\" option, plus a 10-minute debugging task. Information-seeking behavior was coded with a query and prompt taxonomy (base versus follow-up, copied versus edited, and phrasing categories such as Keyword-Based, Explanation, and Evidence-Based) and an activity codebook tracking time spent locating sources, learning by example, learning theory, and using the IDE. The mechanism that carries the claim is the contrast these instruments reveal: a large quiz gap on the harder concept combined with systematic differences in how students phrased requests to the two tools.","core_discovery":"The central claim is that information gathering for self-directed programming learning has different effectiveness depending on topic difficulty, and that traditional web search can beat ChatGPT on a harder concept. On currying, participants using ChatGPT scored an average of $-0.69$ (SD=2.2) on the conceptual quiz, versus $2.0$ (SD=2.06) for participants using Google and YouTube, a statistically significant difference ($p=.005$, effect size $r=0.50$); on the easier IIFE task, the corresponding comparison was not significant. The authors interpret the currying gap as evidence that the multi-source, keyword-driven process of web search yields a more complete understanding of a hard concept, while ChatGPT's direct, on-demand answers may narrow what a learner explores. They also report that participants wrote 237 prompts in the AI condition against 148 queries in the NoAI condition, and that LLM prompts were significantly more likely to be follow-ups and explanation-seeking questions. A debugging task showed no significant treatment difference, which the paper reads as a productivity-versus-learning distinction: LLM assistance can help produce working code without guaranteeing conceptual grasp.","pith_inferences":["A testable extension of this result is that the quiz gap between ChatGPT and web search should increase with topic difficulty and persist when learning is tested after a delay rather than immediately.","The query-analysis data suggest a mechanism the paper does not directly test: keyword search exposes learners to several curated sources, whereas follow-up prompts on an LLM can confine the learner to one conversational thread; a study that forces the same prompting style in both tools could isolate this cause.","The debugging results imply that in classrooms where AI tools are allowed, instructors should evaluate transfer with new problems rather than accepting successful code patches as evidence of learning.","A design implication not proven here is that enriching LLM responses with heterogeneous content or multiple suggested perspectives might recover some of web search's breadth; the paper's data only indirectly motivate this."],"forward_implications":["If the currying result generalizes, educators should not treat LLMs as drop-in replacements for web search in self-directed learning of difficult topics.","Students using LLMs for learning may need prompting guidance that encourages broad concept coverage rather than narrow follow-up questions.","The absence of a quiz difference on the easier concept indicates that blanket claims about LLM efficacy in programming education need to be qualified by topic difficulty.","The debugging-task result suggests that code that runs is a weak signal of understanding when AI assistance is available; assessments should probe conceptual knowledge separately.","For tool designers, the data imply that making LLM responses more holistic, for instance by pointing to multiple perspectives or external resources, could narrow the learning gap."],"supporting_citations":[{"why":"Shows that hybrid manual coding plus AI assistance can outperform AI alone, setting up the paper's question of when LLMs help learning.","marker":"[15]"},{"why":"Reviews LLMs in computing education and warns about overreliance, framing the open problem the study addresses.","marker":"[29]"},{"why":"Reports that generative AI helps some novice programmers and harms others, motivating the expectation of differential outcomes.","marker":"[30]"},{"why":"Establishes that web-based information gathering often means navigating multiple sources, the behavior contrasted with direct LLM answers.","marker":"[36]"},{"why":"Identifies that direct comparisons of web search and GenAI information-seeking are scarce, positioning the study's contribution.","marker":"[38]"},{"why":"Supplies the prompt taxonomy the paper adapts to classify ChatGPT prompt phrasing.","marker":"[14]"},{"why":"Supplies the non-factoid question taxonomy adapted for coding question-style queries in both conditions.","marker":"[6]"},{"why":"Argues that AI code generation changes the difficulty of programming, supporting the productivity-versus-learning interpretation of the results.","marker":"[5]"}],"fun_headline_variants":["ChatGPT loses to Google on hard CS2 concept","Currying stumps ChatGPT: Web search wins in CS2 study","Web search beats ChatGPT on difficult programming quiz","Google beats ChatGPT for tough coding concepts, study finds","Currying quiz: Google beats ChatGPT for CS2 learners"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 16 students who learned currying with ChatGPT and the 16 who learned it through web search were comparable in programming ability, motivation, and prior exposure, since the paper balances task order but does not measure these traits.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT loses to Google on hard CS2 concept","Currying stumps ChatGPT: Web search wins in CS2 study","Web search beats ChatGPT on difficult programming quiz","Google beats ChatGPT for tough coding concepts, study finds","Currying quiz: Google beats ChatGPT for CS2 learners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000881,"raw_usage":{"total_tokens":3797,"prompt_tokens":922,"completion_tokens":2875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":2797}},"tokens_in":538,"tokens_out":2875,"duration_ms":19740,"temperature":1.0,"reasoning_tokens":2797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:41:34.631815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A larger preregistered study that randomly assigns students to ChatGPT or web search for currying, records prior JavaScript and programming experience, and finds no quiz-score gap (or a gap favoring ChatGPT) would overturn the central claim.","supporting_citations":[{"cited_title":"How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment","cited_arxiv_id":null,"evidence_quote":"Shows that hybrid manual coding plus AI assistance can outperform AI alone, setting up the paper's question of when LLMs help learning."},{"cited_title":"The Robots are Here: Navigating the Generative AI Revolution in Computing Education","cited_arxiv_id":"2310.00658","evidence_quote":"Reviews LLMs in computing education and warns about overreliance, framing the open problem the study addresses."},{"cited_title":"The widening gap: The benefits and harms of generative ai for novice programmers","cited_arxiv_id":null,"evidence_quote":"Reports that generative AI helps some novice programmers and harms others, motivating the expectation of differential outcomes."},{"cited_title":"How knowledge workers use the web","cited_arxiv_id":null,"evidence_quote":"Establishes that web-based information gathering often means navigating multiple sources, the behavior contrasted with direct LLM answers."},{"cited_title":"An investigation of the drivers of novice programmers’ intentions to use web search and genai","cited_arxiv_id":null,"evidence_quote":"Identifies that direct comparisons of web search and GenAI information-seeking are scarce, positioning the study's contribution."},{"cited_title":"TELeR: A general taxonomy of LLM prompts for benchmarking complex tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the prompt taxonomy the paper adapts to classify ChatGPT prompt phrasing."},{"cited_title":"Bruce Croft, and Mark Sanderson","cited_arxiv_id":null,"evidence_quote":"Supplies the non-factoid question taxonomy adapted for coding question-style queries in both conditions."},{"cited_title":"Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos","cited_arxiv_id":null,"evidence_quote":"Argues that AI code generation changes the difficulty of programming, supporting the productivity-versus-learning interpretation of the results."}],"review_version":1}