{"id":"12d7cf8a-9adf-458c-bc97-600d2240bc19","arxiv_id":"2412.12800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Non-native English speakers can solve introductory programming problems by prompting large language models in their native languages, but success depends heavily on the language and students often fall back to English.","lead":"Students in Portugal, China, and Saudi Arabia tried to solve coding tasks by writing instructions to an AI chatbot in their own languages. Many succeeded, but success varied sharply by language, and students often felt that English produced more accurate code than their native tongue.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's inference that students 'solved the Prompt Problems in their native language' conflates aggregate prompt counts with successful submission streams; the reported data do not show that successful attempts were actually in the native language.","rationale":"The reader's weakest_assumption concerns the comparability of the three language groups and the explanation for Arabic's lower success rate. That is a secondary interpretive issue. My concern targets the central feasibility claim directly: the paper's conclusion that students solved the problems in their native language does not follow from the presented aggregate data. Table 3 counts prompts, not outcomes, and the single stream-level sentence appears to describe all submission streams rather than successful ones. For Chinese, the correct-prompt counts are split across Native, English, Mixed, and Code, so the possibility that students succeeded only after switching to English is real and unaddressed. This is load-bearing because the title and abstract promise that native-language prompting works, but the reported evidence does not establish that the successful solutions were actually achieved through native-language prompts. The qualitative quotes show students experienced difficulty, but they do not connect the final correct solutions to native-only prompting. A per-stream re-analysis would settle this. If the re-analysis shows high pass rates for exclusively-Native streams, the claim holds and the paper's contribution stands; if not, the verdict should move toward REJECT or UNVERDICTED. Since the reader already recommended CONDITIONAL, and this concern adds a specific missing analysis rather than disproving the result, I keep the verdict as CONDITIONAL (UNCHANGED) but with a different condition than the reader's: require the per-stream outcome breakdown, not just group comparability controls.","tokens_in":16780,"tokens_out":9698,"duration_ms":89901,"concrete_test":"Re-analyze the logged Promptly submission data to compute, for each language group and each problem, the pass rate of submission streams whose prompts were exclusively category N, and compare it with the pass rate of streams that used English, Mixed, or Code at any point. If the exclusively-Native pass rate is materially lower than the overall pass rate reported in Table 2 (especially for Chinese P2/P3 and Arabic P1/P2), then the paper's claim that students successfully solved the problems in their native language is not supported by the current analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 concludes that 'students were able to successfully solve the Prompt Problems in their native language' from two facts: (i) Table 2 shows most students eventually passed the test cases, and (ii) Table 3 shows most prompts were categorized as 'N' (native language with English technical terms). However, Table 3 reports per-prompt counts, not per-submission-stream outcomes. The text's only stream-level statement, 'Of 172 successful submission streams, 152 used only 1 state, and 127 of those were Native,' is internally inconsistent: the total number of submission streams across problems sums to 172 (82 Arabic + 28 Chinese + 62 Portuguese), and 152 + 9 + 9 + 1 + 1 = 172, so the '172' refers to all submission streams, not successful ones. Thus the paper never reports the pass rate of streams that stayed exclusively in the native language. The Chinese group is a concrete risk: Table 3 shows only 6 of 19 correct prompts were Native, with 6 English, 5 Mixed, and 2 Code, so Chinese successes may have involved switching to English. Without a per-stream crosstabulation of strategy and success, the central feasibility claim is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an exploratory study in which three groups of non-native English speaking students (Arabic, Chinese, and Portuguese) solved 'Prompt Problems' by writing prompts to a generative AI model in their native language. The authors present per-problem success rates (Table 2), a categorization of prompting strategies by language (Table 3), and qualitative themes from post-activity surveys. The central claim is that students can successfully use their native language to solve programming problems, with particular difficulty observed for Arabic, which the authors attribute partly to low-resource language support in LLMs. The paper also reports student perceptions about expressivity, model performance, and the entrenchment of English in programming.","tokens_in":16961,"tokens_out":4360,"duration_ms":41161,"significance":"If the central claim were soundly established, the result would be significant for computing education: it would offer evidence that LLM-based prompting can lower English-language barriers for non-native speakers, with implications for inclusive pedagogy and global access to programming instruction. The study is timely and addresses a genuinely under-explored question. The use of an external success criterion (passing hidden test cases) is a strength, as is the collection of authentic classroom data from three institutions and the inclusion of student voices via qualitative analysis. However, as analyzed, the data do not demonstrate that successful submissions were actually produced by native-language prompting, and the cross-language comparisons are confounded. The paper's contribution is better described as a descriptive feasibility observation than as a demonstrated result.","major_comments":[{"comment":"The inference that 'students were able to successfully solve the Prompt Problems in their native language' is not supported by the data as presented. Table 2 reports pass rates for all students who attempted each problem, but it does not indicate the language of the successful prompts. Table 3 reports per-prompt counts, not per-submission-stream outcomes. The only stream-level statement in the paper, 'Of 172 successful submission streams, 152 used only 1 state, and 127 of those were Native,' is internally inconsistent: 172 equals the total number of submission streams (82 Arabic + 28 Chinese + 62 Portuguese from Table 2), not the number of successful streams, which is 132 (27+15+12+12+5+2+21+20+18). The sentence therefore appears to refer to all submission streams, not successful ones. The paper never reports the pass rate of streams that stayed exclusively in the native language. For instance, among Chinese students only 6 of 19 correct prompts were categorized as 'Native,' with 6 'English' and 5 'Mixed,' so successful Chinese streams may have involved switching to English. To establish the central claim, the authors need to provide a per-stream crosstabulation of prompting strategy and success, or explicitly weaken the claim to 'students can pass some tasks using prompts that include their native language.'","section":"Section 4.1, Tables 2 and 3"},{"comment":"The data cleaning step removes all prompts by students who only prompted in English. This selection biases the sample toward students who attempted native-language prompting and removes any possibility of an English-only baseline. RQ1 asks 'How successful are students at solving Prompt Problems in their non-English native languages?' but the reported success rates are computed only for students who used at least some non-English words in their prompts. Without an English-prompting control group or at least a report of the number and success rates of excluded students, the paper cannot distinguish native-language feasibility from general LLM problem-solving ability. The authors should report the exclusion counts and, if possible, compare success rates between native-language and English-prompting students within the same cohorts.","section":"Section 3.2.1"},{"comment":"The cross-language comparison, particularly the claim that Arabic students experienced greater difficulty partially due to limited training data, is confounded. Section 5.3 admits that the Arabic group were undergraduates using Java at a different institution, the Chinese group were postgraduates using Python, and the Portuguese group were undergraduates using Python. These differences alone could explain the observed lower Arabic success rates. The conclusion in Section 6 states that Arabic-speaking students 'appeared to face greater challenges, with lower accuracy and higher model misinterpretations' and attributes this to training-data limitations, but the data do not support a causal attribution. The authors should either present within-group evidence (e.g., same problems, same programming language, comparable educational level) or rephrase the discussion to present the training-data explanation as one of several plausible hypotheses. As written, the conclusion overreaches what the confounded design can support.","section":"Section 5.3, Section 6"}],"minor_comments":[{"comment":"The manuscript contains a typographical error: 'a Portugese university' should be 'a Portuguese university.'","section":"Section 3.1"},{"comment":"The category labels are inconsistent between the text and Table 3: the text defines categories O, E, N, M, C, but Table 3 labels the columns 'Other,' 'English,' 'Native,' 'Mixed,' 'Code.' Please align the terminology.","section":"Section 3.2.1"},{"comment":"The sentence 'Native was the most frequent starting state as well as the most frequent ending state' would be more informative if accompanied by counts or a transition table, especially given the paper's emphasis on state changes.","section":"Section 4.1"},{"comment":"The phrase 'Anecdotally, this theme was much more prevalent in responses in the Arabic-speaking sample' is odd because the authors are reporting on their own coded data; consider replacing 'anecdotally' with 'in our coded data' or a similar phrase.","section":"Section 4.2.1"},{"comment":"The sentence 'It's possible that the student of the future does not think programmatically in English because of these advances' is speculative; consider flagging it more explicitly as a forward-looking hypothesis rather than an implication of the data.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the authors have assembled a valuable dataset. However, the central inference from the quantitative data is currently not valid, and the cross-language comparison is confounded. The issues are fixable: a per-stream analysis of prompting strategy and success, a transparent report of the excluded English-only students, and more cautious wording about the low-resource-language explanation would substantially strengthen the paper. I recommend major revision rather than rejection, because the underlying data are likely sufficient to support a weaker and still interesting claim about native-language prompting as a feasible pathway for some learners."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is genuinely the first study I've seen where students themselves—not researchers—wrote prompts in Arabic, Chinese, and Portuguese to solve code-generation tasks, and the pass/fail signal comes from hidden test cases rather than self-report. Second, the paper's central sentence—'students were able to successfully solve the Prompt Problems in their native language'—is not backed by the data as reported. The authors need to fix this before the claim stands.\n\nWhat it does well: the task design is neat, the qualitative themes (native language is more expressive but English gets better model performance; students are habituated to English in programming) are believable and useful, and the authors are upfront about several study limitations. The Arabic low-resource story is plausible and aligns with prior NLP work.\n\nSoft spots. The big one is a numbers problem. The text says 'Of 172 successful submission streams, 152 used only 1 state, and 127 of those were Native.' But the sum of attempts across the three problems is 172 (82+28+62), so '172' is the total number of streams, not successful ones. And the 1-state streams include 9+9+1+1 for other categories, so the 127 Native figure is a count of streams, not a count of passing streams. The paper never reports how many streams that stayed exclusively in Native actually passed. Without that crosstab, Table 2 (aggregate pass rates) and Table 3 (per-prompt language categories) cannot support the conclusion that native-language prompting is what led to success. The Chinese data make the risk concrete: only 6 of 40 Native prompts were correct, and 6 correct prompts were English, so a Chinese student who passed may well have switched to English.\n\nOther issues: no English baseline, so we can't tell if native prompting helps or hurts relative to English; students who only prompted in English were removed in cleaning; the three cohorts differ in programming language (Java vs Python), educational level, and institution, making the Arabic-Chinese comparison uninterpretable; no statistical tests; the LLM version is unspecified. Several of these are acknowledged, which is honest, but they still limit the quantitative interpretation.\n\nBottom line: this is a worthwhile exploratory study with a real first dataset, but the central feasibility claim needs a per-stream analysis and ideally an English comparison. I'd send it to review, with the stream-level inconsistency flagged. It's citable as early evidence, but not as demonstrated fact.","headline":"First student-facing multilingual prompting study, but the headline claim overreaches: the data don't actually show that successful solutions came from native-language prompts.","tokens_in":17579,"tokens_out":3131,"would_cite":true,"duration_ms":27896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that non-native English speakers can solve programming problems by prompting generative AI in their native languages, and provides first classroom evidence from Arabic, Chinese, and Portuguese learners.","keywords":["generative AI","multilingual prompting","Prompt Problems","non-native English speakers","computing education","low-resource languages","Arabic","large language models"],"falsifier":"Conduct the same three Prompt Problems with matched groups of the same educational level, same programming language (say Python), and same institution, prompting in Arabic, Chinese, and Portuguese through the same model; if Arabic pass rates match Portuguese and Chinese, the paper's low-resource-language explanation is false.","tokens_in":16547,"feed_emoji":"💬","tokens_out":4875,"duration_ms":46141,"temperature":0.7,"pith_summary":"The paper tries to establish that generative AI can break the language barrier in programming education: students who are not native English speakers can solve programming exercises by prompting an AI chatbot in their own languages. It reports the first classroom exploration of this idea, with students writing prompts in Arabic, Chinese, and Portuguese to generate code for standard CS1-style problems, and the generated code being checked automatically against test cases. Most students completed the tasks in their native language, and Portuguese and Chinese speakers succeeded at high rates; Arabic speakers succeeded less often, which the paper attributes to the model's weaker support for a low-resource language. The paper also documents a persistent tension: students felt more expressive in their native language but often found English prompts produced more accurate code, especially because programming syntax and keywords are English-centric. If the finding holds, multilingual prompting could lower a major barrier for the large majority of the world's programmers who learn in a second language.","feed_headline":"Non-native speakers can solve coding tasks in their own language","feed_subtitle":"Arabic, Chinese, and Portuguese speakers solved coding tasks by prompting an AI in their own languages, lowering the English barrier.","key_machinery":"The central object is the Prompt Problem, a programming exercise in which a student is shown a computational task visually and must write a natural-language prompt for an LLM so that the generated code passes a suite of test cases. The tool that runs this loop, Promptly, executes the generated code and reports pass or fail, giving an objective measure of whether a prompt succeeded. In this paper the same exercise format is used across three language groups, and prompts are categorized by how much English, native language, or code they contain. The Prompt Problem mechanism matters because it isolates the student's ability to specify a solution in natural language from their ability to write syntax, which is exactly the skill that English-centric programming instruction tends to suppress.","core_discovery":"The central claim is that non-native English speakers can successfully use their native language as the interface to generative AI for programming, solving Prompt Problems by writing prompts in Arabic, Chinese, or Portuguese rather than in English. Success is measured by whether the code the model generates passes a hidden test suite; by that measure, a majority of students in all three groups completed at least the first problem, and the Portuguese and Chinese groups showed high success rates overall. Arabic speakers completed problems at a noticeably lower rate, and the paper argues this reflects the small share of Arabic in the training data of large language models. Across all groups, students reported that native-language prompting felt more expressive and natural, but that they had to supply English terms for programming concepts, and many believed English prompts gave more accurate results. The paper concludes that native-language prompting is viable today for some languages, while acknowledging that the AI's language support and the student's familiarity with English coding terms shape how well it works.","pith_inferences":["If the Arabic gap is truly a training-data effect, then rerunning the same problems with a newer model should shrink it; this is a cheap, direct test of the paper's main explanation.","The same logic suggests a policy implication the authors only gesture at: investment in low-resource language data for code-generating models could be one of the most effective ways to widen access to programming education worldwide.","The 'thinking in English' reports hint that the deeper barrier is the English-centric design of programming languages themselves; GenAI may lower, not remove, that barrier until languages or interfaces are localized.","A testable extension would measure learning outcomes, not just pass rates: does native-language Prompt Problem practice transfer to better performance on later English-only programming tasks, or does it mainly build prompting skill?"],"forward_implications":["Instructors can assign Prompt Problems in learners' native languages and still grade them automatically with test cases, creating accessible entry points for non-native English speakers.","Students who understand programming concepts but struggle with English can demonstrate that understanding through native-language prompts, at least in high-resource languages.","Arabic-speaking learners currently face an uneven experience: the paper's data suggest their prompts are more likely to be misunderstood, so educators should plan extra iteration time or pair native-language prompting with an English fallback.","Because the model's English-centric training drives many of the difficulties, improvements in multilingual training data should directly improve native-language success rates.","The finding that students code-switch to English technical terms implies that curricula still need to teach programming vocabulary in English, even if problem-solving happens in the native language."],"supporting_citations":[{"why":"Defines Prompt Problems and the test-case evaluation mechanism that this study uses as its measure of success.","marker":"[13]"},{"why":"Shows LLMs can generate programming exercises in non-English languages such as Tamil, Spanish, and Vietnamese, motivating the multilingual prompting approach.","marker":"[31]"},{"why":"Documents the cognitive load and barriers that English-centric programming creates for non-native speakers, establishing the problem the paper addresses.","marker":"[4]"},{"why":"Supplies the definition of low-resource languages that the paper uses to explain the lower Arabic success rate.","marker":"[45]"},{"why":"Demonstrates that prompt refinements improve LLM code-generation accuracy, providing the basis for teaching prompt crafting through Prompt Problems.","marker":"[12]"},{"why":"Shows multilingual LLM-based exercises with Indic languages are viable, giving the nearest prior evidence that multilingual support in computing education can work.","marker":"[29]"},{"why":"Surveys the capabilities of generative AI in computing education and supports the premise that LLMs solve introductory programming problems well.","marker":"[49]"},{"why":"Establishes that LLMs can generate coherent programming exercises and code explanations, a foundation for the new exercise type.","marker":"[55]"}],"fun_headline_variants":["AI writes code from prompts in Arabic, Chinese, Portuguese","Native-language prompts let non-English speakers code with AI","Prompt in your language, get code from AI","AI codes from multilingual prompts: Arabic, Chinese, Portuguese"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's explanation of the Arabic results assumes the three student groups are similar enough in educational level, programming language, and institution that the lower Arabic success rate should be attributed to the AI model's weaker Arabic support rather than to those background differences.","fun_headline_variants_meta":{"raw":{"variants":["AI writes code from prompts in Arabic, Chinese, Portuguese","Native-language prompts let non-English speakers code with AI","Prompt in your language, get code from AI","AI codes from multilingual prompts: Arabic, Chinese, Portuguese"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001864,"raw_usage":{"total_tokens":7311,"prompt_tokens":929,"completion_tokens":6382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":6318}},"tokens_in":545,"tokens_out":6382,"duration_ms":38986,"temperature":1.0,"reasoning_tokens":6318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:42:23.475632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct the same three Prompt Problems with matched groups of the same educational level, same programming language (say Python), and same institution, prompting in Arabic, Chinese, and Portuguese through the same model; if Arabic pass rates match Portuguese and Chinese, the paper's low-resource-language explanation is false.","supporting_citations":[{"cited_title":"Becker, and Brent N","cited_arxiv_id":null,"evidence_quote":"Defines Prompt Problems and the test-case evaluation mechanism that this study uses as its measure of success."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can generate programming exercises in non-English languages such as Tamil, Spanish, and Vietnamese, motivating the multilingual prompting approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the cognitive load and barriers that English-centric programming creates for non-native speakers, establishing the problem the paper addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition of low-resource languages that the paper uses to explain the lower Arabic success rate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that prompt refinements improve LLM code-generation accuracy, providing the basis for teaching prompt crafting through Prompt Problems."},{"cited_title":"Explain in Plain Language Questions with Indic Languages: Drawbacks, Affordances, and Opportunities","cited_arxiv_id":"2409.20297","evidence_quote":"Shows multilingual LLM-based exercises with Indic languages are viable, giving the nearest prior evidence that multilingual support in computing education can work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys the capabilities of generative AI in computing education and supports the premise that LLMs solve introductory programming problems well."},{"cited_title":"Patel, and Richard Halverson","cited_arxiv_id":null,"evidence_quote":"Establishes that LLMs can generate coherent programming exercises and code explanations, a foundation for the new exercise type."}],"review_version":1}