{"id":"3b56fd69-bf4e-497f-a04b-7fab2d45e9cc","arxiv_id":"2608.12905","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A large dataset of 57.5K GitHub prompts is annotated with a new ontology, enabling quantitative analysis of how transactional prompts are used in real code.","lead":"This paper collects 57,500 transactional prompts from GitHub code and turns them into a structured, searchable dataset with a new ontology. It shows how programmers write instructions for large language models, revealing patterns in languages, tasks, and prompt styles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Extraction success rate is unmeasured; if dynamic prompt construction often fails static resolution, the 57.5K corpus is a biased subset of transactional prompts, undermining the quantitative claims.","rationale":"The reader's weakest assumption identifies exactly this concern: the extraction pipeline's resolution success rate is not reported, and the collection is limited to two APIs in Python. My stress-test confirms this is the most load-bearing issue because it directly affects whether the dataset is representative enough to support the paper's empirical claims. The concern is not that the authors are hiding something; the Limitations section is candid about API/Python selection bias. Rather, the unmeasured resolution bias is a distinct, potentially larger source of bias that is not disclosed. The reader's conditional verdict already flags the need for clarification, so I do not recommend changing the verdict. A conditional acceptance with a request to report the extraction success rate and analyze failure modes would be consistent with the evidence. The ontology and the released interface are real contributions, and the error analysis, while based on a single expert and 100 points, is transparent about low-accuracy fields. Those issues affect downstream use, not the existence or basic utility of the resource. The missing resolution rate, however, is the single gap that most undercuts the generalizability of every quantitative finding in Section 5, and it is concretely testable from the released pipeline.","tokens_in":41762,"tokens_out":6064,"duration_ms":62142,"concrete_test":"Instrument the extraction pipeline to count, over all 95,806 source files (or a stratified random sample of 1,000 files), the total number of chat.completions.create and PromptTemplate call sites, and the number for which the Appendix A resolution steps produce a non-empty, non-placeholder-only prompt text. If the success rate is below, say, 80%, or if the failed call sites show a significantly higher prevalence of dynamic construction patterns (f-strings, concatenation, function-returned strings, variables defined in imported modules) than the successful ones, then the 57.5K corpus is a biased subset of transactional prompts and the Section 5 quantitative claims require re-scoping or re-analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a large, real-world collection of transactional prompts depends on the static-analysis extraction pipeline (Section 2, Appendix A) turning GitHub API call sites into actual prompt texts. The paper reports 95,806 starting files and 145,553 objects after extraction, then 57,640 unique prompts after filtering, but it never reports the resolution success rate: how many API call sites were attempted, and how many failed to yield a resolvable prompt text. The manual verification of 1,000 extracted prompts checks only the correctness of successful extractions, not the completeness or representativeness of the successful subset. If a substantial fraction of call sites build prompts dynamically (runtime f-strings, concatenation across modules, config-driven templates, or function-returned strings), those prompts are either partially resolved with placeholder-only text and then filtered out (Appendix B) or are discarded entirely. This would systematically over-represent static, literal prompts and under-represent dynamic prompt construction. The bias is consequential for the ontology-driven analysis: Section 5's variability labels (fixed vs. varying), grounding proportions (footnote 13 already concedes dynamically injected context is missed), and instruction-kind frequencies would all be skewed toward whatever patterns are visible to static resolution. The paper explicitly acknowledges the API and Python selection bias in the Limitations section, but does not acknowledge or quantify the resolution-rate bias, so the empirical distributions are presented as describing transactional prompts in general, when they may describe only the resolvable subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a large collection of 57.5K unique 'transactional prompts' extracted from public GitHub repositories that invoke either the OpenAI chat.completions.create API or the LangChain PromptTemplate constructor. It proposes a structured ontology that describes prompt languages, tasks, domains, input/output characteristics, instruction semantics, and prompting techniques; applies an LLM-based annotation pipeline to the full corpus; and reports a per-field error analysis on 100 manually reviewed prompts. The authors provide descriptive statistics over the annotated corpus (language, modality, domain, task, grounding, instruction kinds, message structure, prompting techniques) and release the dataset, annotations, and a web interface for exploration.","tokens_in":41987,"tokens_out":3651,"duration_ms":39864,"significance":"If the resource is reliable, it would be the first large-scale structured corpus of transactional prompts and a useful starting point for linguistic, empirical, and engineering studies of prompts in software. The paper ships a substantial artifact: the dataset, the ontology, the annotation prompts, a web UI, and unusually transparent per-field error tables. The ontology and the explicit error analysis are valuable contributions even if some quantitative claims later require revision. The main risk is that the extraction pipeline and the annotation quality limit the validity of the reported distributions; these are correctness concerns that can be addressed with additional measurements and more cautious wording, not fundamental flaws in the resource concept.","major_comments":[{"comment":"The extraction pipeline's resolution success rate is not reported. The paper moves from 95,806 starting files and 145,553 objects to 57,640 unique prompts, but it never states how many API call sites were attempted, how many failed to resolve to a concrete prompt text, or which failure modes dominated. The manual verification of 1,000 extracted prompts validates precision of the successful extractions, not recall or representativeness. If a substantial fraction of call sites construct prompts dynamically (runtime f-strings, cross-module concatenation, config-driven templates, function-returned strings), those prompts are either filtered out as placeholder-only or never resolved, systematically over-representing static literal prompts. This bias directly affects the central quantitative claims in Section 5, including grounding proportions, variability labels, instruction-kind frequencies, and language/task/domain distributions. Please report per-stage counts of attempted, resolved, filtered, and deduplicated items, categorize the resolution failures, and discuss how the unresolved subset might differ.","section":"§2, Appendix A"},{"comment":"The annotation accuracy is uneven and the error analysis is thin for the number of claims made. Output Type is only 60.4% accurate and Directions Text is 69.4%, while the error analysis is based on 100 data points, a single expert, and no inter-annotator reliability measure; several field-level accuracies rest on 100–150 evaluated units. The paper calls this a 'comprehensive error analysis' but does not provide confidence intervals or an explicit statement of how the low-accuracy fields should be treated by downstream users. Since the released resource includes these fields and the ontology is a central contribution, the low-accuracy fields need either improved annotation, explicit low-confidence flags, or a clearer caveat that quantitative analyses using them are provisional.","section":"§4, Table 8"},{"comment":"The grounding claim (89.25% of prompts are grounded) is vulnerable to the extraction bias acknowledged in footnote 13. The footnote concedes that dynamically injected context is not captured, and Appendix B filters out prompts that consist only of unresolved variables or placeholders. These two effects could jointly distort the grounding estimate in either direction: prompts with dynamic context are removed, while static prompts with no context are retained. Please quantify the fraction of prompts in which context is added dynamically or through unresolved placeholders, and report the grounding proportion under alternative inclusion assumptions, or explicitly mark the 89.25% figure as an upper/lower bound.","section":"§5.3, footnote 13"}],"minor_comments":[{"comment":"The 'Input characteristics' paragraph appears malformed: item (3) ends with 'for 2.' and is followed by a stray '2.', and the opening quote in '(1) overall high-level instructions (“answer the question provided by the user)' is unbalanced. Please rewrite this list so each item is complete and the quotation marks are consistent.","section":"§3"},{"comment":"The sentence 'Overall, the dataset includes 39,4875 such instruction blocks' contains an apparent typo: given 57,640 prompts and an average of 6.85 blocks per prompt, the expected total is roughly 394,000. Please verify the number and correct the digit grouping.","section":"§5.2"},{"comment":"The text says 'The following seven highly represented languages' but then lists eight languages (French, Russian, German, Indonesian, Vietnamese, Polish, Italian and Dutch). Please correct the count or the list.","section":"§5.1"},{"comment":"The caption maps both audio and video to 'AUD'; video should have its own abbreviation (e.g., 'VID'). The same issue appears in the inline key ('video=AUD').","section":"Figure 12 caption"},{"comment":"The instruction-block annotation prompt lists a fixed set of block names, but its own example uses 'task description', which is not in that list. Please align the allowed vocabulary with all examples, or explicitly state that the list is open-ended.","section":"Appendix J"}],"recommendation":"major_revision","confidential_remarks":"The reviewer's strongest concern about extraction completeness is legitimate and should be addressed with explicit resolution-rate statistics before the quantitative claims are published. The paper is within scope for cs.CL and the resource itself is valuable; I would support publication after the authors either measure the extraction gap or substantially soften the distributional claims. The low-accuracy annotation fields also need a clearer user-facing caveat. No concerns about novelty or citation practice beyond what is already acknowledged in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real contribution and I'd send it to review, but the paper overreaches slightly when it presents the Section 5 distributions as describing transactional prompts in general. The dataset itself, the ontology, and the honest error analysis are the value.\n\nWhat is genuinely new: 57.5K unique prompts extracted from GitHub via two APIs, with recursive static analysis of variable assignments and function parameters, filtering, deduplication, and a public browsing UI. That is a substantial step beyond PromptSet's raw strings, and the paper wisely avoids overclaiming novelty in the taxonomy — the ontology builds on prior work and is explicitly extensible. The annotation prompts in Appendix J are detailed enough to be reusable, and the per-field accuracy table is the right kind of transparency. Many fields are above 90%; the weak spots — Output Type at 60.4%, Directions Text at 69.4% — are flagged clearly, though buried in the appendix.\n\nThe soft spots are real but not fatal. First, the stress-test concern is valid: the paper never reports the extraction success rate. We learn 95,806 starting files become 145,553 objects and then 57,640 prompts, but not how many call sites were attempted or how many failed to resolve to readable text. The manual verification of 1,000 prompts checks correctness of successful extractions, not coverage. Dynamic construction — runtime f-strings, concatenation across modules, config-driven templates, function-returned strings — is exactly what static resolution will systematically miss, and Appendix B confirms that placeholder-only texts are filtered out. So the language, task, and domain distributions are conditioned on the resolvable subset, and that bias is not acknowledged in the Limitations beyond the API/Python selection. This matters for any downstream use of the quantitative claims; it matters less for the resource itself.\n\nSecond, the error analysis is thin: 100 points, one expert. That is enough to catch gross failures but not enough to trust fine-grained labels like Output Type at scale. The authors know this, but downstream users will need a caution sticker.\n\nThird, the comparison to PromptSet is asserted rather than measured. No overlap statistics or quantitative evidence that their extraction is cleaner. Minor, since Tafreshipour et al. supports the claim, but a table would help.\n\nWho is this for? Anyone studying prompts as linguistic objects, building prompt-optimization tools, or benchmarking structured prompt understanding. The dataset and ontology deserve a serious referee; I would not desk reject. My recommendation: accept with a required revision that reports the resolution denominator and reframes the empirical analysis as describing resolvable call sites, plus a larger multi-annotator sample for the low-accuracy fields.","headline":"Serious resource paper: a much cleaner transactional-prompt dataset than PromptSet plus a useful ontology, but the quantitative claims rest on an unmeasured extraction success rate.","tokens_in":42507,"tokens_out":2047,"would_cite":true,"duration_ms":23406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents a dataset of 57,640 unique transactional prompts from public code, an ontology that turns raw prompt strings into structured records, and an analysis finding a Zipf-like spread across languages, tasks, domains, and…","keywords":["transactional prompts","prompt ontology","prompt analysis","GitHub","LLM prompting","multilingual prompts","prompt engineering","natural language interfaces"],"falsifier":"Run the same extraction and ontology annotation on a sample of repositories that use other LLM client libraries or non-Python languages; if the distributions of languages, tasks, domains, and instruction types shift materially, then the reported Zipf-like patterns are artifacts of the two-API Python-only selection rather than properties of transactional prompts at large.","tokens_in":41568,"feed_emoji":"💬","tokens_out":7770,"duration_ms":69923,"temperature":0.7,"pith_summary":"Prompts are usually treated as ad-hoc text, but this paper argues they are linguistic objects worth systematic study. It collects 57,640 unique transactional prompts — natural-language instructions embedded in real software — by statically resolving message and template strings from Python files that call the chat.completions.create API or LangChain's PromptTemplate. It then defines an ontology with fields for language, task, domain, input/output characteristics, instruction structure, and prompting techniques, and annotates the collection with an LLM pipeline checked against manual judgment. The resulting analysis describes a Zipf-like distribution: English dominates at 84.66% of identifiable prompt text, text-to-text is the prevailing modality, question answering leads the task list, and most prompts combine a core task with supporting constraints. The dataset and a browsing interface are released to make prompts available as first-class research objects.","feed_headline":"57,500 real GitHub prompts get a structural ontology","feed_subtitle":"A new ontology maps tasks, languages, and constraints in 57.6K prompts, exposing Zipf-like patterns in how developers talk to LLMs.","key_machinery":"The carrying mechanism is the transactional prompt ontology paired with a static-analysis extraction pipeline. The pipeline locates target API calls in Python source, resolves message and template strings by recursively tracking variable assignments and function parameters across files, filters out empty or placeholder-only texts, and deduplicates; a manual check of 1,000 prompts was done before scaling. The ontology then splits each prompt into role messages and instruction blocks, labels each block with one of 42 semantic kinds, marks central and negative instructions, records input and output units with their type, structure, modality, language, and variability, and assigns prompting techniques from a fixed inventory of 12. This transforms raw prompt strings into records with enough structure to support quantitative and linguistic analysis.","core_discovery":"The central claim is that transactional prompts — prompts written to run repeatedly inside software workflows — form a distinct, structured genre of language use that can be captured in a shared ontology and studied quantitatively. The paper supports this by extracting 57,640 unique prompts from GitHub (36,916 from chat.completions.create and 20,724 from LangChain PromptTemplate), annotating each with a multi-layered ontology covering detected languages and explicit language mentions; coarse and fine task and domain; input context, directions, and question units with variability, type, structure, modality, and language; output units with type, structure, modality, language, and answer paradigm; instruction sequences with 42 semantic kinds, central vs. meta status, and negativity; and 12 prompting techniques. On this structured data it finds, among other results, that 89.25% of prompts are grounded in an input context rather than relying on parametric knowledge, that 81.8% of instruction blocks are meta instructions rather than the central task, that constraints make up 33.3% of all instruction blocks, and that the system-user message pair has become the standard format for transactional prompts.","pith_inferences":["An extension of the ontology to interactive and agentic prompts — not pursued here — would test whether the system-user standard and the observed constraint density are specific to the transactional register.","The gap between 62 used and 151 mentioned languages hints that prompts are a site of language planning; whether mentioned languages actually shape output quality is a testable consequence the paper leaves open.","Because the data is a snapshot from two APIs and Python only, the absolute frequencies are less portable than the structural ratios; re-running the pipeline on a different client library would reveal which patterns are API-specific conventions."],"forward_implications":["If prompts are structured objects, then instruction-following benchmarks can be built from naturally occurring constraints rather than synthetic templates.","The finding that 89.25% of transactional prompts are context-grounded supports the view that production LLM use is predominantly grounded in supplied context rather than parametric knowledge.","The predominance of meta instructions (81.8%) suggests that effective prompt design is mostly about control — format, constraints, roles — rather than about stating the task itself.","The Zipf-like distribution across 62 used languages and 151 mentioned languages can inform multilingual prompt engineering and evaluation.","A shared ontology makes it possible to compare prompts across repositories, tasks, and time, enabling diachronic studies of prompt evolution."],"supporting_citations":[{"why":"Supplies the standard practice for collecting prompts from GitHub repositories.","marker":"Liu et al., 2025"},{"why":"Precedent for mining large code corpora from repositories.","marker":"Li et al., 2022"},{"why":"Precedent for large-scale GitHub dataset construction.","marker":"Mir et al., 2021"},{"why":"Precedent for probabilistic analysis of code corpora.","marker":"Raychev et al., 2016"},{"why":"Provides Zipf's law as the linguistic distribution lens for the long-tail analysis.","marker":"Piantadosi, 2014"},{"why":"Supports the claim that Zipf-like distributions are typical of linguistic units.","marker":"Linders and Louwerse, 2022"},{"why":"The main prior transactional-prompt dataset (PromptSet), used as the comparison for collection method and extraction effort.","marker":"Pister et al., 2024"},{"why":"Source inventory for the 12 prompting techniques defined in the ontology.","marker":"Schulhoff et al., 2025"}],"fun_headline_variants":["57K GitHub prompts reveal a structured prompt genre","Zipf-like power laws govern real prompt usage","89% of prompts rely on context, not model memory","System-user pairs dominate today's prompt patterns","A new ontology for transactional prompts in the wild"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the static-analysis pipeline, applied only to Python files that call two specific APIs, resolves a representative sample of transactional prompts from public code.","fun_headline_variants_meta":{"raw":{"variants":["57K GitHub prompts reveal a structured prompt genre","Zipf-like power laws govern real prompt usage","89% of prompts rely on context, not model memory","System-user pairs dominate today's prompt patterns","A new ontology for transactional prompts in the wild"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3511,"prompt_tokens":991,"completion_tokens":2520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2448}},"tokens_in":607,"tokens_out":2520,"duration_ms":20285,"temperature":1.0,"reasoning_tokens":2448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:51:06.914220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same extraction and ontology annotation on a sample of repositories that use other LLM client libraries or non-Python languages; if the distributions of languages, tasks, domains, and instruction types shift materially, then the reported Zipf-like patterns are artifacts of the two-API Python-only selection rather than properties of transactional prompts at large.","supporting_citations":[],"review_version":1}