{"id":"dfa95791-6128-4198-91f3-67a0721b5973","arxiv_id":"2505.18444","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Closed-category words in identifier names carry behavioral meaning, with patterns that correlate with code context and system domain.","lead":"This paper studies how rare grammatical function words, such as prepositions, conjunctions, determiners, and numerals, are used inside software identifier names, and introduces a manually labeled dataset of 1,275 identifiers from 30 open-source projects. It reports that these words are used deliberately to encode behavior like control flow, data transformation, and timing, with usage varying by code context and system domain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ1 axial codes are validated only by inter-rater agreement among co-trained annotators; without external evidence of developer intent or reader comprehension, the central claim that closed-category terms are 'deliberate, behaviorally meaningful' rests on shared inference, not demonstrated meaning.","rationale":"The reader's weakest assumption identifies exactly this gap: the behavioral axial codes are treated as ground truth about developer intent, validated only by inter-rater agreement rather than external measures. My independent reading agrees; this is the single most load-bearing concern because the paper's headline claim is a semantic/cognitive claim ('deliberate, behaviorally meaningful'), and the only evidence for that semantics is the annotators' shared interpretation. If the axial codes are plausible projections rather than recovered intent, then the RQ1 taxonomy, the selective codes, and the RQ2 explanations built on those codes have no independent grounding. The high kappa does not resolve this, because co-trained annotators with access to source context can converge on plausible post-hoc narratives. Credit is due for the carefully constructed CCID dataset, the transparent annotation process, and the significant context correlations in RQ2; those survive as descriptive findings. But the leap from 'annotators can consistently categorize these names' to 'developers deliberately use these terms to convey behavior' requires external validation. The paper's own future-work section acknowledges this need. Therefore I do not move the verdict: CONDITIONAL remains appropriate, and the condition should require either softening the central claim or adding a validation study of the kind proposed in the concrete test.","tokens_in":34549,"tokens_out":7721,"duration_ms":68832,"concrete_test":"Run a pre-registered comprehension-style experiment on a stratified random sample of 80-120 CCID identifiers. For each identifier, show the original name plus its code context to one condition, and a matched variant in which the closed-category token is replaced by a neutral placeholder (e.g., 'waitForReload' becomes 'waitXReload', 'lastBucket' becomes 'XBucket') to another condition. Ask developers to select which of the paper's axial codes best describes the identifier's role. If accuracy with original names is not significantly above chance and not significantly higher than with placeholders, the axial codes are not recoverable behavioral content, and the central 'deliberate/meaningful' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion (Section 10) is that closed-category terms are 'deliberate, behaviorally meaningful tools in the software naming arsenal'. The only evidence for 'behaviorally meaningful' is the RQ1 grounded-theory coding (Section 5.1): four annotators, after a joint calibration session, assign open codes and group them into axial codes, with Fleiss' kappa 0.916-1.0. High agreement shows the coding scheme is consistently applied by people who trained together and saw the same source context; it does not show the codes correspond to what developers intended or what readers reliably recover. No external criterion is offered: no developer interviews, no commit-intent check, no comprehension experiment, no code-behavior verification (e.g., whether 'Boolean Flow' variables actually guard branches). Because the selective codes are then used to explain the significant context correlations in RQ2 and to select domain-specific systems in Table 13, the interpretive superstructure inherits this vulnerability. The paper itself, in Section 10 future work, calls for 'further validation of the behavioral categories ... especially through comprehension studies', conceding the missing validation. The Table 6/1001 mismatch and the uncorrected threshold sweep are real but secondary; the epistemic status of the axial codes is the load-bearing issue for the paper's headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies closed syntactic category words (prepositions, determiners, conjunctions, numerals) in identifier names. It constructs the Closed Category Identifier Dataset (CCID) by sampling 1,275 candidate identifiers from 30 open-source C/C++/Java systems and manually verifying and annotating 1,001 of them with part-of-speech grammar patterns. RQ1 applies a grounded-theory-inspired coding procedure (open, axial, and selective coding) to a subset of 618 identifiers following the top-10 grammar patterns, yielding behavioral taxonomies for each category (e.g., Type Casting / Interpretation, Boolean Flow / Control Flag, Temporal / Most Recent Element) and a cross-category synthesis. RQ2 tests correlations between closed-category tag usage and programming language, source-code context, and system domain, using Pearson chi-square tests with standardized residuals and Mann-Whitney U tests over a support-threshold sweep. The paper concludes that closed-category terms are deliberate, behaviorally meaningful components of identifier names and derives implications for naming tools, education, and future research.","tokens_in":34797,"tokens_out":8174,"duration_ms":66326,"significance":"The contribution is valuable: the CCID is a new manually annotated dataset, and the paper provides one of the first detailed qualitative taxonomies of closed-category terms in identifiers. The high inter-rater agreement (Fleiss' kappa 0.916-1.0) supports the reliability of the annotation scheme as applied, and the authors ship scripts and data for reproducibility. The paper is also well situated in prior grammar-pattern and part-of-speech-tagging work, and the RQ2 context correlations, if reproducible, offer concrete hypotheses about naming conventions. However, the significance is conditional: the dataset counts are internally inconsistent, and the central claim that closed-category terms are 'deliberate, behaviorally meaningful' rests on interpretive coding without external validation of developer intent or reader comprehension.","major_comments":[{"comment":"The paper's headline conclusion—that closed-category terms are 'deliberate, behaviorally meaningful tools'—rests entirely on behavioral axial codes assigned by four co-trained annotators who examined identifiers in their source context. Fleiss' kappa shows the coding scheme was applied consistently, but it does not show that the codes capture developers' actual intent or that readers reliably recover the same meanings; the coding step is an interpretation, not a measurement of intent. The paper itself concedes in §10 that 'further validation of the behavioral categories ... especially through comprehension studies' is needed. Since RQ2's domain analysis (§6.1.3, Table 13) then uses these axial codes to select systems, the interpretive superstructure inherits this validity gap. Please either add an external validation (e.g., a developer survey, commit-intent analysis, or a comprehension experiment) or rewrite the conclusions and RQ2 framing to treat the behavioral codes as hypotheses rather than established developer intent.","section":"§5.1, §10"},{"comment":"The size and composition of the CCID are presented inconsistently. The abstract and §4.1 describe the dataset as 1,275 identifiers; the text later states that the final CCID consists of 1,001 verified identifiers; and Table 6 reports a 'Balanced population' of 1,275 while being introduced as 'the CCID, but broken down by program context.' Table 5 also shows verified P totals (382) exceeding candidate P totals (364), which is impossible if verified identifiers are a subset of the candidate pool, and the note about tag-level versus identifier-level counting does not resolve the discrepancy. Because §6.1.2 says the RQ2 chi-square analyses use all 1,001 identifiers, the context totals in Tables 5 and 6 must be reconciled and clearly labeled as pre-verification candidates, verified identifier-level counts, or tag-level counts. Without this, the counts underlying Table 11 cannot be reproduced or checked.","section":"§4.1, Tables 5 and 6"},{"comment":"Several axial codes are inferred from very few examples: Mathematical / Constraint Context has 2 items (§5.3); Quantity Threshold / Optional Extensibility has 4 items and Default / Fallback Value Representation has 2 items (§5.4); and Shared Interface for Alternatives, Combined Configuration / UI Concept, Boolean Concept Name, and Boolean Multi-Condition Test each have 1 item (§5.5). The selective-coding synthesis in §5.6 nevertheless presents these as recurring behavioral roles and uses them to support the general conclusion. For categories with single-digit counts, the paper should present them as preliminary observations requiring larger samples, or visibly down-weight them in the cross-category conclusions; the current wording overstates the support these sparse codes provide.","section":"§5.3–§5.5"},{"comment":"The domain comparison is not as strong as the text suggests. The domain-specific systems in Table 13 were selected using the paper's own axial codes, so the comparison partly re-tests the taxonomy's internal assumptions. In addition, the Mann-Whitney threshold sweep is performed at many thresholds without multiple-comparison correction, which inflates the chance of finding 'significant' peaks; the tests also use predefined closed-term lexicons without verifying the grammatical role of each term in context, as the authors acknowledge in §6.1.3. The claim that 'domain-specific systems use closed-category terms more frequently than general-purpose ones' should be labeled exploratory, or the analysis should be augmented with a correction (e.g., Bonferroni or permutation-based) and with a domain selection that is not derived from the same coding that motivates the studied categories.","section":"§6.1.3, Figures 2–4"}],"minor_comments":[{"comment":"The term 'CCID corpus' is used for the 279,000-identifier source set and later 'CCID' is used for the 1,001-identifier annotated dataset; rename the source corpus (e.g., 'source corpus') to avoid conflating the two artifacts.","section":"§4.1"},{"comment":"The m34 example first says 'matrix row 3 col 3' but then explains that 3 refers to the row index and 4 refers to the column; align the example with the intended interpretation.","section":"§5.2"},{"comment":"Typos should be corrected: §8 'simialar' → 'similar'; §10 'beahvoral' → 'behavioral'; §5.2 'collissions' → 'collisions'; several headers and table entries contain stray spaces (e.g., 'T ags', 'T echnology', 'V ersion').","section":"§8, §10, §5.2"},{"comment":"The raw counts used in the chi-square tests should be reported or linked unambiguously to a single reconciled dataset table, since Table 11's df=12 implies a 4×5 table and the residual values depend on which context totals (Table 5 or Table 6) were used.","section":"§6.1.2, Tables 9–12"},{"comment":"The threshold-sweep figures would be easier to interpret with the number of retained systems at each threshold and ideally pointwise confidence intervals; currently the reader cannot tell how quickly the system sample shrinks at high support thresholds.","section":"§6.1.3, Figures 2–4"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical contribution that fits the journal's scope, and the dataset is potentially reusable by the community. My main reservations are internal consistency of the dataset counts and the gap between interpretive coding and the strong intentionality claims. I would expect a revision to fix the Table 5/6 contradictions, soften the conclusions about developer intent, and reframe the domain comparison as exploratory unless a multiple-comparison correction and a less circular domain-selection procedure are supplied. Please also verify at the production stage that the linked data and scripts actually reproduce the reported 1,001-identifier totals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you work on identifier naming or software linguistics. It is the first systematic look at prepositions, determiners, conjunctions, and numerals inside identifiers, and it ships a new manually annotated dataset (CCID) plus a coding scheme. The descriptive core is sound: the PoS annotation is careful, inter-rater agreement is high, and the recurring grammar patterns (P N, DT N, N D, etc.) do line up with plausible behavioral roles like control flow, transformation, and sequence position. The dual-axis framework for numerals is a genuine improvement over the authors' earlier digit study, and the context correlations (prepositions in function names, numerals in parameters/classes) are believable and useful.\n\nThe soft spots are real but mostly second-order. The Table 6/1001 discrepancy is sloppy: Table 6 shows 1275 total and is labeled as the CCID, while the text and Section 9 say the verified CCID is 1001 identifiers. That needs a correction, as does the data repository URL, which as printed contains spaces (\"closed category emse analysis scripts\") and will not resolve. The domain comparison in RQ2 is the weakest section. The domain-specific systems in Table 13 were selected using the authors' own axial codes, so the Mann-Whitney results partly test the coding scheme rather than an independent hypothesis. The paper calls this exploratory, and the effect sizes are small, so I would not call it fatal, but it should be presented as a pilot.\n\nThe stress-test note is on target about the axial codes. Four co-trained annotators agreeing at kappa 0.916–1.0 shows the coding scheme is applied consistently; it does not establish that the codes reflect what developers intended or what readers recover. Yet this is not a fatal flaw either. The taxonomy is still a reasonable descriptive framework, and the paper itself concedes in Section 10 that comprehension studies are needed. I would simply soften the conclusion from \"deliberate, behaviorally meaningful tools\" to \"consistent, interpretable naming patterns that warrant experimental validation.\"\n\nThe uncorrected threshold sweep in Figures 2–4 is a minor issue; it is exploratory and the effect sizes are modest.\n\nWho gets value from this? Researchers studying naming conventions, PoS tagging of code, and program comprehension. The CCID is worth having even if some interpretations are disputed. I would bring it to a reading group and would cite it if I worked in this area.\n\nRecommendation: send it to peer review. A good referee can push for corrected data reporting, a working URL, and claims that match the evidence. Conditional accept after revision.","headline":"A solid, first-of-its-kind empirical study of closed-category words in identifiers, with a reusable dataset and a credible descriptive taxonomy, but the strongest behavioral claims outrun the evidence.","tokens_in":35398,"tokens_out":1866,"would_cite":true,"duration_ms":18249,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Closed-category words in code names carry deliberate behavioral meaning, a new study finds.","keywords":["identifier naming","part-of-speech tagging","closed-category terms","grammar patterns","program comprehension","grounded theory","software linguistics","naming conventions"],"falsifier":"A comprehension experiment where developers are shown alternative names for the same variable (e.g., textures vs find all textures) and asked to describe the program's behavior would settle the core claim: if closed-category terms do not measurably improve accuracy or speed of behavioral inference, their deliberate status is weakened. A simpler check is whether the axial-code framework itself is unstable: asking a second independent team to re-code the CCID identifiers should reproduce high agreement (near the reported Fleiss' kappa of .916–1.0) on the behavioral categories; if the categories diverge substantially, the grounded theory is not robust.","tokens_in":34323,"feed_emoji":"🔤","tokens_out":1442,"duration_ms":14487,"temperature":0.7,"pith_summary":"This paper argues that the small, fixed set of grammatical words programmers sometimes include in identifier names—prepositions, determiners, conjunctions, and numerals—are not accidental filler but deliberate tools for encoding program behavior. Using a new manually annotated dataset of 1,275 identifiers drawn from 30 open-source systems, it shows that these terms systematically express ideas such as control flow, data transformation, timing, and entity selection. The study matters because naming research has focused almost entirely on nouns and verbs; this work gives the first empirical account of how the rare closed-category words behave and what role they play in program comprehension.","feed_headline":"Small words in code names carry deliberate meaning","feed_subtitle":"A study of 1,275 identifiers shows prepositions, determiners, and numerals encode behavior, not noise.","key_machinery":"The central object is the grammar pattern: a sequence of part-of-speech (PoS) tags abstracting an identifier's phrasal structure, e.g., P N for \"to index\" or N P N for \"action to index\". The paper constructs the Closed Category Identifier Dataset (CCID), a manually annotated corpus that deliberately oversamples identifiers containing preposition, determiner, conjunction, and numeral tags, and uses grounded-theory-style open, axial, and selective coding to link these patterns to behavioral roles. The grammar pattern is what allows the analysis to generalize across surface names and reveals whether a closed-category term forms a unary relation (modifying a single noun) or a binary relation (connecting two operands).","core_discovery":"The central claim is that closed-category terms (prepositions, determiners, conjunctions, numerals) in identifier names are deliberate and behaviorally meaningful, serving functions that map onto program semantics: prepositions express transformation, direction, conditionality, and event triggers; determiners encode position in a sequence, population membership, and exclusion; numerals act as compact indices, version markers, or domain-specific codes; conjunctions encode guarded actions, data pairs, and logical alternatives. The paper demonstrates this through grounded-theory-inspired coding of 1,001 verified identifiers, revealing recurring axial codes such as Type Casting/Interpretation, Boolean Flow, and Temporal/Upcoming Element, and statistically significant correlations with code context: prepositions and conjunctions appear disproportionately in function names, numerals in parameters and class names, and determiners rarely in class names.","pith_inferences":["If closed-category terms are as meaningful as this study argues, then renaming analyses could treat a change from e.g. lastNode to previousNode as semantically near-synonymous, while a change from lastNode to someNode shifts the identifier's behavioral role—an insight that could sharpen automated refactoring recommendations.","The dual-axis framework for numerals (Role x Source of Meaning) hints that numeral-heavy identifiers such as m33 are only comprehensible to developers who share a local convention; tools that detect undocumented numerals and prompt for a comment or a rename could reduce comprehension barriers in domains with heavy matrix or coordinate usage.","The paper's own domain-selection logic suggests a testable extension: comprehension experiments comparing identifiers with and without closed-category terms (e.g., findTextures vs find all textures) could measure whether the behavioral specificity actually improves recall and understanding, a claim the paper asserts but does not test."],"forward_implications":["Naming tools and linters could use grammar patterns to flag unusual or context-mismatched closed-category usage, prompting developers to reflect on their naming choices.","Automated code generation and large language models could be scaffolded to emit identifiers that instantiate common closed-category patterns (e.g., DT NM N, P N), better aligning generated names with human conventions.","Educators could teach closed-category terms as part of a naming lexicon, giving students behavior-specific guidance beyond generic advice like 'use descriptive names'.","The axial-code framework provides a schema for studying how naming choices correlate with code quality, comprehension, and maintainability in follow-up studies.","The finding that determiners like next and last serve a determinative rather than adjectival function suggests that standard PoS tagging of identifiers should treat sequence-positions as determiners for better semantic analysis."],"supporting_citations":[{"why":"Supplies the original grammar-pattern dataset and PoS-annotation approach that CCID extends, and the comparison distribution showing closed-category terms were rare in prior data.","marker":"[55]"},{"why":"Provides the ensemble PoS tagger that the authors found too slow and under-trained for closed categories, motivating manual annotation.","marker":"[57]"},{"why":"srcML is the infrastructure used to extract identifiers from the 30 systems before sampling.","marker":"[20]"},{"why":"Spiral is the identifier-splitting tool whose splits annotators corrected during manual PoS annotation.","marker":"[42]"},{"why":"Reaper is the curation tool that selected the 30 engineered open-source systems used to build the corpus.","marker":"[52]"},{"why":"Provides the earlier single-axis framework for numerals that this paper extends into the dual-axis Role x Source-of-Meaning framework.","marker":"[65]"},{"why":"Shows that test-name grammar patterns differ from production names, justifying the exclusion of test identifiers from the sample.","marker":"[61]"}],"fun_headline_variants":["Prepositions in code names encode behavior, not noise","Closed-category words in identifiers carry semantic weight","Tiny words in code names pack behavioral meaning","Code identifiers' small words map to program behavior","How prepositions and numerals shape code meaning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The behavioral axial codes—categories like Type Casting and Boolean Flow—are treated as facts about developer intent even though they were inferred by four annotators reading identifier names and surrounding code, validated only by inter-rater agreement rather than by any direct measure of what programmers actually meant.","fun_headline_variants_meta":{"raw":{"variants":["Prepositions in code names encode behavior, not noise","Closed-category words in identifiers carry semantic weight","Tiny words in code names pack behavioral meaning","Code identifiers' small words map to program behavior","How prepositions and numerals shape code meaning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001187,"raw_usage":{"total_tokens":4874,"prompt_tokens":894,"completion_tokens":3980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":3909}},"tokens_in":510,"tokens_out":3980,"duration_ms":24937,"temperature":1.0,"reasoning_tokens":3909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:30:43.665080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A comprehension experiment where developers are shown alternative names for the same variable (e.g., textures vs find all textures) and asked to describe the program's behavior would settle the core claim: if closed-category terms do not measurably improve accuracy or speed of behavioral inference, their deliberate status is weakened. A simpler check is whether the axial-code framework itself is unstable: asking a second independent team to re-code the CCID identifiers should reproduce high agreement (near the reported Fleiss' kappa of .916–1.0) on the behavioral categories; if the categories diverge substantially, the grounded theory is not robust.","supporting_citations":[{"cited_title":"Journal of Systems and Software 170, 110740 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the original grammar-pattern dataset and PoS-annotation approach that CCID extends, and the comparison distribution showing closed-category terms were rare in prior data."},{"cited_title":"In: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp","cited_arxiv_id":null,"evidence_quote":"srcML is the infrastructure used to extract identifiers from the 30 systems before sampling."},{"cited_title":"In: Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering, NLBSE ’22, p","cited_arxiv_id":null,"evidence_quote":"Provides the earlier single-axis framework for numerals that this paper extends into the dual-axis Role x Source-of-Meaning framework."}],"review_version":1}