{"id":"74de510f-b69f-42fc-a34e-ad12bac02fb4","arxiv_id":"2607.06004","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs detect falsified ISO/IEC/IEEE 24765 SE definitions well but also reject many correct ones, indicating systematic rejection bias rather than genuine discriminative understanding of SE terminology.","lead":"State-of-the-art LLMs often reject correct ISO software-engineering definitions while catching falsified ones, showing a rejection bias rather than true terminology understanding. Anyone building or relying on LLM-based SE tools should treat consistent use of standard vocabulary as an open reliability risk.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"The leap from (high falsified detection + frequent correct rejection) to “rejection bias rather than genuine understanding” treats ISO 24765 entries as unique, unambiguous ground truth that any understanding model must accept verbatim; multi-definitional SE practice and prompt framing can produce th","rationale":"The reader correctly isolated the operationalization of “understanding” via binary classification against ISO 24765 ground truth (plus sensibility of reasoning tokens) as the weakest assumption. That is precisely the load-bearing hinge: the paper’s strongest claim is not merely the accuracy numbers but the interpretive step that equates the observed pattern with rejection bias rather than genuine discriminative understanding. My concern sharpens the same hinge—uniqueness of the ISO definitions and absence of shallow cues—without introducing a different failure mode. Because the paper’s empirical pattern remains interesting and practically relevant even if the interpretive leap needs qualification, and because full methods, prompts, falsified corpus, and quantitative tables are still required for a final soundness audit, the CONDITIONAL verdict with low confidence is appropriate and needs no change. The proposed expert audit of rejected correct items would directly test whether the inference lands.","tokens_in":2076,"tokens_out":626,"duration_ms":37224,"concrete_test":"Sample 50–100 stratified correct ISO items that the LLMs rejected and the corresponding reasoning traces; have 3–5 blinded SE experts (a) accept/reject each item under the same prompt and (b) judge whether each LLM rejection cites a documented alternative definition or genuine incompleteness in common SE literature versus an invented flaw. If expert rejection rates on correct items approach the LLMs’ rates, or if a substantial fraction of LLM rationales map to real alternatives, the rejection-bias interpretation is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on interpreting the observed pattern—strong detection of semantic/structural falsifications together with frequent rejection of the original ISO/IEC/IEEE 24765:2017 definitions—as evidence of systematic rejection bias rather than discriminative understanding. That interpretation requires two conditions that the study design (as described) does not secure: (1) each ISO definition is the unique, unambiguous correct statement of the term, so that a model that understands the terminology is obliged to accept it; and (2) the falsifications contain no shallow surface cues that permit detection without conceptual grasp. SE terminology is routinely multi-definitional across standards, textbooks, and practice; a model that has internalized alternative formulations may legitimately flag an ISO entry as incomplete, non-unique, or worded differently without lacking understanding. Structural deletions and key-term substitutions can also leave detectable syntactic or distributional artifacts. The additional scoring of whether reasoning tokens “make sense” introduces a further subjective layer. If many rejections of correct items cite real alternative definitions or arise from prompt-induced skepticism rather than a bias against true statements, the headline inference does not follow from the accuracy numbers alone.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript reports an empirical study of whether state-of-the-art LLMs understand standardized software-engineering terminology, operationalized as binary classification of ISO/IEC/IEEE 24765:2017 definitions presented either in their original form or after systematic semantic (key-term substitution) and structural (critical-information removal) falsification. The authors measure classification accuracy and the sensibility of generated reasoning tokens. They report that most models detect falsified definitions with high accuracy yet also reject many correct ISO definitions, and interpret this pattern as a systematic rejection bias rather than genuine discriminative understanding; explicit reasoning is found not to help consistently and may degrade performance via over-thinking. The broader claim is that impressive SE-task performance of LLMs coexists with fundamental gaps in consistent terminology use that can affect both LLM consumers and human developers acting on LLM output.","tokens_in":2353,"tokens_out":948,"duration_ms":28227,"significance":"If the central interpretation holds, the work would be a useful cautionary contribution for the SE and LLM-for-SE communities: it would show that high accuracy on negative (falsified) examples is insufficient evidence of conceptual grasp of standardized vocabulary, and that agentic or reasoning-augmented use does not automatically close the gap. Strengths visible from the design description include grounding in an external, community-standard vocabulary (ISO/IEC/IEEE 24765:2017), a dual semantic/structural falsification scheme, and joint evaluation of accuracy and reasoning-token sensibility. These elements make the study more structured than ad-hoc prompt probes. The practical stakes—miscommunication when LLMs consume or produce SE text—are real. The significance therefore hinges on whether the observed rejection pattern can be cleanly attributed to bias rather than to multi-definitional SE practice, surface cues in the falsifications, or prompt-induced skepticism.","major_comments":[{"comment":"The headline inference—that high falsified detection plus frequent rejection of correct ISO entries indicates “systematic rejection bias rather than genuine discriminative understanding”—treats each ISO/IEC/IEEE 24765:2017 entry as the unique, unambiguous statement that any model with genuine understanding is obliged to accept. SE terminology is routinely multi-definitional across standards, textbooks, and practice. Without an analysis of (a) how often rejected “correct” items have documented alternative formulations and (b) what the models’ rejection rationales actually cite, the leap from accuracy numbers to “rejection bias” is not secured. This is load-bearing for the central claim and should be addressed by coding rejection reasons and/or testing acceptance of known alternative definitions of the same terms.","section":null},{"comment":"The operationalization of “understanding” as classification accuracy on original vs. semantically/structurally falsified definitions, plus sensibility of reasoning tokens, is not shown to rule out shallow surface cues. Key-term substitutions and structural deletions can leave detectable syntactic, distributional, or fluency artifacts that permit high falsified-detection accuracy without conceptual grasp. The manuscript needs either (i) controls that preserve surface statistics while altering meaning, (ii) human baselines on the same items, or (iii) an explicit argument why the chosen falsifications cannot be detected by non-conceptual heuristics. Absent that, the contrast “detects falsified / rejects correct” underdetermines the bias-vs-understanding conclusion.","section":null},{"comment":"Scoring whether reasoning tokens “make sense with respect to understanding the definition” introduces a subjective layer that is itself part of the evidence for the central claim. The abstract does not establish inter-annotator agreement, a coding rubric, or blinding. If sensibility judgments are unstable or confounded with fluency, the claim that explicit reasoning “may even hinder performance through over-thinking” rests on a soft measure. A major revision should report agreement statistics, the coding scheme, and sensitivity of the over-thinking conclusion to alternative sensibility criteria.","section":null},{"comment":"Prompt framing and decision threshold are not shown to be neutral with respect to acceptance vs. rejection. A prompt that invites verification or skepticism can induce systematic rejection of true statements even when the model’s internal representation of the term is adequate. The manuscript should report the exact prompt templates, any temperature/decoding settings, and an ablation that varies framing (e.g., “is this a valid definition of T?” vs. “does this match the ISO definition of T?”). Without that, rejection rates on correct items cannot be cleanly attributed to model bias rather than task framing.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.5","summary":"We thank the referee for a careful and constructive review. The four major comments correctly identify load-bearing points for our central claim of systematic rejection bias rather than genuine discriminative understanding of ISO/IEC/IEEE 24765 terminology. We agree that the current manuscript does not yet secure that claim against multi-definitional SE practice, surface-cue alternatives, subjectivity in sensibility scoring, or prompt-framing confounds. We will revise accordingly: code and report rejection rationales; strengthen (and where feasible control) the falsification design; report the sensibility coding scheme with agreement statistics; and publish exact prompts, decoding settings, and a framing ablation. We retain the core experimental design and the cautionary message for SE/LLM-for-SE, but we will qualify the bias interpretation where the new analyses require it. Below we respond point by point.","responses":[{"response":"We agree this is load-bearing and not yet secured. Our operationalization intentionally anchors on ISO/IEC/IEEE 24765:2017 as the community-standard vocabulary for “standardized SE terminology,” which justifies treating its entries as the reference set for this study. That does not, however, establish uniqueness in practice, nor does it show that rejections of correct ISO entries stem from bias rather than legitimate alternative formulations or other rationales. In revision we will (1) systematically code model rejection rationales on correct ISO items (categories such as alternative definition cited, missing nuance, surface wording, over-strictness, hallucination, etc.), with a clear rubric and agreement statistics; (2) for a sample of frequently rejected terms, collect documented alternative definitions from other standards/textbooks and re-test acceptance; and (3) qualify the “rejection bias” claim in light of those results—retaining it only where rejections are not explained by multi-definitional practice or cited alternatives. We will also discuss multi-definitionality as a boundary condition on interpreting ISO-only acceptance as “understanding.”","revision_made":"yes","referee_comment":"The headline inference treats each ISO/IEC/IEEE 24765:2017 entry as the unique, unambiguous statement that any model with genuine understanding is obliged to accept. SE terminology is multi-definitional. Without analysis of (a) alternative formulations of rejected “correct” items and (b) what rejection rationales cite, the leap to “rejection bias” is not secured. Address by coding rejection reasons and/or testing acceptance of known alternative definitions."},{"response":"This is a fair underdetermination concern. Our dual scheme (key-term substitution; removal of critical information) was intended to force meaning change while preserving definitional form, but we did not demonstrate that high falsified-detection accuracy cannot arise from non-conceptual heuristics. In revision we will: (a) make an explicit, item-level argument for why the chosen falsifications alter definitional content rather than only surface form, including examples and failure modes; (b) add a human baseline on a stratified sample of original and falsified items so that model accuracy and error patterns can be compared to human discriminative performance; and (c) where feasible, introduce a limited control set that better preserves surface statistics (e.g., near-paraphrase or length/fluency-matched distractors) and report whether the detect-falsified / reject-correct pattern persists. We cannot claim a fully cue-free operationalization of “understanding,” and we will state that limitation clearly; the revised evidence will better constrain the bias-vs-understanding interpretation rather than treat accuracy alone as decisive.","revision_made":"yes","referee_comment":"The operationalization of “understanding” as classification accuracy on original vs. semantically/structurally falsified definitions, plus sensibility of reasoning tokens, is not shown to rule out shallow surface cues. Key-term substitutions and structural deletions can leave syntactic, distributional, or fluency artifacts. Need (i) controls preserving surface statistics while altering meaning, (ii) human baselines, or (iii) an explicit argument why the falsifications cannot be detected by non-conceptual heuristics."},{"response":"We agree. Sensibility of reasoning tokens is a soft measure and currently under-specified in the manuscript; the over-thinking claim must not rest on unstable or fluency-confounded judgments. In revision we will: (1) publish the full coding rubric for sensibility (criteria distinguishing conceptual engagement with the definition from fluency, generic skepticism, or irrelevant chain-of-thought); (2) report inter-annotator agreement (e.g., Cohen’s/Fleiss’ κ or percent agreement with adjudication protocol) on a double-coded sample, with blinding to model identity and to gold correctness where possible; (3) separate sensibility from mere fluency in the coding scheme; and (4) re-evaluate the “explicit reasoning may hinder / over-thinking” conclusion under the primary rubric and under at least one alternative sensibility criterion, reporting sensitivity. If agreement is low or the over-thinking effect is not robust, we will soften or retract that claim accordingly. Classification accuracy results will remain primary; sensibility will be clearly labeled as secondary and interpretive.","revision_made":"yes","referee_comment":"Scoring whether reasoning tokens “make sense with respect to understanding the definition” is subjective and part of the evidence for the central claim. The abstract does not establish inter-annotator agreement, a coding rubric, or blinding. If sensibility judgments are unstable or confounded with fluency, the claim that explicit reasoning may hinder performance via over-thinking rests on a soft measure. Report agreement, coding scheme, and sensitivity of the over-thinking conclusion."},{"response":"We agree that framing is a plausible confound for rejection rates on correct ISO entries and that the manuscript must make prompts and decoding fully transparent. In revision we will: (1) report the exact prompt templates used for all conditions (including any system messages and the decision format); (2) report temperature and other decoding settings, and whether they were held fixed; and (3) run a framing ablation that contrasts at least a verification-style prompt with a more ISO-anchored match prompt (along the lines the referee suggests), and report acceptance/rejection rates on correct vs. falsified items under each framing. We will attribute “systematic rejection bias” to the models only to the extent the pattern survives framing variation; where framing drives rejections, we will restate the finding as framing-sensitive skepticism rather than (or in addition to) intrinsic rejection bias. This directly addresses clean attribution of the central pattern.","revision_made":"yes","referee_comment":"Prompt framing and decision threshold are not shown to be neutral with respect to acceptance vs. rejection. A verification/skepticism prompt can induce rejection of true statements. Report exact prompt templates, temperature/decoding settings, and an ablation varying framing (e.g., “is this a valid definition of T?” vs. “does this match the ISO definition of T?”). Without that, rejection rates on correct items cannot be cleanly attributed to model bias rather than task framing."}],"tokens_in":2121,"tokens_out":1513,"duration_ms":27042,"standing_objections":[]},"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: LLMs catch semantic and structural falsifications of ISO/IEC/IEEE 24765 definitions at high rates, yet they also reject many of the genuine entries. That asymmetry is a concrete reliability flag for any SE workflow that depends on consistent vocabulary—requirements, V&V, multi-agent setups. The paper’s stronger claim, that this shows systematic rejection bias rather than genuine discriminative understanding, is less secure than the abstract presents it.\n\nWhat is actually new is the systematic probe itself. Treating the ISO vocabulary as a fixed test set, building negatives by key-term substitution and by stripping critical information, and scoring both classification and whether the reasoning tokens track the definition, is cleaner than the usual “ask the model to define X and eyeball it” studies. That design is a real contribution to LLM-for-SE evaluation, and the practical warning is earned.\n\nThe soft spot is load-bearing and matches the stress-test note. SE terminology is multi-definitional across standards, textbooks, and practice. A model that has internalized alternative formulations can legitimately flag a single ISO wording as incomplete or differently phrased without lacking conceptual grasp. Structural deletions and substitutions can also leave shallow surface cues. Scoring “sensibility” of reasoning tokens adds a subjective layer that needs a tight protocol. From the abstract alone we cannot see the model list, prompts, how substitutions were chosen, inter-annotator checks, or error bars, so the interpretation of the accuracy numbers as pure rejection bias is not yet locked down. Explicit reasoning not helping (or hurting via over-thinking) is a secondary observation that needs the same scrutiny.\n\nThis is for SE researchers and tool builders who care about terminology consistency in LLM pipelines. It is not a core ML result. It deserves a serious referee: the question is important, the design is concrete, and the artifacts (falsified corpus, prompts, tables) should be released and checked. I would send it to peer review and ask the authors to defend ISO uniqueness as ground truth or add a multi-definition baseline. The central pattern is worth publishing; the causal story needs tightening.","headline":"Asymmetric pattern is real and useful; the leap to “rejection bias, not understanding” over-reads ISO wording as unique ground truth.","tokens_in":2956,"tokens_out":520,"would_cite":false,"duration_ms":25274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLMs catch falsified ISO software-engineering definitions yet reject many correct ones, revealing rejection bias rather than genuine understanding of SE terminology.","keywords":["large language models","software engineering terminology","ISO/IEC/IEEE 24765","vocabulary understanding","rejection bias","definition classification","reasoning tokens","SE standards"],"falsifier":"Re-run the same correct-versus-semantic/structural-falsification protocol on a held-out sample of ISO 24765 definitions with an independent LLM family; if that family accepts authentic definitions at rates comparable to its detection of falsifications (balanced precision and recall near expert levels) and its reasoning tokens correctly isolate the critical distinguishing clauses, the rejection-bias claim is falsified.","tokens_in":2965,"feed_emoji":"🤖","tokens_out":909,"duration_ms":36037,"temperature":0.7,"pith_summary":"This paper asks whether large language models truly understand standardized software-engineering terminology drawn from ISO/IEC/IEEE 24765:2017. The authors prompt models with both the authentic definitions and systematically falsified versions created by swapping key terms or stripping critical clauses, then score binary classification accuracy and the sensibility of any reasoning tokens the models produce. Most models detect the falsified definitions at high rates, yet they also reject a substantial fraction of the genuine definitions. The authors read this pattern as a systematic rejection bias rather than discriminative knowledge of the standard vocabulary. Explicit reasoning instructions do not reliably improve the results and can degrade them through over-thinking. If the finding holds, developers and agentic systems that rely on LLMs for terminology-sensitive SE work risk silent miscommunication even when the models appear confident.","feed_headline":"LLMs catch fake SE definitions but reject many real ones","feed_subtitle":"They show rejection bias on ISO 24765 vocabulary, risking silent miscommunication in tools and teams.","key_machinery":"The controlled contrast between authentic ISO/IEC/IEEE 24765:2017 definitions and two families of systematic falsifications (semantic substitution of key terms; structural removal of critical information), evaluated by classification accuracy and by whether the model’s reasoning tokens correctly track the definitional content.","core_discovery":"State-of-the-art LLMs detect semantically and structurally falsified definitions from the ISO/IEC/IEEE 24765:2017 Systems and Software Engineering Vocabulary with high accuracy, yet they also reject many of the authentic definitions. The pattern indicates a systematic rejection bias rather than genuine discriminative understanding of standardized SE terminology; explicit reasoning does not consistently improve discrimination and may hinder it.","pith_inferences":["The same rejection bias may appear for other standardized SE vocabularies (IEEE standards, SWEBOK, etc.), pointing to a broader failure mode for domain-standard language.","Training or alignment objectives that reward cautious rejection of uncertain claims could produce the observed bias and might be mitigated by retrieval grounding against the standard itself.","Benchmarks that report only aggregate accuracy on terminology tasks will mask the imbalance; separate acceptance and rejection rates should become standard reporting practice.","A practical mitigation worth testing is to force the model to quote or retrieve the ISO definition before rendering a correctness judgment."],"forward_implications":["LLMs used on SE tasks that depend on precise terminology may silently reject valid standard definitions and produce inconsistent guidance.","Agentic or multi-step LLM pipelines that assume internal consistency of SE vocabulary inherit the same rejection bias.","Explicit chain-of-thought or reasoning prompts are not a reliable remedy and can introduce over-thinking that worsens terminology judgments.","Human developers acting on LLM-generated text that mishandles standard terms risk real-world miscommunication and defects.","Evaluations of LLMs for SE should include separate true-positive and true-negative rates on terminology probes rather than end-task accuracy alone."],"fun_headline_variants":["LLMs detect SE fakes well yet reject many authentic definitions","Rejection bias: LLMs dismiss correct ISO SE vocabulary terms","LLMs spot falsified SE definitions but reject real ones often","Models catch fake SE terms with accuracy yet fail true ones","SE terminology LLMs reject authentic defs while flagging fakes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The study treats the ISO definitions as unambiguous ground truth and treats success at binary classification of correct versus hand-crafted falsified definitions, plus sensible reasoning tokens, as a sufficient operationalization of “understanding” SE terminology.","fun_headline_variants_meta":{"raw":{"variants":["LLMs detect SE fakes well yet reject many authentic definitions","Rejection bias: LLMs dismiss correct ISO SE vocabulary terms","LLMs spot falsified SE definitions but reject real ones often","Models catch fake SE terms with accuracy yet fail true ones","SE terminology LLMs reject authentic defs while flagging fakes"]},"model":"grok-4.5","cost_usd":0.013306,"raw_usage":{"total_tokens":2805,"prompt_tokens":785,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":133060000,"prompt_tokens_details":{"text_tokens":785,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1953,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":785,"tokens_out":67,"duration_ms":21701,"temperature":1.0,"reasoning_tokens":1953,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T20:19:41.948878+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same correct-versus-semantic/structural-falsification protocol on a held-out sample of ISO 24765 definitions with an independent LLM family; if that family accepts authentic definitions at rates comparable to its detection of falsifications (balanced precision and recall near expert levels) and its reasoning tokens correctly isolate the critical distinguishing clauses, the rejection-bias claim is falsified.","supporting_citations":[],"review_version":1}