{"paper":{"title":"When Relevant Code Is Wrong: Runtime-Aware Context for AI Coding Agents","arxiv_id":"2610.38800","primary_cat":"cs.SE","submitted_at":"2026-10-28T20:05:15+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":1.0,"id":"s0","own":false,"role":"background","text":"AI coding agents increasingly rely on retrieved code and development history to guide software changes.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.04,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":1.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.72,"id":"s1","own":false,"role":"background","text":"However, highly relevant artifacts may describe implementations that are inactive or incompatible with the target deployment.","hedge":{"p":0.86,"label":"hedged_possibility"},"own_p":0.07,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.28,"method":0.0,"result":0.0,"background":0.72,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":0.92,"id":"s2","own":false,"role":"gap","text":"Neither semantic similarity nor recency establishes whether retrieved information remains applicable under feature flags, architectural migrations, and deployment-specific configurations.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.12,"main_claim_p":0.07,"quantitative":false,"quantitative_p":0.09,"role_probabilities":{"gap":0.92,"method":0.0,"result":0.0,"background":0.08,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":3,"p":1.0,"id":"s3","own":true,"role":"contribution","text":"We challenge relevance-oriented retrieval by introducing operational context validity, a task-, deployment-, and time-dependent criterion for determining whether software evidence can legitimately guide implementation decisions.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.78,"main_claim_p":0.21,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":1.0,"interpretation":0.0}},{"i":4,"p":0.55,"id":"s4","own":true,"role":"contribution","text":"We instantiate this new retrieval principle in CASE, an agent-agnostic framework that connects repository artifacts with runtime configurations and architectural evolution, classifies evidence as valid, invalid, or unknown, and generates context packages distinguishing actionable guidance from historical rationale and unresolved constraints.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.1,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.45,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.55,"interpretation":0.0}},{"i":5,"p":0.85,"id":"s5","own":true,"role":"method","text":"We investigate its feasibility through a reproducible preliminary evaluation involving six development tasks, 15 artifacts, and three deployments.","hedge":{"p":0.93,"label":"asserted"},"own_p":0.95,"main_claim_p":0.03,"quantitative":true,"quantitative_p":0.95,"role_probabilities":{"gap":0.0,"method":0.85,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.15,"interpretation":0.0}},{"i":6,"p":1.0,"id":"s6","own":true,"role":"result","text":"Conventional retrieval exposes 16 incompatible artifact occurrences as actionable, whereas CASE excludes them while retaining all applicable artifacts under the prototype's reference rules.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.93,"main_claim_p":0.17,"quantitative":true,"quantitative_p":0.97,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":7,"p":1.0,"id":"s7","own":true,"role":"result","text":"Recency filtering reduces valid recall to 0.43, while removing feature-flag evidence from CASE reduces it to 0.13.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.08,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":8,"p":0.75,"id":"s8","own":true,"role":"interpretation","text":"Although these results demonstrate feasibility rather than improvements in coding-agent correctness, they reveal limitations of relevance-based retrieval and motivate a new research direction: enabling coding agents to reason not only about which information is relevant, but also whether it remains operationally valid.","hedge":{"p":0.9,"label":"asserted"},"own_p":0.88,"main_claim_p":0.82,"quantitative":false,"quantitative_p":0.13,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.09,"future_work":0.16,"contribution":0.0,"interpretation":0.75}}]},"compiled":{"main_claim":{"p":0.95,"text":"Although these results demonstrate feasibility rather than improvements in coding-agent correctness, they reveal limitations of relevance-based retrieval and motivate a new research direction: enabling coding agents to reason not only about which information is relevant, but also whether it remains operationally valid.","location":"abstract s8","hedge_strength":{"p":0.9,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":6908,"cost_usd_reported":0.0},"compiled_at":"2026-10-30","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.1,"states_limitation":0.97,"human_participants":0.08,"controlled_comparison":0.78,"randomized_assignment":0.02,"states_quantitative_result":0.98,"compares_with_prior_results":0.93,"mentions_code_or_data_availability":0.09},"evidential_basis":{"p":0.99,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.99,"simulation_or_computation":0.01}},"contribution_type":{"p":1.0,"value":"method or tool","probabilities":{"review":0.0,"method_or_tool":1.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":0.0,"position_or_commentary":0.0}},"contribution_statement":{"p":0.7,"text":"We challenge relevance-oriented retrieval by introducing operational context validity, a task-, deployment-, and time-dependent criterion for determining whether software evidence can legitimately guide implementation decisions.","location":"abstract s3"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":9,"sentences_in_abstract":9},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}