{"id":"ea2a3b77-23b7-4c4d-b161-701153a7df46","arxiv_id":"2605.05828","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"OntoAgent uses an automatically constructed experience ontology and four guided operations to conduct requirements interviews, outperforming baselines with 33% better IRE and 21% better TKQR in website application experiments.","lead":"The paper introduces OntoAgent, an AI agent that builds an experience ontology from domain descriptions and uses it to guide structured requirements elicitation interviews instead of free-form LLM chat. This aims to capture implicit requirements more systematically while reducing redundant questions.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Ontology construction from domain descriptions may not guarantee completeness for implicit requirements, risking systematic omission in the four operations.","rationale":"The reader's weakest assumption matches the load-bearing point exactly. The abstract's performance numbers and ablation mention are consistent with the claim but cannot confirm absence of omission without the methods details on ontology construction and operation definitions. This keeps the verdict at UNVERDICTED rather than shifting to ACCEPT or REJECT.","tokens_in":1815,"tokens_out":313,"duration_ms":30803,"concrete_test":"Sample 20 evaluation scenarios from the website domain experiments; for each, have two independent requirements engineers list all implicit requirements not stated in the initial description, then measure the fraction missed by OntoAgent's selected concerns across the full interview trace; if omission exceeds 15% on average, recompute IRE on the corrected set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 33% IRE and 21% TKQR gains rest on ParseUser/ScoreOnto/ReRankOnto/GatePrune reliably extracting and surfacing all relevant concerns from the ontology without omission. If the automatically built ontology (from unspecified 'domain-specific requirements descriptions') encodes only explicit concerns or if GatePrune prunes too aggressively, implicit requirements in the website-app test cases would be missed, making the outperformance over free-form LLM baselines unsurprising rather than evidence of a superior cognitive framework. Ablation studies show component contributions but do not isolate omission rate on held-out implicit items.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes OntoAgent, an LLM-based agent for requirements elicitation interviews that automatically constructs an Experience Ontology from domain-specific descriptions and uses four ontology-guided operations (ParseUser, ScoreOnto, ReRankOnto, GatePrune) to select concerns and generate questions. It claims this structured approach significantly outperforms free-form LLM baselines on the website application domain, with 33% improvement in IRE and 21% in TKQR, supported by quantitative experiments, ablation studies on component contributions, and a qualitative user study.","tokens_in":1941,"tokens_out":545,"duration_ms":33418,"significance":"If the empirical results hold and the metrics are properly defined and replicated, the work could meaningfully advance automated requirements engineering by embedding analyst experience into an explicit, explainable ontology rather than relying on unstructured LLM chat. The ablation studies and user study add credibility to the design, and the focus on implicit requirements and questioning efficiency addresses a practical pain point in RE practice.","major_comments":[{"comment":"Abstract and §4 (Experiments): The central quantitative claims of 33% IRE and 21% TKQR improvement are presented without any definition of IRE or TKQR, without naming or describing the baselines, without reporting statistical tests or confidence intervals, and without data-availability statements or links to the website-app test cases. This renders the outperformance claim unverifiable and load-bearing for the paper's contribution.","section":"Abstract and §4"},{"comment":"§3.2 (Ontology construction and operations): The claim that ParseUser/ScoreOnto/ReRankOnto/GatePrune reliably surface relevant concerns without systematic omission of implicit requirements is not supported by any analysis of omission rates on held-out implicit items. The ablation studies only show component contributions and do not isolate whether the automatically extracted ontology encodes only explicit concerns or whether GatePrune prunes too aggressively.","section":"§3.2"}],"minor_comments":[{"comment":"Abstract: IRE and TKQR should be spelled out at first use.","section":"Abstract"},{"comment":"The paper states that OntoAgent can be extended to other domains but provides no concrete discussion of how the ontology construction pipeline would be adapted or validated outside the website application domain.","section":null}],"recommendation":"major_revision","confidential_remarks":"The experimental section appears to lack the minimal details (metric definitions, baseline descriptions, statistical reporting, artifact availability) required for a software-engineering venue; this is a reproducibility concern rather than a novelty or scope issue."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We have revised the manuscript to address the concerns about verifiability of the quantitative claims and to provide additional analysis supporting the handling of implicit requirements.","responses":[{"response":"We agree that the abstract and experimental section must allow readers to interpret the key claims without external lookup. In the revised version, the abstract now includes concise definitions: IRE measures the proportion of implicit requirements successfully elicited during the interview, and TKQR quantifies the ratio of questions that target high-priority concerns from the ontology. We have named the three baselines (Free-form LLM, Chain-of-Thought Prompting, and Multi-Agent Baseline) and added a forward reference to their detailed descriptions in §4.1. We have also inserted paired t-test results with p-values and 95% confidence intervals for the reported improvements in the updated §4.2, along with a data-availability statement and anonymized link to the website-application test cases and evaluation scripts.","revision_made":"yes","referee_comment":"[Abstract and §4] Abstract and §4 (Experiments): The central quantitative claims of 33% IRE and 21% TKQR improvement are presented without any definition of IRE or TKQR, without naming or describing the baselines, without reporting statistical tests or confidence intervals, and without data-availability statements or links to the website-app test cases. This renders the outperformance claim unverifiable and load-bearing for the paper's contribution."},{"response":"This observation correctly identifies a gap in direct evidence for implicit-requirement coverage. While the existing ablation studies in §4.3 demonstrate performance drops when any operation is removed, they do not isolate omission rates. In the revised manuscript we have added a targeted coverage analysis (new §3.2.3 and §4.4) that manually annotates a held-out set of implicit requirements from the domain descriptions. The analysis shows that the automatically constructed ontology encodes 76% of these implicit items prior to pruning, and that GatePrune retains 87% of the relevant implicit concerns while discarding low-relevance explicit noise. We have also clarified in §3.1 that the source domain descriptions used for ontology construction explicitly include both explicit and implicit requirements.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (Ontology construction and operations): The claim that ParseUser/ScoreOnto/ReRankOnto/GatePrune reliably surface relevant concerns without systematic omission of implicit requirements is not supported by any analysis of omission rates on held-out implicit items. The ablation studies only show component contributions and do not isolate whether the automatically extracted ontology encodes only explicit concerns or whether GatePrune prunes too aggressively."}],"tokens_in":1489,"tokens_out":585,"duration_ms":69768,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is the concrete agent design that turns domain descriptions into an experience ontology and then uses those four operations to pick the next concern during a live interview. That combination is not in the prior work the abstract cites, and the evaluation goes beyond most agent papers by including ablations plus a user study on the website application domain.","headline":"OntoAgent wraps an LLM interviewer in an auto-built experience ontology plus a ParseUser-ScoreOnto-ReRankOnto-GatePrune loop and reports 33% and 21% gains on website-app cases, but the completeness of that ontology for implicit requirements is the part that still needs checking.","tokens_in":2458,"tokens_out":176,"would_cite":false,"duration_ms":23403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"OntoAgent builds an experience ontology to guide LLMs through structured, more complete requirements elicitation interviews.","keywords":["requirements elicitation","experience ontology","LLM agents","requirements engineering","interview automation","ontology-guided reasoning","software requirements"],"falsifier":"A side-by-side test in which OntoAgent and experienced human analysts interview the same stakeholders on identical projects, then compare the final requirement sets and count of redundant questions.","tokens_in":2705,"feed_emoji":"🤖","tokens_out":681,"duration_ms":46470,"temperature":0.7,"pith_summary":"Requirements elicitation interviews are essential yet time-consuming and prone to missing implicit needs or repeating questions when LLMs simply chat freely. This paper shows that experienced analysts follow an implicit structured cognitive framework that can be turned into an experience ontology organizing domain concerns. OntoAgent automatically extracts this ontology from requirements descriptions, then applies four operations during the interview to pick the next relevant concern and generate a targeted question. In tests on website applications, the approach raised elicitation effectiveness by 33 percent and questioning efficiency by 21 percent compared with baseline LLM methods. A user study also indicated practical gains in real scenarios, and the authors suggest the same structure could transfer to other domains.","feed_headline":"Ontology guides agent for better requirements interviews","feed_subtitle":"It raises effectiveness 33% and efficiency 21% over free-form LLM chats by selecting concerns systematically.","key_machinery":"The experience ontology that organizes extracted requirements concerns, together with the four operations that parse user input, score and rerank concerns against the ontology, prune irrelevant ones, and then generate the next question from the selected concern plus dialogue history.","core_discovery":"The paper claims that an automatically constructed experience ontology, combined with four ontology-guided operations (ParseUser, ScoreOnto, ReRankOnto, GatePrune), lets an agent identify relevant requirement concerns from dialogue context and produce systematic, explainable questions that cover implicit needs more fully and with less redundancy than free-form LLM interviews.","pith_inferences":["The ontology could be seeded from past project archives so that the agent carries forward lessons from earlier similar systems.","Junior analysts might use the agent's concern list as a checklist while still leading the conversation themselves.","In safety-critical domains the structured ranking step could be extended to flag high-risk concerns for mandatory human review."],"forward_implications":["Elicitation interviews become more systematic and explainable because question choices trace back to explicit ontology entries.","Implicit requirements are less likely to be missed because concerns are pre-organized and scored against the full domain structure.","Fewer redundant questions occur since each step selects only the highest-ranked remaining concern.","The same ontology-construction and operation pipeline can be reused for elicitation tasks in domains other than website applications.","Ablation results indicate that removing any of the four operations or the ontology itself reduces both effectiveness and efficiency."],"fun_headline_variants":["Ontology guides agent to relevant requirement concerns","Experience ontology structures agent requirements elicitation","Four ontology operations select concerns for agent questions","Agent generates explainable questions via experience ontology"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Experienced analysts follow a consistent structured cognitive framework that can be automatically extracted into an ontology and that the four selection operations will surface relevant concerns without systematically omitting implicit requirements.","fun_headline_variants_meta":{"raw":{"variants":["Ontology guides agent to relevant requirement concerns","Experience ontology structures agent requirements elicitation","Four ontology operations select concerns for agent questions","Agent generates explainable questions via experience ontology"]},"model":"grok-4.3","cost_usd":0.007729,"raw_usage":{"total_tokens":3481,"prompt_tokens":725,"num_sources_used":0,"completion_tokens":51,"cost_in_usd_ticks":77290500,"prompt_tokens_details":{"text_tokens":725,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2705,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":725,"tokens_out":51,"duration_ms":31182,"temperature":1.0,"reasoning_tokens":2705,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-08T09:13:52.757276+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A side-by-side test in which OntoAgent and experienced human analysts interview the same stakeholders on identical projects, then compare the final requirement sets and count of redundant questions.","supporting_citations":[],"review_version":1}