{"id":"4e3a8eff-5c8d-400b-adf3-7162ab320b84","arxiv_id":"2604.01148","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"BugScribe links incomplete Android bug reports to app UI execution context so an LLM can produce higher-quality Observed Behavior, Expected Behavior, and Steps to Reproduce.","lead":"BugScribe uses large language models plus app UI execution data to complete and correct incomplete Android bug reports. Smart generalists may care because incomplete reports waste developer time, and automating their repair could speed mobile software maintenance.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Dual use of the authors’ quality model for both context selection and scoring risks measuring self-consistency rather than independent report usefulness.","rationale":"The reader correctly isolated the dual-use quality model as the weakest assumption supporting the strongest claim. The abstract itself announces the model’s dual role, so the concern is visible without the full text. No stronger internal inconsistency is extractable from the abstract alone; sample size, baseline details, and statistical tests cannot be audited yet. Therefore the UNVERDICTED / LOW-confidence stance remains appropriate; the concrete human-ranking check would either dissolve or confirm the circularity once the full artifacts appear. No adjustment to the reader’s verdict is warranted.","tokens_in":2070,"tokens_out":444,"duration_ms":9659,"concrete_test":"Obtain the full paper’s quality-model definition and the 48-report dataset; have three independent raters (blind to source) rank original, BugScribe, and baseline components on usefulness for reproduction/diagnosis; compute rank correlation with the model scores. If Spearman ρ < 0.6 or human preference reverses the model ranking on >25 % of pairs, the claimed gains do not transfer and the dual-use concern lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (large S2R and smaller OB/EB quality gains vs. originals and three LLM baselines on 48 reports) rests on a quality model that the abstract states is used both “to identify the most effective context for each component” and, by implication, to quantify those gains. Because the same model supplies the selection criterion and the evaluation metric, any reported improvement can be an artifact of better matching the model’s internal criteria rather than producing reports that independent humans would judge more complete, accurate, or useful. The manually constructed ground truth does not break the circularity if the model itself is the scorer. Without an independent human ranking or an external metric, the 44.1 %–82.3 % S2R figures remain unanchored to the construct the paper claims to improve.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript presents BugScribe, an LLM-based approach that enhances incomplete Android app bug reports by generating Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2R). It frames enhancement as linking user-written reports to app-specific UI execution information and uses a component-specific grounding strategy that supplies the most relevant context to the LLM for each component. To design and evaluate the approach, the authors develop a bug report quality model that is used both to select effective context per component and to quantify quality. On 48 bug reports from 26 Android apps with manually constructed ground truth, BugScribe is reported to outperform the original reports and three LLM baselines, with S2R quality gains of 44.1%–82.3% and OB/EB gains of 3.8%–35.2%.","tokens_in":2247,"tokens_out":998,"duration_ms":13942,"significance":"Incomplete, ambiguous UI-centric bug reports are a genuine and recurring problem in mobile software engineering. An automated method that reliably completes OB, EB, and S2R from execution-linked context would be practically valuable for developers, testers, and issue trackers. The component-specific grounding idea and the explicit comparison against original reports plus multiple LLM baselines are useful contributions if the evaluation is shown to be independent of the design metric. The manually constructed ground truth and multi-app sample are positive steps toward a falsifiable claim, though their strength depends on how independence and statistical support are established in the full evaluation.","major_comments":[{"comment":"Abstract: The same author-developed bug report quality model is used both “to identify the most effective context for each component” and, by the evaluation design, to measure the reported quality gains. This dual use is load-bearing for the central claim (44.1%–82.3% S2R and 3.8%–35.2% OB/EB improvements). Without an independent human ranking, external usefulness metric, or ablation that scores with a metric not used for context selection, the gains may partly reflect self-consistency with the model rather than independent report quality. The manuscript needs a clearly separated evaluation path (e.g., blinded human preference, inter-rater reliability on usefulness, or an external scorer) that does not reuse the design-time quality model as the primary success metric.","section":null},{"comment":"Abstract (evaluation paragraph): Results rest on 48 reports from 26 apps with manually constructed ground truth, yet the abstract reports only percentage improvements and does not mention statistical tests, confidence intervals, or effect-size uncertainty. For a claim of large S2R gains over originals and three baselines, the evaluation must show that differences are significant and robust to report/app sampling; otherwise the headline ranges remain under-supported.","section":null},{"comment":"Abstract: “Manually constructed ground truth” is central to validating generated OB/EB/S2R, but the abstract does not state how ground truth was produced, whether multiple annotators were used, or what inter-rater agreement was achieved. If ground-truth construction was guided by the same quality model used for scoring, independence is further weakened. The full paper must document the annotation protocol and reliability measures and show that ground truth is not merely a reification of the quality model.","section":null}],"minor_comments":[{"comment":"Abstract: The three LLM-based baselines are named only as a count; naming them and briefly stating how they differ from BugScribe’s grounding strategy would make the comparison claim clearer even at abstract length.","section":null},{"comment":"Abstract: “Improving S2R quality by 44.1%–82.3% and OB/EB quality by 3.8%–35.2%” should specify the baseline of each range (original vs. each LLM baseline) so readers can interpret the spread without the full tables.","section":null},{"comment":"Abstract: The quality model is introduced as supporting both design and evaluation; a one-clause definition of what dimensions it scores (completeness, accuracy, actionability, etc.) would reduce ambiguity about the construct being optimized.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review; full text was not available. The dual-use quality-model concern is structural and already visible in the abstract, so it is appropriate as a major comment even without the full paper. If the full manuscript already separates design-time context selection from an independent human or external evaluation, the major comments can be largely discharged. Scope (cs.SE / Android bug reports) appears a reasonable fit for a software-engineering venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that BugScribe takes incomplete Android bug reports, links them to app UI execution information, and uses component-specific context to have an LLM rewrite Observed Behavior, Expected Behavior, and Steps to Reproduce. They report large S2R quality gains (44–82 %) and smaller OB/EB gains against the originals and three LLM baselines on 48 reports from 26 apps.\n\nWhat is actually new is the formulation—enhancement as connecting free-text reports to execution traces—and the per-component grounding strategy that selects different context slices for OB, EB, and S2R. Building a quality model to drive that selection is a reasonable engineering choice, and the problem itself is real: most mobile defects are visual and user reports are often thin. The pipeline looks usable for triage tooling.\n\nThe soft spot is exactly the dual use the abstract states: the same author-built quality model both chooses the “most effective context” and underpins the quality scores that produce those percentage lifts. That couples design to metric, so the gains may partly measure self-consistency rather than independent human usefulness. The manually constructed ground truth does not fully break the loop if the model is still the scorer. Sample size is modest, and the abstract gives no error bars, statistical tests, or inter-rater numbers. These are real but proportionate concerns for an abstract-only read; they do not erase the concrete system.\n\nThis is for SE people working on bug-report quality, mobile maintenance, or LLM grounding. A reading group could usefully dissect the context-selection design. It deserves a serious referee rather than a desk reject—the contribution is concrete enough and the problem practical enough. Send it out; expect reviewers to demand a clean separation between the quality model and the evaluation metric, plus independent human rankings.","headline":"Practical LLM system for completing Android bug reports via UI grounding; the dual-use quality model is the main reason the big S2R numbers need independent checks.","tokens_in":2854,"tokens_out":461,"would_cite":false,"duration_ms":13652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"BugScribe links user bug reports to Android UI execution so an LLM can rewrite Observed Behavior, Expected Behavior, and Steps to Reproduce into higher-quality components.","keywords":["bug report enhancement","Android","LLM","Steps to Reproduce","Observed Behavior","Expected Behavior","UI execution grounding","bug report quality"],"falsifier":"On a held-out set of Android bug reports with independent human quality ratings (or developer reproduction success rates), check whether BugScribe still outperforms the original reports and the same LLM baselines when scored by humans rather than by the authors’ quality model.","tokens_in":2957,"feed_emoji":"🐛","tokens_out":623,"duration_ms":4610,"temperature":0.7,"pith_summary":"Most mobile-app defects are visible on screen, yet the reports people file are often incomplete, ambiguous, or wrong—especially when the defect is UI-centric and hard for non-experts to describe. BugScribe treats automatic enhancement as the problem of connecting a free-form user report to the app’s actual UI execution. Using an LLM and a component-specific grounding strategy, it infers and rewrites three key pieces of every report: Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). The authors first build a quality model that scores how useful each component is; they then use that model both to choose the most effective execution context for each component and to measure gains. On 48 real reports from 26 Android apps with manually constructed ground truth, BugScribe produces higher-quality components than the original reports and three LLM baselines, lifting S2R quality by 44–82% and OB/EB quality by roughly 4–35%. If the method generalizes, developers and automated triage tools would receive clearer, more complete reports without forcing users to write better ones.","feed_headline":"LLM rewrites Android bug reports using app UI execution","feed_subtitle":"BugScribe lifts Steps-to-Reproduce quality 44–82% over original reports and three baselines","key_machinery":"Component-specific grounding: a strategy that selects, for each of OB, EB, and S2R, the subset of UI execution context that an accompanying quality model judges most useful for the LLM, then feeds that context into a generation prompt tailored to that component.","core_discovery":"BugScribe, an LLM-powered system that grounds each bug-report component in the most relevant app-specific UI execution information, generates more accurate, complete, and correct Observed Behavior, Expected Behavior, and Steps to Reproduce than the original user reports and three strong LLM baselines.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["BugScribe grounds LLMs in UI execution to complete Android bug reports","LLM links reports to app UI traces for accurate OB, EB, and S2Rs","Component-specific UI context lifts Android bug-report S2R quality","BugScribe rewrites incomplete Android reports from app execution data","UI-grounded LLM generates fuller Android bug reports than baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The authors’ own bug-report quality model, used both to pick the best context for each component and to score the improvements, faithfully reflects independent human judgments of report usefulness rather than simply rewarding text that matches the model’s own criteria.","fun_headline_variants_meta":{"raw":{"variants":["BugScribe grounds LLMs in UI execution to complete Android bug reports","LLM links reports to app UI traces for accurate OB, EB, and S2Rs","Component-specific UI context lifts Android bug-report S2R quality","BugScribe rewrites incomplete Android reports from app execution data","UI-grounded LLM generates fuller Android bug reports than baselines"]},"model":"grok-4.5","effort":"low","cost_usd":0.003372,"raw_usage":{"total_tokens":1126,"prompt_tokens":798,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":33720000,"prompt_tokens_details":{"text_tokens":798,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":250,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":798,"tokens_out":78,"duration_ms":2654,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T14:35:06.464574+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out set of Android bug reports with independent human quality ratings (or developer reproduction success rates), check whether BugScribe still outperforms the original reports and the same LLM baselines when scored by humans rather than by the authors’ quality model.","supporting_citations":[],"review_version":1}