{"id":"5f983f5e-a4d0-4970-9749-021020d1e9de","arxiv_id":"2507.10062","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Gemma3 vision-language models can label UI snapshot test failure causes with 84% recall on a small synthetic iOS dataset, but prompt-based selective ignore is unreliable.","lead":"LLMShot points a vision-language model at the three images generated by a failing snapshot test, the baseline, the new render, and the pixel diff, and asks it to name what changed, such as color, spacing, or text. On a new 17-case iOS dataset the Gemma3-12B model reached about 84% recall, but an attempted ignore-category prompt extension failed badly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 84.21% recall / 66.67% precision headline is computed on the same 17 cases used to iteratively design the prompt, with no held-out split and no baseline; until replicated on an independent app or a frozen-prompt holdout, the performance claim may be optimistic.","rationale":"The paper's central claim is that LLMShot, a VLM prompt, classifies snapshot test failure root causes with 84.21% recall and 66.67% precision (Gemma3-12B), and that this is the first automated semantic analysis of snapshot failures. For that claim to hold, the 17-case evaluation must be a fair estimate of performance on unseen snapshot failures. The most load-bearing assumption is therefore not the taxonomy's completeness but the independence of the evaluation from the design process. Section III-A's 'iterative design process' for the prompt, combined with Section IV-A's author-constructed and author-labeled dataset, means the reported numbers are in-sample. No held-out split, no prompt freeze point, and no baseline are described; Section VII concedes the single-app and missing-baseline limitations. This is precisely the reader's weakest assumption. I agree with the reader's assessment. The conditional verdict is appropriate: the artifact is public and the prompt is testable, so the concern is addressable by independent replication. I do not propose rejection because the paper is internally consistent, the negative ignore-mechanism result is honestly reported, and the public dataset is a real contribution. One additional internal inconsistency worth correcting is Table II's Semantic Diff row (4B: 0.921 vs 12B: 0.079), which appears to be a typo or swapped values; it does not bear on the classification claim but should be fixed.","tokens_in":8336,"tokens_out":3635,"duration_ms":38900,"concrete_test":"Independently replicate on a different open-source SwiftUI app: generate roughly 20 snapshot failures, run the exact Listing 1 prompt with Gemma3-12B at temperature 0.1 with no further tuning, and compare recall/precision on independently validated labels to the reported 84.21%/66.67%. A drop below about 60% recall or 50% precision would show the headline is an artifact of the 17-case tuning set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states the prompt was 'developed through an iterative design process'; Section IV-A describes a 17-case dataset built from the authors' own SwiftUI app and labeled by two authors. Section V-A then reports Finding 1 (84.21% recall, 66.67% precision for Gemma3-12B) on that same dataset. Because the prompt iterations and taxonomy were informed by these exact cases, the reported numbers are in-sample estimates: they measure how well the prompt fits the 17 cases, not how well it would classify unseen snapshot failures. The paper does not describe freezing the prompt before evaluation, a held-out split, leave-one-out evaluation, or any independent dataset. Section VII acknowledges the single-app limitation and missing baseline, but the abstract and conclusion still state 'strong classification performance' as a general capability. This is the load-bearing support for the paper's central contribution: if the numbers do not transfer, the 'first automated semantic analysis' is still a useful prototype, but the quantitative claim is unsupported. The absence of any baseline additionally means we cannot tell whether the classification exceeds a trivial heuristic (e.g., always predicting the most frequent category).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLMShot, a prompt-based framework that uses Gemma3 vision-language models (4B and 12B variants) to classify snapshot test failures into a predefined semantic taxonomy (e.g., COLOR_CHANGE, PADDING_CHANGE) and to output structured JSON containing pixel and semantic difference estimates, affected UI elements, and natural-language explanations. The evaluation is performed on a new publicly available dataset of 17 failures generated from a purpose-built SwiftUI iOS application with runtime feature flags, with ground-truth labels assigned by two of the authors. The paper reports that Gemma3-12B achieves 84.21% recall and 66.67% precision (Finding 1), and it reports a negative result for a prompt-based ignore mechanism that shows severe performance degradation (Finding 2). The authors claim that LLMShot is the first automated approach to semantic snapshot test analysis.","tokens_in":8569,"tokens_out":2603,"duration_ms":31477,"significance":"If the reported classification performance transfers beyond the authors' 17-case dataset, LLMShot would be a useful step toward automated triage of snapshot test failures, and the released dataset with dynamic artifacts (baseline, failure, diff images, metadata, labels) is a valuable community resource. The paper's strengths include the public dataset, the structured JSON output that is amenable to CI/CD integration, the use of open-weight models with a fixed low temperature, and the honest reporting of a negative result for the ignore mechanism. However, the central quantitative claim is currently supported only by an in-sample evaluation: the prompt and taxonomy were iteratively designed on the same 17 cases used to compute the headline metrics, with no held-out split, no frozen-prompt evaluation, and no baseline comparison. The significance of the paper therefore depends on whether the authors can provide evidence of generalization beyond the exact dataset used to develop the method.","major_comments":[{"comment":"The headline performance (84.21% recall, 66.67% precision for Gemma3-12B) is computed on the same 17 cases that were used to iteratively design the prompt (Section III-A) and to construct and label the dataset (Section IV-A). This is an in-sample estimate: it measures how well the prompt fits the 17 known cases, not how well it classifies unseen snapshot failures. The paper does not describe freezing the prompt before evaluation, a held-out split, leave-one-out evaluation, or an independent dataset. This is load-bearing because the abstract, Finding 1, and the conclusion all present 'strong classification performance' as a general capability. I ask the authors to either add an evaluation protocol that separates prompt development from testing (e.g., freeze the prompt after design and evaluate on a held-out set, or use leave-one-out cross-validation with a fixed prompt) or substantially soften the general claim to a proof-of-concept result.","section":"Section V-A, Table II; Section III-A; Section IV-A"},{"comment":"The claim that LLMShot achieves 'strong' classification performance is not supported by any baseline comparison. With 17 cases and a skewed category distribution (COLOR_CHANGE appears in 5 of 17 tests, ANIMATION_CHANGE in 1), a trivial heuristic such as always predicting the most frequent category, or predicting all categories, could achieve nontrivial recall or hit rate. Without a baseline (e.g., a majority-class predictor, a keyword-based image-diff heuristic, or a prior visual-oracle tool such as DiffDroid), the reported recall and precision values are not interpretable as evidence that LLMShot exceeds a simple alternative. This concern is partly acknowledged in Section VII, but the abstract and conclusion still state the performance as 'strong' without qualification.","section":"Section VII, Comparison to Baselines; Section V-A"},{"comment":"The ground truth is labeled by two of the authors, who also built the application, designed the taxonomy, and iteratively tuned the prompt. The paper does not report inter-rater agreement, and the labeling process is described as resolving disagreements through joint discussion, which can mask systematic labeling biases. With only 17 cases from a single purpose-built iOS application, the external validity threat acknowledged in Section VII is not merely a minor limitation; it directly affects whether the quantitative findings can be generalized to real snapshot testing workflows. I recommend reporting inter-rater agreement statistics and, if possible, augmenting the dataset with cases from an independently developed application or from real project histories.","section":"Section IV-A, Ground Truth Labeling; Section VII"}],"minor_comments":[{"comment":"The table headings read 'PERFORMANCE OF LLMS HOT' and 'PERFORMANCE OF LLMS HOT GEMMA 3:4B'; these appear to be typos for 'LLMShot' and should be corrected.","section":"Table II and Table III headings"},{"comment":"The Semantic Diff value for Gemma3-4B is reported as 0.921 ± 0.046, while Gemma3-12B is 0.079 ± 0.079. Given the text states that the 12B model provides 'more accurate pixel difference estimation,' the 4B semantic difference of 0.921 seems inconsistent with a near-identical 5.3% mean pixel difference and may be a typo (perhaps 0.092). Please verify and correct this value.","section":"Table II, Semantic Diff row"},{"comment":"The IFA Hit Rate is reported as 18.75%, but the baseline and IFGT columns use 17 tests as the denominator; it is unclear whether IFA is computed on 16 cases because one case was excluded. Please clarify the denominator and the exact procedure for the 'Ignore From Analysis' strategy.","section":"Table III, IFA column"},{"comment":"The prompt template uses placeholders such as {reference}, {failure}, and {difference}, but the paper does not specify how the diff image is computed (e.g., pixel-level absolute difference, highlighted diff, or a tool-generated image). For reproducibility, please describe the diff-image generation process.","section":"Section III-A, Listing 1"},{"comment":"The paper states that a temperature of 0.1 was used but does not report the number of runs per test or any variance across runs. Since the authors acknowledge LLM randomness in Section VII, reporting the number of independent runs and the stability of the classification metrics would strengthen reproducibility.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable short empirical study with a useful public dataset and an honest negative result, but the central quantitative claim is currently based on an in-sample evaluation with no baseline and a very small, author-constructed dataset. The requested revisions (independence between prompt design and evaluation, baseline comparisons, and reporting of inter-rater reliability) are substantial but feasible within the scope of the manuscript. I would not reject the paper, but it should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The thing to know is that LLMShot is a plausible idea with honest reporting, but the 84.21% recall / 66.67% precision headline is measured on the same 17 cases that were used to iteratively design the prompt, with no held-out split and no baseline. That makes the numbers optimistic, not fraudulent. The authors explicitly acknowledge the single-app limitation and missing baseline, but the abstract still calls the classification performance \"strong,\" which overshoots what the evidence supports.\n\nWhat is actually new: the paper is the first to apply VLMs to semantic classification of snapshot test failure causes, as far as I can tell. Prior visual oracles (DiffDroid, Owl Eyes, GLIB) detect defects without explaining them; this paper defines a taxonomy, uses a structured prompt with JSON output, and reports a negative result on ignore mechanisms instead of sweeping it under the rug. The public dataset, even if small, is a real artifact for the community. The writing is clear, and the threat-to-validity section is candid.\n\nSoft spots, in proportion: the main one is external validity. Seventeen instances from one SwiftUI app, labeled by two of the authors, with the prompt tuned on those same instances, is a proof-of-concept, not a benchmark. The paper needs a frozen-prompt holdout or an independent app to back the \"strong classification performance\" claim. A trivial baseline (e.g., always predict the most frequent category) would also help calibrate the numbers. There is also a visible anomaly in Table II: the Semantic Diff. for gemma3:4b is 0.921 ± 0.046 and for gemma3:12b is 0.079 ± 0.079. The 4B value looks like a typo or a swapped column, and a referee should ask for a correction.\n\nCitation pattern is fine; they engage with the relevant prior work on visual oracles and VLM-based GUI testing. No self-citation inflation.\n\nWho this is for: people working on GUI testing, snapshot testing, or VLM-based test analysis. They will find the prompt template and dataset useful starting points. The paper deserves a serious referee, but it needs revision before acceptance: rerun with a held-out split or independent data, add a baseline, fix Table II, and soften the \"first\" and \"strong\" claims if the numbers don't transfer. My recommendation: send to review with clear requests for those changes.","headline":"A useful proof-of-concept for VLM-based snapshot triage, but the headline numbers are in-sample estimates: same 17 cases used for prompt design and evaluation.","tokens_in":9099,"tokens_out":2045,"would_cite":true,"duration_ms":23484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model can classify snapshot test failure root causes, with 84.21% recall on a 17-case dataset, while prompting it to ignore known changes fails.","keywords":["snapshot testing","vision-language models","UI test maintenance","test failure triage","semantic classification","Gemma3","iOS UI testing","automated test analysis"],"falsifier":"Take the LLMShot prompt unchanged and run it on a held-out set of snapshot failures gathered from several production Android, web, or iOS apps, with ground-truth labels assigned by developers who were not involved in designing the prompt; if recall on that set falls well below 84% or the model's unknown rate rises sharply, the generalization claim would be falsified.","tokens_in":8162,"feed_emoji":"📸","tokens_out":5924,"duration_ms":61542,"temperature":0.7,"pith_summary":"LLMShot claims that a vision-language model shown the reference image, the failing image, and the pixel-level diff can classify the root cause of a snapshot test failure into semantic categories such as color, padding, layout, text, or animation change, and can produce a natural-language explanation of what changed. The paper reports that the 12B Gemma3 model achieves 84.21% recall and 66.67% precision on its 17-case dataset built from a SwiftUI app with feature flags, while the 4B model trails slightly, and presents this as the first automated semantic analysis of snapshot test failures. It also finds that prompting the model to ignore a known category makes classification collapse, with compliance around 31–35%, so selective filtering remains unsolved. If the result generalizes, developers could automatically triage snapshot failures and spend manual effort only on genuine regressions.","feed_headline":"VLM labels snapshot test failures with 84% recall","feed_subtitle":"LLMShot turns pixel diffs into semantic root causes, but telling the model to ignore known changes still fails.","key_machinery":"The load-bearing mechanism is a structured three-image JSON prompt: it feeds the reference snapshot, the failing snapshot, and the pixel diff to the VLM along with the category list and definitions, and requires a JSON response with a list of root-cause labels (including a dynamically invented UNKNOWN-<reason> fallback), numeric pixel and semantic difference scores, affected UI element names, and a short explanation. This prompt, combined with the taxonomy of change types in Table I, is what converts raw pixel differences into semantic classifications, and it defines the evaluation metrics (hit rate, recall, precision, unknown rate, and pixel-difference error) that the paper reports.","core_discovery":"On its own terms, the paper's central claim is that semantic triage of snapshot test failures is achievable by prompting a VLM with the before image, the after image, and the difference image simultaneously, asking it to label the change according to an eight-category taxonomy and to return structured JSON. The 12B model reaches 84.21% recall and 66.67% precision, a hit rate of 82.35% (at least one correct label per failing test), an unknown rate of 5.88%, and a mean pixel-difference estimation error of 0.059. The 4B model reaches 78.95% recall and 57.69% precision. The same experiments show that appending an ignore instruction for an acceptable category degrades the hit rate to 18.75% (IFA) or 41.18% (IFGT) and leaves only about a third of tests compliant with the instruction, which the paper reads as evidence that prompt-based controllable visual reasoning is not yet reliable.","pith_inferences":["The 17-case, single-app dataset likely under-represents the long-tail distribution of real UI changes; a multi-app, multi-platform benchmark would probably lower the measured precision and reveal taxonomy gaps, so the reported numbers should be read as an upper bound on current VLM capability.","Pairing LLMShot's semantic labels with deterministic pixel-diff thresholds could offset its precision weakness, since the model's probability of adding a spurious label may correlate with small visual changes that a threshold rule can suppress.","The ignore-category failure suggests a concrete next experiment: fine-tune a VLM on paired before/after images with explicit 'ignore X' instructions and measure whether compliance and F1 recover, which would isolate whether the bottleneck is prompting or the models' control mechanism.","Because the dataset is generated by toggling feature flags in one app, a natural extension is to mine real project histories for failing snapshot pairs and rerun the same evaluation, giving an out-of-distribution test of both the framework and the taxonomy."],"forward_implications":["CI/CD pipelines could automatically attach a suggested root cause, affected elements, and explanation to every failing snapshot test, so a developer only opens cases that the model flags as genuine regressions.","The structured JSON output is machine-readable, so failure analyses could be routed into issue trackers or dashboards without manual transcription.","The 4B model's smaller footprint gives a plausible low-cost deployment for continuous integration, accepting lower precision to avoid large GPU requirements.","The UNKNOWN-<reason> mechanism means the classifier can signal a previously unseen change type rather than forcing a label, which is important as UI change patterns evolve.","Because the prompt-based ignore mechanism fails, the paper's own results imply that useful selective filtering will require fine-tuning or another approach beyond instruction-following."],"supporting_citations":[{"why":"Establishes snapshot testing as widely used and empirically studied, motivating the triage problem.","marker":"[2]"},{"why":"Documents snapshot testing's benefits, fragility, and false-positive rate, and calls for detailed descriptions that LLMShot automates.","marker":"[3]"},{"why":"A prior visual-oracle tool that detects cross-platform inconsistencies but does not explain changes; used as a contrast point for the explanatory gap.","marker":"[10]"},{"why":"The only prior snapshot-testing dataset, limited to static project attributes, which the new dataset is designed to surpass.","marker":"[18]"},{"why":"The snapshot capture library used to generate baseline and failure images in the evaluation.","marker":"[22]"},{"why":"Technical report for the Gemma3 model family whose 4B and 12B variants are evaluated.","marker":"[23]"},{"why":"The local inference tool used to run the models during experiments.","marker":"[24]"}],"fun_headline_variants":["LLMShot: VLM triages snapshot failures with 84% recall","Vision-language model automates snapshot test triage","LLMShot: 84% recall on snapshot failures, ignore prompts fall short","VLM-based snapshot triage hits 84% recall, but ignore instructions fail","LLMShot: First automated semantic snapshot test analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on the assumption that the authors' 17-case dataset, generated from a single SwiftUI app with feature flags and labeled by the authors on the same cases used to iteratively refine the prompt, is representative enough of real snapshot-testing failures that the reported recall and precision will carry over to other applications and teams.","fun_headline_variants_meta":{"raw":{"variants":["LLMShot: VLM triages snapshot failures with 84% recall","Vision-language model automates snapshot test triage","LLMShot: 84% recall on snapshot failures, ignore prompts fall short","VLM-based snapshot triage hits 84% recall, but ignore instructions fail","LLMShot: First automated semantic snapshot test analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3541,"prompt_tokens":961,"completion_tokens":2580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2488}},"tokens_in":577,"tokens_out":2580,"duration_ms":19988,"temperature":1.0,"reasoning_tokens":2488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:40:20.534860+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the LLMShot prompt unchanged and run it on a held-out set of snapshot failures gathered from several production Android, web, or iOS apps, with ground-truth labels assigned by developers who were not involved in designing the prompt; if recall on that set falls well below 84% or the model's unknown rate rises sharply, the generalization claim would be falsified.","supporting_citations":[{"cited_title":"An empirical study on the use of snapshot testing,","cited_arxiv_id":null,"evidence_quote":"Establishes snapshot testing as widely used and empirically studied, motivating the triage problem."},{"cited_title":"Snapshot testing in practice: Benefits and drawbacks,","cited_arxiv_id":null,"evidence_quote":"Documents snapshot testing's benefits, fragility, and false-positive rate, and calls for detailed descriptions that LLMShot automates."},{"cited_title":"Automated cross-platform inconsistency detection for mobile apps,","cited_arxiv_id":null,"evidence_quote":"A prior visual-oracle tool that detects cross-platform inconsistencies but does not explain changes; used as a contrast point for the explanatory gap."},{"cited_title":"Snapshot testing dataset,","cited_arxiv_id":null,"evidence_quote":"The only prior snapshot-testing dataset, limited to static project attributes, which the new dataset is designed to surpass."},{"cited_title":"Pointfreeco/swift-snapshot-testing: delightful swift snapshot testing","cited_arxiv_id":null,"evidence_quote":"The snapshot capture library used to generate baseline and failure images in the evaluation."},{"cited_title":"Ollama: A lightweight, extensible framework for building and running language models on the local machine,","cited_arxiv_id":null,"evidence_quote":"The local inference tool used to run the models during experiments."}],"review_version":1}