{"id":"9125da23-e49d-4085-b546-9493acc3a55a","arxiv_id":"2505.05895","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A LoRA fine-tuned Molmo-7B model, trained on synthetic automotive UI data with reasoning and pass/fail evaluation labels, improves visual grounding on a new automotive benchmark and on the external ScreenSpot test.","lead":"The authors release AutomotiveUI-Bench-4K, a dataset of 998 automotive infotainment screens with 4,208 human annotations, and fine-tune a 7B vision-language model (ELAM-7B) to locate and evaluate UI elements on those screens. The model improves cross-domain visual grounding by 5.6 points over its base on ScreenSpot and runs on a 24 GB consumer GPU.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No explicit train/eval image split: in-domain gains on AutomotiveUI-Bench-4K may be inflated by training on the same images used for evaluation.","rationale":"The reader's verdict already flags the missing training/evaluation image split as a weakness, and my analysis confirms this as the most load-bearing concern. The reader's designated weakest assumption was the correctness and diversity of synthetic captions; however, I see the train/eval overlap as more fundamental because it threatens the validity of the in-domain benchmark numbers regardless of synthetic label quality. The paper's own limitation section acknowledges errors in synthetic training data, and the error analysis probes model failures, but even perfect synthetic labels would not fix the problem if evaluation images were seen during training. The ScreenSpot external benchmark partially mitigates overfitting concerns for grounding, but it does not test the evaluation-capability claim, which is the paper's main contribution. Therefore the conditional verdict stands: the authors must provide an explicit disjoint split and re-report the in-domain metrics. My recommendation is UNCHANGED because the reader already reached CONDITIONAL, and this concern reinforces that outcome rather than moving it to accept or reject.","tokens_in":23071,"tokens_out":3524,"duration_ms":36013,"concrete_test":"Download AutomotiveUI-Bench-4K and the released ELAM training split/code from Hugging Face; compute exact file hashes (SHA-256) of all benchmark images and all images used in synthetic training (including any crops/augmentations). If any overlap exists, re-run the main Table 2 evaluation on the subset of benchmark images that are disjoint from training and report all six metrics; if overlap is zero, state this explicitly. Additionally, to rule out selection bias, re-select the LoRA rank on a held-out validation split and report Table 5 numbers for that split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central in-domain claim is that LoRA fine-tuning on synthetic automotive UI data yields +16.3 TA_vg, +6.1 ER_vg, and +11.3 ERevl on AutomotiveUI-Bench-4K. For that to be evidence of generalization, the benchmark images used for evaluation must be disjoint from the images used to generate synthetic training data. The paper does not state this. Section 3.2 describes generating synthetic captions from bounding boxes 'from the dataset', and Section 5.1 reports a synthetic training set of 17,708 annotations across 6,230 images; the benchmark itself is 998 images / 4,208 annotations. No sentence in Sections 3-5 or Appendix A.1 excludes AutomotiveUI-Bench-4K images from the training pipeline. If the 998 evaluation images (or crops/augmentations of them) are contained in the 6,230-image synthetic set, the reported gains can be inflated by memorization of layouts, icon positions, and passed/failed labels, making the in-domain evaluation not a measure of the model's ability to handle unseen automotive UIs. The ScreenSpot result is a useful external check, but it does not validate the in-domain evaluation-capability claim, which is the paper's main contribution. Hyperparameter selection on the same benchmark (Table 5, LoRA rank) adds a secondary optimism bias. A disjoint image-level split and a report of metrics on that split is required.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ELAM-7B, a Molmo-7B model fine-tuned with LoRA on synthetic automotive UI data for visual grounding and pass/fail evaluation of UI states. It releases AutomotiveUI-Bench-4K, a manually curated benchmark of 998 infotainment images with 4,208 annotations, and describes a synthetic data pipeline that uses Gemini 2.0 Flash Thinking as a teacher and GPT-4o-mini for rephrasing. The reported in-domain results are large gains over the Molmo baseline on AutomotiveUI-Bench-4K: +16.3 points for test-action visual grounding (TA_vg), +6.1 points for expected-result grounding (ER_vg), and +11.3 points for expected-result evaluation (ER_evl). On the external ScreenSpot benchmark, the model reaches 80.8% average accuracy, a +5.6% improvement over Molmo-7B-D-0924. The paper also includes ablations over reasoning, rephrasing, and LoRA rank, a t-SNE-based failure analysis, and a detailed manual error taxonomy.","tokens_in":23312,"tokens_out":3673,"duration_ms":37008,"significance":"The contribution is potentially valuable: the released dataset fills a clear gap in automotive UI grounding and evaluation, the training pipeline is cost-efficient and open-sourced, and the ScreenSpot result provides an external generalization check that speaks positively of the approach. The paper is unusually candid about its limitations, explicitly acknowledging training-data misclassifications and an overemphasis on presence/visibility in expected results. If the in-domain numbers survive a clean image-disjoint split, the work would be a solid engineering contribution with a useful benchmark. As it stands, the headline in-domain claims rest on an unstated train/eval image split and on hyperparameters selected on the evaluation set, both of which are load-bearing for the paper's central thesis.","major_comments":[{"comment":"The paper never states that the 998 AutomotiveUI-Bench-4K evaluation images are disjoint from the 6,230 images used to generate the synthetic training data. Section 3.2 describes generating captions from bounding boxes 'from the dataset,' and Section 5.1 reports a synthetic set of 6,230 images without excluding the benchmark images. If any of the 998 evaluation images (or crops/augmentations) overlap the training set, the reported +16.3 TA_vg, +6.1 ER_vg, and +11.3 ERevl gains in Table 2 can be inflated by memorization of icon positions, layouts, and passed/failed labels. The ScreenSpot result mitigates the cross-domain generalization concern but does not validate the in-domain evaluation-capability claim, which is the paper's main contribution. The authors should state the exact image-level split and report metrics on a held-out subset that is provably disjoint from synthetic training images.","section":"§5.1, §5.2, Appendix A.1"},{"comment":"The LoRA rank was selected on the evaluation benchmark itself: Section 5.1 states 'An optimal LoRa rank r = 64 was selected based on an ablation study (Table 5)' with Table 5 reporting performance on AutomotiveUI-Bench-4K. This selection-on-test-set introduces optimism bias that is not accounted for in the reported numbers. A validation split (or nested selection) is needed for the final in-domain metrics.","section":"§5.1, Table 5"},{"comment":"The paper's own error analysis and limitation paragraph report that the training data overemphasizes presence/visibility of elements and contains misclassified UI element states (e.g., toggle switches). Since ERevl is the paper's claimed evaluation capability, this acknowledged data bias means the 78.2% evaluation accuracy on AutomotiveUI-Bench-4K may not measure state-based evaluation as the paper intends. The authors should quantify how many expected-result ground truths in AutomotiveUI-Bench-4K require state evaluation rather than simple presence/visibility, and report ERevl separately for these two categories.","section":"§5.6.2, §6"},{"comment":"All reported percentages are single-run results with no standard deviations, seed counts, or confidence intervals. Given the small benchmark (4,208 annotations) and the hyperparameter selection issue in Table 5, differences of 1–5 points in the ablation tables may not be statistically meaningful. The authors should report variance across multiple training seeds or at least bootstrap confidence intervals for the main metrics.","section":"§5.2, Tables 2, 4, 5"}],"minor_comments":[{"comment":"The acronym is spelled inconsistently as 'LoRa' and 'LoRA' across Sections 3.3, 5.1, and 5.4; please unify the spelling.","section":"Throughout"},{"comment":"The word 'superscipts' should be 'superscripts'.","section":"Table 2 caption"},{"comment":"There is a duplicated definite article in 'The the medium sensitivity icon'; please correct the typo.","section":"Figure 20 caption"},{"comment":"The abstract says ELAM 'closely matching or surpassing specialized models for desktop, mobile, and web,' but in Table 3, UGround-V1-7B (86.3) and OS-Atlas-Base-7B (81.0) outperform ELAM on average; the wording should be qualified to avoid overstating the comparison.","section":"Abstract and Table 3"},{"comment":"The 'Baseline' row in Table 4 is not explicitly defined; please state whether it corresponds to no reasoning and no rephrasing, and how it differs from the 'Rephrasing' and 'Reasoning' rows.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the external ScreenSpot validation is reassuring, but the missing train/eval image split is a central correctness issue for the headline in-domain numbers. The authors' candid limitation section is a review strength, not a weakness, and should be preserved in revision. The dataset and code release are valuable assets that make the requested re-evaluation feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The headline: useful new benchmark and a plausible method, but the central in-domain evaluation is compromised by an unstated train/eval image split. The authors need to fix that before anyone relies on the numbers.\n\nWhat's actually new: AutomotiveUI-Bench-4K is the first automotive-specific UI grounding/evaluation dataset I know of, with dual test-action and expected-result labels including pass/fail. The synthetic data pipeline that generates reasoning and rephrased captions is a sensible way to get evaluation-aware training data for a 7B model. The ScreenSpot result—80.8% average, +5.6% over Molmo-7B—is an external signal that fine-tuning didn't hurt cross-domain generalization. That's the strongest part of the paper.\n\nWhat's soft: The big one is the split. Section 3.2 says bounding box regions 'from the dataset' were used for synthetic caption generation; Section 5.1 says 17,708 annotations across 6,230 images. The benchmark has 998 images. The paper never says the training images are disjoint from the evaluation images. If they aren't, the +16.3 TA_vg and +11.3 ERevl gains could just be the model memorizing layouts and labels. The ScreenSpot result doesn't validate the in-domain evaluation claim, because that's a different task and domain. Secondary issues: LoRA rank was selected on the same benchmark (Table 5), and there are no seeds or error bars. These matter less, but they add optimism bias. The authors are transparent about data issues—they note misclassified training examples and overemphasis on presence/visibility—and they're appropriately cautious about safety-critical deployment. That honesty is good, but it doesn't address the split.\n\nWho it's for: Anyone working on GUI grounding for automotive infotainment, or on evaluation-aware fine-tuning for small VLMs. The dataset is a real resource, and the ScreenSpot gain is worth discussing.\n\nRecommendation: It deserves a serious peer review, but with a required revision: state the split, release code with hashes, report seeds or intervals. If the split reveals leakage, the paper should be reframed as a cross-domain adaptation study. I'd engage with it, but I'd wait for the clarifications before citing the in-domain numbers.","headline":"A useful automotive UI benchmark and a plausible fine-tuning recipe, but the in-domain gains are unproven until the train/eval image split is disclosed.","tokens_in":23865,"tokens_out":3157,"would_cite":true,"duration_ms":31379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model fine-tuned on synthetic automotive UI captions can point to interface elements and judge whether expected results pass or fail, and it keeps this skill on desktop, mobile, and web screens.","keywords":["vision-language model","automotive UI","visual grounding","UI evaluation","synthetic data generation","large action model","parameter-efficient fine-tuning","infotainment testing"],"falsifier":"Audit a random sample of the 17,708 synthetic annotations against the original images and count mislabeled control states, such as toggle switches marked on when they are off. If the mislabel rate is high, or if ELAM-7B's expected-result accuracy on a new test set of state-based checks like 'seat heating is on' or 'temperature is set to 22 degrees' is no better than the base model's roughly 67%, the claimed evaluation capability is not real.","tokens_in":22861,"feed_emoji":"🚗","tokens_out":7972,"duration_ms":77948,"temperature":0.7,"pith_summary":"This paper is trying to establish that a relatively small vision-language model can learn to both locate and judge automotive infotainment screens from synthetic training data alone. The authors fine-tune a 7-billion-parameter open model with low-rank adaptation on a pipeline that generates, for each marked UI element, a reasoning trace plus either a test action or an expected result with a passed/failed verdict. On their new benchmark of 998 automotive screens with 4,208 annotations, the resulting model, ELAM-7B, improves test-action grounding by 16.3 percentage points and expected-result evaluation by 11.3 points over the base model. It also scores 80.8% average accuracy on the cross-domain ScreenSpot benchmark, matching or beating specialized mobile and web UI models despite being trained mostly on car screens. If true, this would let automotive UI validation move beyond fragile OCR and template matching toward a local, cost-efficient model that can be updated as interfaces change.","feed_headline":"Model learns to locate and judge car-infotainment screens","feed_subtitle":"Fine-tuned on synthetic automotive data, it beats its base and still matches web and mobile UI models.","key_machinery":"The load-bearing machinery is the synthetic data pipeline and the dual-label annotation format it produces. For each UI element, a teacher model sees an image with marked bounding boxes, a technique called Set-of-Mark prompting, and writes a test action or an expected result together with a reasoning trace and a passed/failed conclusion; a smaller model then rephrases the text so the student does not overfit one teacher style. The student is Molmo-7B-D-0924, whose native pointing output, a normalized coordinate in a point tag, provides the grounding mechanism. Low-rank adaptation, or LoRA, is the parameter-efficient fine-tuning method that shifts the model into the automotive domain while preserving its general abilities. Ablations in the paper attribute part of the gain to the reasoning traces and part to rephrasing, with a rank-64 LoRA configuration giving the best results.","core_discovery":"The central claim is that domain shift and missing evaluation capability can be solved together by fine-tuning a general grounding-capable vision-language model on synthetic automotive data that explicitly pairs each localization with an evaluative label. Concretely, ELAM-7B, built from Molmo-7B-D-0924 with LoRA, reaches 87.6% accuracy on pointing to the target of a test action and 78.2% on classifying whether an expected result passes or fails, against 71.3% and 66.9% for the base model on AutomotiveUI-Bench-4K. The same model reaches 80.8% average accuracy on ScreenSpot, 5.6 points above the base and 2.2 points above the 72-billion-parameter variant, roughly matching specialized UI grounding systems. The paper argues this shows the synthetic pipeline transfers across domains and that evaluation capability, not just interaction, can be instilled in a small model trainable on consumer hardware.","pith_inferences":["Editorial inference: the same pipeline logic should generalize to other narrow UI domains with scarce data, such as medical-device screens, industrial HMIs, or kiosks, where a teacher model can generate reasoning-labeled synthetic annotations and a 7B student can be fine-tuned locally.","Editorial inference: the paper's own error statistics suggest a cheap, testable improvement: fix the undersized and mis-centered bounding boxes, which account for roughly a quarter of expected-result errors, and add annotations that target control state rather than mere visibility; this could close much of the 15-point gap to the human expert's 93.2% evaluation accuracy.","Editorial inference: a stricter test would separate the two learned skills by evaluating the model on state-based expectations only, such as whether a toggle is on or off or a value is selected; if accuracy falls back to the baseline's roughly 67%, the apparent evaluation gain may come mostly from presence and visibility patterns rather than genuine state reasoning."],"forward_implications":["An automotive UI validation model can run on a single consumer GPU with 24 GB of VRAM, with measured invoke times around 2.4 seconds for a test action and 3.4 seconds for an expected result, making in-house deployment practical.","A single model now performs both interaction grounding and pass/fail state evaluation, two capabilities that previous small UI models kept separate, so one inference pass can cover both test-action and expected-result checks.","Training primarily on automotive screens does not hurt general UI grounding; on ScreenSpot the fine-tuned model improves over its 7B base and even over a 72B variant, so synthetic domain adaptation appears to transfer to other UI domains.","The evaluation capability on expected results is the weaker and more persistent limitation: the paper itself cautions against using the model as the sole authority for UI verification, especially for safety-critical functions, and recommends keeping a human in the loop."],"supporting_citations":[{"why":"Supplies the Molmo-7B-D-0924 base model, its pointing mechanism, and the PixMo pretraining that the fine-tuning builds on.","marker":"[5]"},{"why":"Provides the arrow-marked bounding box idea for synthetic caption generation and the ScreenSpot comparison numbers used to measure cross-domain generalization.","marker":"[10]"},{"why":"Introduces ScreenSpot, the benchmark used to show that training on automotive data does not hurt general UI grounding.","marker":"[4]"},{"why":"Defines low-rank adaptation, the parameter-efficient fine-tuning method that makes small-data adaptation feasible.","marker":"[12]"},{"why":"Supplies the Set-of-Mark prompting technique used to ask teacher models to describe marked UI elements.","marker":"[39]"},{"why":"Offers the specialized UI grounding model whose ScreenSpot accuracy ELAM-7B is claimed to match or exceed.","marker":"[20]"},{"why":"Provides the small interaction-only model used in the 270M-parameter comparison, supporting the claim that a larger grounding foundation matters.","marker":"[27]"}],"fun_headline_variants":["Fine-tuned VLM for car UIs beats base, matches specialized","Synthetic car UI data yields 80.8% ScreenSpot accuracy","LoRA fine-tuning makes car UI model rival web/mobile","Automotive UI benchmark and model now open sourced","ELAM-7B: car UI grounding and evaluation on consumer GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the synthetic captions generated by teacher models, and then rephrased, being correct and varied enough to teach genuine UI-state evaluation; the paper reports that some training samples mislabel basic element states and that expectations overemphasize presence and visibility, which is exactly where that assumption is weakest.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned VLM for car UIs beats base, matches specialized","Synthetic car UI data yields 80.8% ScreenSpot accuracy","LoRA fine-tuning makes car UI model rival web/mobile","Automotive UI benchmark and model now open sourced","ELAM-7B: car UI grounding and evaluation on consumer GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2856,"prompt_tokens":988,"completion_tokens":1868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1778}},"tokens_in":604,"tokens_out":1868,"duration_ms":15536,"temperature":1.0,"reasoning_tokens":1778,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:53:40.497406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the 17,708 synthetic annotations against the original images and count mislabeled control states, such as toggle switches marked on when they are off. If the mislabel rate is high, or if ELAM-7B's expected-result accuracy on a new test set of state-based checks like 'seat heating is on' or 'temperature is set to 22 degrees' is no better than the base model's roughly 67%, the claimed evaluation capability is not real.","supporting_citations":[{"cited_title":"Navi- gating the digital world as humans do: Universal visual grounding for GUI agents","cited_arxiv_id":null,"evidence_quote":"Provides the arrow-marked bounding box idea for synthetic caption generation and the ScreenSpot comparison numbers used to measure cross-domain generalization."},{"cited_title":"J., S HEN , Y., WALLIS , P., A LLEN -ZHU, Z., L I, Y., WANG , S., W ANG , L., C HEN , W., ET AL","cited_arxiv_id":null,"evidence_quote":"Defines low-rank adaptation, the parameter-efficient fine-tuning method that makes small-data adaptation feasible."},{"cited_title":"Q., L I, L., G AO, D., Y ANG , Z., W U, S., B AI, Z., L EI, W., W ANG , L., AND SHOU , M","cited_arxiv_id":null,"evidence_quote":"Offers the specialized UI grounding model whose ScreenSpot accuracy ELAM-7B is claimed to match or exceed."}],"review_version":1}