Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Visual Prompting with Iterative Refinement for Design Critique Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A six-LLM prompt-only pipeline wins human-expert preference 58% of the time.

desk verdict Solid, useful engineering recipe for grounded UI critique via iterative visual prompting, with consistent empirical gains but an evaluation that never directly tests the paired text+bbox output. read the letter →

arxiv 2412.16829 v2 pith:EUXBOZ7T submitted 2024-12-22 cs.AI

Peitong Duan , Chin-Yi Cheng , Bjoern Hartmann , Yang Li This is my paper · ORCID
classification cs.AI
keywords UIdesigncritiquevisualpromptingiterativerefinementmultimodalLLMgroundingboundingboxopen-vocabularydetectionfew-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that prompt-only orchestration of multiple multimodal LLMs can generate UI design critiques that are both better written and more visually grounded than a few-shot baseline. The pipeline takes a UI screenshot and design guidelines, produces a list of design comments, and attaches each comment to a bounding box, then iteratively refines both the text and the boxes until a validation LLM accepts them. Across Gemini-1.5-pro and GPT-4o, the authors report consistent improvements in comment similarity and bounding-box IoU, and human design experts preferred the pipeline's comment sets over the baseline's 58% of the time. The same pipeline also improves open-vocabulary object and attribute detection mAP by up to 9.1 points, which the authors read as evidence that the mechanism generalizes beyond design critique.

What carries the argument

The engine is an iterative visual prompting loop: six LLMs split into text generation and refinement, validation, and bounding-box generation and refinement. Text is generated first, filtered, then grounded by a BoxGen LLM that reads coordinate markers along the screenshot edges; a BoxRefine LLM then evaluates the candidate box through a zoomed-in patch of the region it encloses, and either accepts it or returns refined coordinates, with the refinement history included in the prompt. A Validation LLM classifies each text-and-box pair into both-correct, incorrect-text, incorrect-box, or both-incorrect, routing pairs to the appropriate refinement branch or discarding them. Zoomed-in patches plus coordinate markers are the key visual-prompting device that lets the LLM make fine-grained spatial corrections, and using separate LLMs for generation and refinement is what prevents self-bias.

What would settle it

Take a random sample of UICrit text-and-box pairs, obtain human ground-truth labels for text correctness and box correctness, run the pipeline's Validation LLM on the same pairs, and compute its per-category accuracy; if the classifier's accuracy is near chance, or if replacing it with human validation does not beat the LLM-only version, the claimed benefit of iterative refinement collapses.

Watch

Extended reading notes

Core claim

The central claim is that iteratively refining both the text and its bounding box, with each refinement step conditioned on a zoomed-in view of the current box and guided by few-shot examples, is enough to make a multimodal LLM produce visually grounded design feedback that approaches human expert quality. The paper's six-LLM pipeline starts with text generation, filters invalid comments, generates candidate boxes, then loops between box refinement, text refinement, and validation until a pair is accepted or discarded. In human evaluation, the pipeline's comment sets were ranked midway between human annotations and the baseline, were preferred over the baseline 58% of the time, and closed half the gap to human performance on comment-set ranking. On automatic metrics, every added module incrementally improved comment similarity and estimated IoU for both models, and the same architecture transferred to open-vocabulary object and attribute detection with mAP gains up to 9.1.

Load-bearing premise

The pipeline's gains rest on the Validation LLM correctly judging whether a comment and its bounding box are accurate; if that judgement is wrong, valid comments get discarded or sent to the wrong refinement branch, and the iterative loop can do more harm than good.

Editorial extensions

If this is right

  • UI design critique can be improved without any finetuning: a purely prompt-based pipeline beats the few-shot baseline on comment quality and grounding for both Gemini-1.5-pro and GPT-4o.
  • Human experts find the pipeline's output more useful: they prefer its comment sets over the baseline's 58% of the time and rank them halfway between the baseline and human-written critiques.
  • The mechanism transfers to other grounded-text tasks: applying the same pipeline to open-vocabulary object and attribute detection raises mAP over the baseline by up to 9.1 points using the same few-shot sampling scheme.
  • Because the pipeline is modular, each step can be improved independently; the authors show that text filtering, iterative box refinement, and validation each add a measurable increment of IoU or comment similarity.
  • Automated critique output of this quality could serve as a reward function for automated UI generation, a use the paper motivates in its introduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to swap the LLM Validation module for a human-in-the-loop validator: the paper itself notes the validation step is imperfect, so measuring how much of the gain comes from validation accuracy rather than from refinement per se would isolate the bottleneck.
  • The zoomed-in-patch trick is not specific to rectangles: the same show-the-model-the-current-proposal-enlarged pattern should transfer to segmentation masks, keypoint localization, or layout grounding tasks.
  • The gains likely depend on base-model strength and scale: the paper attributes GPT-4o's higher IoU partly to its larger size, so a stronger base model might need fewer refinement rounds while a weaker one might need more validation interventions.
  • The OVAD and OVD gains, especially the 9.1-point jump for GPT-4o, are measured on one benchmark; whether the same margin holds for other open-vocabulary backbones and more attribute-heavy splits is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an iterative visual prompting pipeline for UI design critique generation that coordinates six LLMs to generate, filter, validate, and iteratively refine both the text of design comments and their bounding boxes. The pipeline is evaluated on UICrit with Gemini-1.5-pro and GPT-4o, reporting improvements over a few-shot baseline on comment similarity, estimated IoU, and human expert preference, as well as on open-vocabulary object and attribute detection (OVAD/OVD), where the authors report mAP gains of up to 9.1. The central claim is that prompt-only, iterative refinement with visual prompting improves both the textual quality and the visual grounding of multimodal LLM output across domains.

Significance. If the result holds, the paper would contribute a modular, resource-efficient prompting technique for improving visually grounded text generation in multimodal LLMs, with practical relevance to design feedback and potential transfer to other tasks. The paper's strengths include the detailed modular architecture, the inclusion of two base LLMs, two tasks, an expert human study, extensive ablations of visual prompting and iterative refinement, out-of-domain qualitative evaluation, and a cost analysis. However, the evidence for improved grounding in the full pipeline is indirect: the human evaluation never tests the actual (text, predicted-box) pairs, and the main automatic grounding metric for the full pipeline is a noisy proxy. The paper therefore currently supports the text-quality claim more strongly than the joint text+grounding claim.

major comments (4)
  1. [Section 5.5, Table 3] The human evaluation protocol decouples text quality from grounding quality. Raters are shown a screenshot marked with a ground-truth bounding box 'determined and agreed upon by the authors' and asked to rate the comment for that region; they never see the pipeline's predicted bounding box. Consequently, the 58% comment-set preference and the comment quality ratings validate only the text component, not the 'design feedback output' that includes the predicted bounding boxes. The BBox IoU column in Table 3 is an automatic comparison for comments that survived text-quality filtering, so it is selection-biased and is not part of the human preference judgment. Please either run a human evaluation in which raters see the system's predicted box together with the comment, or report a direct human judgment of predicted-box accuracy for the full set of generated comments, and adjust the abstract/conclusion claims accordingly.
  2. [Section 5.4, Table 2] The 'Estimated IoU' metric in the full-pipeline evaluation uses the bounding box of the most semantically similar UICrit comment as a pseudo-ground-truth, matching by max sentenceBERT cosine over a non-exhaustive ground-truth set. This can pair a valid generated comment with a semantically similar but spatially different annotation, making the IoU estimate unreliable in either direction. The authors themselves note that the baseline's very large boxes 'would decrease the chance of the IoU being zero, which may have inflated its estimated IoU' (Section 5.4), but no equivalent correction or confidence interval is provided for the pipeline's estimated IoU. Please provide a direct grounding evaluation on pipeline-generated comments (e.g., human raters judge whether the predicted box correctly localizes the comment's target) and report confidence intervals or significance tests for the IoU differences.
  3. [Section 7, Section A.4.1, Table 2] The validation module is load-bearing: it decides whether a (text, box) pair is discarded, sent for text refinement, sent for box refinement, or kept. The paper acknowledges in Section 7 that 'the LLM-based validation steps are not fully accurate' and Figure 9 in the appendix shows that the pipeline sometimes eliminates valid comments. However, Table 2's final row, '+ Validation, IR of Text & Bbox', combines validation with the subsequent refinement steps, so it does not reveal the contribution of validation alone. Without an ablation that isolates validation (e.g., full pipeline without validation, full pipeline with validation but no refinement branches), or a report of validation precision/recall against human judgments, the effectiveness of the central refinement mechanism remains unclear.
  4. [Sections 5.5 and A.6] The human evaluation uses 18 experts rating 33 UIs for Gemini-1.5-pro only, with Fleiss kappa of 0.22 for comment quality and 0.29 for comment-set ranking. The reported differences are modest (comment quality 0.47 vs. 0.45; set rank 2.0 vs. 2.3; IoU 0.451 vs. 0.423) and no statistical tests or confidence intervals are provided. The claim that the comment set of the pipeline was 'preferred over the baseline's 58% of the time' needs a significance test against chance and an analysis that accounts for rater and item variability. Please add this statistical analysis or temper the human-preference claims accordingly.
minor comments (6)
  1. [Section 2.2] The phrase 'an 20% average performance gain' should read 'a 20% average performance gain'.
  2. [Section 5.5, Appendix A.6] The text 'See Section 5.5 (Appendix) for more details on the study method' appears in Section 5.5 itself; the cross-reference should point to Appendix A.6.
  3. [Tables 1, 2, and 4] The abbreviated model names 'Gemini1.5tn' and 'GPT4tn' are not defined in any caption; please spell out or footnote them.
  4. [Section 6.2] The baseline for OVAD/OVD is described only as the same baseline from Section 5.2; please specify whether the baseline includes the same few-shot text generation and filtering steps for this task, since the comparison's fairness depends on that detail.
  5. [Appendix A.7, Algorithm 1] The pseudocode contains spacing artifacts such as 'perturb f rac' and is not typeset as clean algorithm code; please reformat it for readability.
  6. [Appendix A.6] The sentence 'we maximized the diversity of the UI screenshots by randomly sampling an even number of UIs from each of the UI task categories' is internally inconsistent, since random sampling does not guarantee maximal diversity; please clarify the sampling procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline's improvements are evaluated against external human raters and an independent benchmark, not against its own training or few-shot inputs.

full rationale

The derivation is self-contained in the relevant sense. The pipeline is a fixed prompt/LLM orchestration; no parameter is fitted to the evaluation outputs, and no equation defines the reported improvements in terms of the pipeline's own inputs. Few-shot examples are drawn from a UICrit split explicitly separate from the examples used for evaluation, so the test set is not used to construct the method. The baseline is prior work by the same authors, but it is a fixed prior pipeline rather than a fitted component of this paper, and the central claim is corroborated by external human expert raters (Section 5.5) and by the independent OVAD benchmark of Bravo et al. (Section 6). The human study does decouple text from predicted boxes by showing raters ground-truth bounding boxes, and the Estimated IoU metric is admitted to be noisy and potentially favorable to the baseline's large boxes; these are evaluation-validity concerns, not circular reductions. The one methodological choice that looks suspicious—making the baseline comment-similarity identical to the Text Generation row by reusing the same initial comments—is a controlled component comparison and does not define the pipeline's improvement into existence. No step in the paper fits the enumerated circularity patterns, and no load-bearing claim reduces to a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This empirical paper introduces no new physical entities. Its central claim rests on hard-to-verify assumptions about commercial LLM behavior, the accuracy of LLM-based validation, and the validity of proxy metrics. The hand-set hyperparameters (few-shot sizes, perturbation settings, IoU threshold) are free parameters that affect the reported numbers but are not fitted in a statistical sense.

free parameters (4)
  • maximum refinement iterations (implicit) = not reported
    The refinement loops terminate when the LLM outputs the termination phrase or when a maximum iteration count is reached; the maximum is never specified, so it is an unstated hand-set constant.
  • few-shot perturbation parameters (perturb_frac, max_num_perturb) = not reported
    Algorithm 1 uses these to generate few-shot bounding box refinement examples; values are chosen by the authors and not reported.
  • IoU threshold for mAP matching = 0.5
    Used to match predicted boxes to ground truth in open-vocabulary detection; a hand-set evaluation threshold (Section 6.1).
  • number of few-shot examples per step = not reported
    Each of the six LLM roles is given N few-shot examples, but N is never specified in the main text or appendix.
assumptions (4)
  • domain assumption The commercial multimodal LLMs (Gemini-1.5-pro, GPT-4o) follow complex prompt instructions and improve outputs when asked to refine themselves.
    The entire pipeline is built on this behavioral assumption; no proof or controlled analysis beyond the reported results.
  • domain assumption The LLM-based Validation module can classify (text, bbox) pairs accurately enough to drive beneficial refinement.
    Section 7 states validation 'is not fully accurate'; the pipeline's benefit depends on it.
  • domain assumption sentenceBERT cosine similarity to the most similar ground-truth comment is a valid proxy for open-ended critique quality.
    Section 5.4 uses this proxy because UICrit's ground-truth comments are non-exhaustive; the proxy is known to be imperfect.
  • domain assumption The few-shot examples sampled by CLIP/sentenceBERT similarity are representative enough to transfer to test screens.
    The method selects few-shot examples by embedding similarity (Appendix A.2); the assumption that this selection generalizes is untested beyond the reported results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Prompting with Iterative Refinement for Design Critique Generation." pith.science (2026). https://pith.science/paper/EUXBOZ7T

@misc{pith2026241216829,
  author       = {Pith},
  title        = {Pith review of: Visual Prompting with Iterative Refinement for Design Critique Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EUXBOZ7T}},
  note         = {Machine review of arXiv:2412.16829}
}
read the original abstract

Feedback is crucial for every design process, such as user interface (UI) design, and automating design critiques can significantly improve the efficiency of the design workflow. Although existing multimodal large language models (LLMs) excel in many tasks, they often struggle with generating high-quality design critiques -- a complex task that requires producing detailed design comments that are visually grounded in a given design's image. Building on recent advancements in iterative refinement of text output and visual prompting methods, we propose an iterative visual prompting approach for UI critique that takes an input UI screenshot and design guidelines and generates a list of design comments, along with corresponding bounding boxes that map each comment to a specific region in the screenshot. The entire process is driven completely by LLMs, which iteratively refine both the text output and bounding boxes using few-shot samples tailored for each step. We evaluated our approach using Gemini-1.5-pro and GPT-4o, and found that human experts generally preferred the design critiques generated by our pipeline over those by the baseline, with the pipeline reducing the gap from human performance by 50% for one rating metric. To assess the generalizability of our approach to other multimodal tasks, we applied our pipeline to open-vocabulary object and attribute detection, and experiments showed that our method also outperformed the baseline.

Figures

Figures reproduced from arXiv: 2412.16829 by the authors.

Figure 1
Figure 1. Illustration of the UI Design Critique Task, which takes in a UI screenshot and a set of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The figure illustrates our prompting pipeline, which takes an image and a task prompt as [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An example of the inputs to the Bounding Box Refinement LLM. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Illustration of the Open Vocabulary Object and Attribute Detection Task. The example [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Illustration of four example outputs from the pipeline. The screenshots are marked with [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Illustration of four example outputs from the pipeline. The screenshots are marked with [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Illustration of four example outputs from the pipeline. The screenshots are marked with [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Illustration of outputs from the pipeline and baseline, highlighting two cases where our [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Illustration of outputs from the pipeline and baseline, highlighting two cases where the [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Illustration of outputs from the pipeline and finetuned Llama-3.2 11b. The screenshots [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Example design feedback and bounding boxes generated by our pipeline for four modern [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Example design feedback and bounding boxes generated by our pipeline for four modern [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Example design feedback and bounding boxes generated by our pipeline for five modern [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Side by side comparison of the bounding boxes generated by the pipeline (“Output [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Side by side comparison of the bounding boxes generated by the pipeline (“Output [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: An example of iterative bounding box refinement, with the comment it is conditioned on [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: An example of iterative comment refinement, with the bounding box it is conditioned on [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]
Figure 18
Figure 18. Figure 18: The form used for individual comment quality rating and comment set ranking. [PITH_FULL_IMAGE:figures/full_fig_p033_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation

    cs.LG 2025-08 reject novelty 4.0 of 10

    An LVLM-driven iterative text-to-image framework whose claimed performance scores are explicitly labeled fictitious, so no empirical result is established.

Reference graph

Works this paper leans on

24 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    vs 0.333 (visual similarity match)). A.2.2 O PEN VOCABULARY OBJECT AND ATTRIBUTE DETECTION For text generation (i.e., category and attributes) and filtering, we sampled images based on the semantic similarity of their CLIP embeddings. Negative text samples for the filtering step were generated by sampling irrelevant text from other images. For bounding bo...

    work page 2019

  2. [3]

    Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine

    URL https://arxiv.org/abs/2407.21783. Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. MOKA: Open-World Robotic Manip- ulation through Mark-Based Visual Prompting. InProceedings of Robotics: Science and Systems, Delft, Netherlands, July

    arXiv

  3. [4]

    doi: 10.15607/RSS.2024.XX.062. J.L. Fleiss et al. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382,

    doi:10.15607/rss.2024.xx.062 2024

  4. [8]

    While design principles often overlap between mobile and web interfaces, their layouts and screenshot dimensions differ significantly

    Additionally, the pipeline even managed to generate helpful feedback and bounding boxes for websites. While design principles often overlap between mobile and web interfaces, their layouts and screenshot dimensions differ significantly. This suggests that the LLM was able to generalize and adapt its knowledge to generate and refine bounding boxes for webs...

    work page 2024

  5. [9]

    Calibrating LLM-based evaluator

    Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating LLM-based evaluator. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re...

    work page 2024

  6. [11]

    URL https://arxiv.org/abs/2303.08774. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38...

    arXiv

  7. [12]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

    work page 2019

  8. [14]

    doi: https: //doi.org/10.1016/j.buildenv.2021.107780

    ISSN 0360-1323. doi: https: //doi.org/10.1016/j.buildenv.2021.107780. URL https://www.sciencedirect.com/ science/article/pii/S0360132321001876. Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. Screen parsing: Towards reverse engineering of ui models from screenshots. In The 34th Annual ACM Symposium on User In- terface Software and Technology, ...

    work page arXiv 2021

Show all 24 references
  1. [15]

    ISBN 9781450386357

    Associa- tion for Computing Machinery. ISBN 9781450386357. doi: 10.1145/3472749.3474763. URL https://doi.org/10.1145/3472749.3474763. Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, and Jeffrey Nichols. Uiclip: A data-driven model for assessing user inte...

  2. [16]

    org/abs/2404.12500

    URL https://arxiv. org/abs/2404.12500. Anbang Xu, Shih-Wen Huang, and Brian Bailey. V oyant: generating structured feedback on vi- sual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing , C...

    arXiv

  3. [17]

    ISBN 9781450325400

    Association for Computing Machinery. ISBN 9781450325400. doi: 10.1145/2531602.2531604. URL https://doi.org/10.1145/2531602.2531604. Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. LLMRefine: Pi...

  4. [18]

    Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma

    URL https://arxiv.org/abs/ 2405.17104. Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning. 2024 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 2081–2088,

    2024 arXiv

  5. [19]

    (2023), is a benchmark task that involves detecting objects and their associated attributes, along with bounding boxes mark- ing their locations in the image

    A A PPENDIX A.1 O PEN VOCABULARY OBJECT AND ATTRIBUTE DETECTION TASK Open vocabulary object and attribute detection, developed by Bravo et al. (2023), is a benchmark task that involves detecting objects and their associated attributes, along with bounding boxes mark- ing their...

    2023

  6. [21]

    Duan et al. (2024b), we evaluated the pipeline’s performance to determine whether it generalizes to modern UIs and other out-of-domain UIs, such as websites, using only few-shot examples selected from UICrit. Figure 11 displays the generated feedback for four modern Android UI...

    2024

  7. [23]

    to extract UI element and group locations from the screenshot. We demonstrate the results of using the DOM tree (taken from the XML-based Android View Hier- archy available in RICO(Deka et al., 2017)) to refine the pipeline’s bounding boxes for some of the UICrit UIs in Figure...

    2017

  8. [24]

    FINISH" button uses the same red color as the header, which can be confusing for users. To fix this, the

    22 Algorithm 1 Generate Bounding Box Refinement Few-shot Examples Require: the bounding box to be perturbed input bbox, the fraction that the bounding box’s coor- dinates will be perturbed perturb f rac Ensure: The coordinates of input bbox perturbed by perturb f rac 1: functi...

    2024

  9. [1990]

    ISBN 0201509326

    Association for Computing Machinery. ISBN 0201509326. doi: 10.1145/97243. 97281. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and Lama Ahmad et. al. Gpt-4 technical report,

    doi

  10. [2008]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal

    doi: 10.1145/1460355.1460357. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations,

  11. [2014]

    ISBN 978-3-319-10602-1

    Springer International Publishing. ISBN 978-3-319-10602-1. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv pr...

    arXiv

  12. [2017]

    ISBN 9781450349819

    Association for Computing Machinery. ISBN 9781450349819. doi: 10.1145/3126594.3126651. URL https://doi.org/10.1145/3126594.3126651. Michael Dorkenwald, Nimrod Barazani, Cees G. M. Snoek, and Yuki M. Asano. Pin: Positional insert unlocks object localisation abilities in vlms,

  13. [2019]

    org/abs/1908.10084

    URL https://arxiv. org/abs/1908.10084. Shreya Shankar, J. D. Zamfirescu-Pereira, Bj ¨orn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences,

    1908 arXiv

  14. [2021]

    ISBN 9781450380188

    Association for Computing Machinery. ISBN 9781450380188. doi: 10.1145/3397482.3450727. URL https://doi.org/10.1145/3397482.3450727. 11 Krzysztof Z Gajos, Daniel S Weld, and Jacob O Wobbrock. Automatically generating personalized user interfaces with supple. Artificial intellig...

  15. [2023]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision – ECCV 2014 , pp. ...

    2014

  16. [2024]

    Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann

    URL https://arxiv.org/abs/ 2402.08657. Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. Generating automatic feedback on ui mockups with large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY , USA, 20...

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.