Pith Number

pith:AO6BK5V7

pith:2026:AO6BK5V74MJ4CT5CUTUPHZTKEB

not attested not anchored not stored refs pending

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

Biao Yi, Huajun Chen, Songze Li, Tianqi Liu, Wen Zhang, Xiaoke Guo, Zhaoyan Gong, Zhiqiang Liu

Treating GUI reasoning as a cyclic Screen-UI-Action process lets MLLMs explicitly learn element localization, semantics, and usage for more precise and interpretable decisions.

arxiv:2604.06995 v2 · 2026-04-08 · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{AO6BK5V74MJ4CT5CUTUPHZTKEB}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise element discovery and performs interpretable reasoning.

C2weakest assumption

That inserting an explicit UI-element learning step into a cyclic Screen-UI-Action process will produce both higher accuracy and better interpretability in MLLMs, without the added structure introducing new failure modes or requiring impractical amounts of supervision.

C3one line summary

UI-in-the-Loop makes multimodal models explicitly learn UI element locations, meanings, and uses in a cyclic screen-element-action loop, delivering better UI comprehension and GUI reasoning on a new 26K-sample benchmark.

Cited by

1 paper in Pith

TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

Receipt and verification

First computed	2026-06-02T01:03:46.790309Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

03bc1576bfe313c14fa2a4e8f3e66a207c3ddae444676165c3d5d5068eb6ca29

Aliases

arxiv: 2604.06995 · arxiv_version: 2604.06995v2 · doi: 10.48550/arxiv.2604.06995 · pith_short_12: AO6BK5V74MJ4 · pith_short_16: AO6BK5V74MJ4CT5C · pith_short_8: AO6BK5V7

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/AO6BK5V74MJ4CT5CUTUPHZTKEB \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 03bc1576bfe313c14fa2a4e8f3e66a207c3ddae444676165c3d5d5068eb6ca29

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "d855ca8afa8f69d148a7dc780e8dbdad764e529552e63863fcb918193d6ace74",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.AI",
    "submitted_at": "2026-04-08T12:12:09Z",
    "title_canon_sha256": "e1c36b436ae993ba052abaa403f4afc28f0e3a81eaef99b709c1dfe563392e99"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2604.06995",
    "kind": "arxiv",
    "version": 2
  }
}