Computer Vision and Pattern Recognition · arXiv:2610.37093 v1 · submitted 27 Oct 2026 · 12 references
Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
The paper says
“Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces.”
more of the abstract
We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A well-specified training recipe and an honest ablation, but every headline delta is a single-run point estimate and the small HR-Bench transfer gains sit inside item-level noise. This line is wrong
3 major objections · 8 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 12 references say.
The paper says, at Sections 4.1-4.5; Tables 1-3; Figure 2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
All headline deltas are single point estimates from one training run and one evaluation pass; no seeds, confidence intervals, or significance tests appear anywhere in the paper. This is load-bearing for the central transfer claim: the +2.2 and +3.1 pp HR-Bench gains (Section 4.4) correspond to 4-6 items on a 200-item-per-resolution benchmark (Appendix A), within the worst-case binomial standard error …
more of objection 1
… of about 3.5 pp, and the +3.2 pp ChartQAPro and +3.0 pp PerceptionBench gains have no bound on the dominant unknown, namely the seed-to-seed variance of the G=8 GRPO training itself. The PerceptionBench numbers additionally pass through a single unvalidated LLM judge (DeepSeek-V4-Flash; Section 4.1) with no human-agreement sample, so stage-dependent judge bias toward the RL model's answer style cannot be excluded. To support the abstract-level claim that 'these gains transfer to out-of-domain non-GUI benchmarks... exceeding those obtained from cold-start supervised fine-tuning alone', please report at least three fully independent RL runs per backbone with per-seed scores or paired bootstrap intervals over items, and add a human-judged agreement check on a PerceptionBench subsample for both SFT and RL checkpoints.
The paper says, at Sections 4.3-4.5; abstract; conclusion
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.83):
“Their scores consequently characterize answer generation under different historical observations, rather than the outcome of independently executed crop-selection policies.”
Pith says: Objection 2 of 3 · major
The empirical finding is that RL on GUI data improves non-GUI benchmark scores; the paper's stronger conclusion is that this reflects a learned, domain-general observation strategy ('supporting observation strategies that generalize beyond GUI-specific imitation'). The outcome-only RL row in Table 3 is a well-designed internal control and is credited here: the same GUI data, crop tool, and RL recipe transfer worse than SFT (40.87 vs 42.10 on PerceptionBench), which speaks against a generic 'RL helps everything' confound.
more of objection 2
However, the controls do not cover the data-domain and action-protocol dimension: there is no matched configuration trained on non-GUI spatial supervision or with a different action interface, and no process-level measurement shows that the RL checkpoint selects better crops on the transfer benchmarks. The paper itself acknowledges adjacent limits: Appendix F.1 states that protocol adherence alone is not evidence that RL learned when inspection is useful, and Appendix C states that the reconstruction experiment characterizes answer generation rather than the outcome of independently executed crop-selection policies. The unresolved gap is therefore real. A concrete, feasible test would be to measure, on PerceptionBench items, the frequency with which the first crop or the aggregate crop sequence contains the ground-truth answer region for the SFT versus RL policies, and to show that the change in that rate tracks the +RL gain. Absent such evidence or a matched control, I recommend tempering the interpretation from 'learned observation strategy' to 'transferable benefits under a shared inspection protocol'.
The paper says, at Section 4.5; Table 3
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The reward-design conclusions are read from single-run differences of a few tenths to a few points between configurations: removing the critic lowers ScreenSpot-Pro/PerceptionBench by 0.69/1.81 points, removing the bonus by 0.57/2.50 points, and gating paired with the bonus changes scores by 0.13/1.16 points.
more of objection 3
At the sample sizes involved (3,000 PerceptionBench records; ScreenSpot-Pro application groups of roughly 100-300 items each), these contrasts are within plausible noise, so conclusions such as 'gating does not consistently improve performance across the evaluated configurations' and the mechanism distinction between outcome-only RL (40.87) and the spatially supervised configuration (45.07) rest on unquantified differences. Because the paper's contribution includes the specific claim that supervising intermediate observation decisions drives transfer, this ablation table needs the same seed or bootstrap treatment as the main results; otherwise the corresponding claims should be explicitly framed as single-run observations rather than established effects.
8 minor objections
The paper says, at Abstract; Section 1; Figure 1(d); Table 2
Pith says: minor objection 1 of 8
The same PerceptionBench result is reported as 'approximately 3.0' points (Section 1, Table 2), as 2.97 points (Figure 1(d)), and as '+3.0' in Table 2; please use one consistent precision. Also fix the missing space in 'LookBeyond Screens' (Section 1) and unify the judge name written as 'Deepseekv4-flash-0731' (Section 4.1) with 'DeepSeek-V4-Flash' elsewhere.
The paper says, at Table 1
Pith says: minor objection 2 of 8
The base Qwen3.5-35B-A3B row reports only an overall score of 1.27 with dashes in all application groups; since dashes are defined as unreported scores, it would be clearer to mark this row 'n/a - no reliable protocol following' and to state explicitly what the 1.27 value represents.
show 6 more
The paper says, at Section 4.5; Table 3; Equation (11)
Pith says: minor objection 3 of 8
The relationship between the 'Ungated' rows of Table 3 and the selective response mask of Equation (11) is not stated: if the ungated condition also disables the mask, this should be said explicitly, and if it does not, the marginal effect of response masking on the transfer gains is currently unquantified and should be ablated, since selective response updates are one of the paper's listed contributions.
The paper says, at Equation (6); Section 3.2
Pith says: minor objection 4 of 8
Equation (6) defines K only over successful crop regions, while Section 3.2 states that every admitted crop attempt, including failed executions, consumes a tool-call slot; please clarify whether failed attempts are intentionally excluded from the 0.05K tool-use penalty and from the repetition penalty.
The paper says, at Section 3.1; Equation (2); Section 4.1
Pith says: minor objection 5 of 8
The paper reports no statistics for the rejection sampling in Equation (2), such as the teacher's raw success rate or the yield of DCS, even though the SFT trajectory counts differ by a factor of 3.5 between backbones (35k versus 10k); one sentence with these numbers would contextualize the asymmetry.
The paper says, at Table 4; Appendix C
Pith says: minor objection 6 of 8
Table 4 mixes two checkpoints: the 'Direct reasoning' row uses the base model while the other three rows use the post-RL model; the caveat appears only in the running text, so the caption should state the checkpoint difference, since readers commonly interpret tables standalone.
The paper says, at Section 3.1
Pith says: minor objection 7 of 8
The sentence stating that reference screenshots for synthetic image generation exclude evaluation-benchmark images is a protocol statement rather than a verified audit; please add a near-duplicate or image-similarity check between generated training screenshots and the evaluation images of ScreenSpot-Pro and OSWorld-G-R, or state explicitly that only the reference screenshots were manually curated.
The paper says, at Appendix F.2
Pith says: minor objection 8 of 8
The critic prompt is rendered with an 80-word comment limit in English, while the paper notes the Chinese implementation uses an 80-character limit; since the critic contributes weight 0.3 in Equation (10), the two versions should be aligned or the discrepancy disclosed as a potential cross-lingual scoring difference.
What would settle it
Quantify the uncertainty of the RL minus SFT transfer deltas in one protocol: (i) re-score the existing SFT and RL checkpoints at item level on HR-Bench 4K/8K (n=200 each), ChartQAPro (n=1,948), and PerceptionBench (n=3,000), run paired permutation or bootstrap tests per benchmark, and human-validate a stratified sample of roughly 200 PerceptionBench judge decisions to test for stage-dependent bias; (ii) run one fresh training seed of the Qwen3.5-35B-A3B full recipe and re-evaluate under the identical inference protocol, reporting the seed-to-seed spread of each RL minus SFT delta. If any headline delta fails item-level significance (expected for HR-Bench), if judge agreement differs by training stage, or if the fresh-seed PerceptionBench or ChartQAPro delta is negative or within one seed-spread of zero, the transfer claim is not established at its reported magnitude.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 28 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 8 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Worth a look if you work on agentic perception or RL for VLMs. The real contribution is a concrete training recipe: SFT on teacher-generated GUI grounding trajectories, then GRPO with outcome-gated spatial rewards and selective response masking, evaluated for whether crop-and-inspect behavior transfers to non-GUI benchmarks. Using GUI target boxes as supervision for observation decisions is genuinely new relative to the cited crop-and-inspect and grounding work.
The strongest evidence is Table 3. Outcome-only RL improves ScreenSpot-Pro but drops PerceptionBench (79.19/40.87 vs. SFT's 78.56/42.10), while the spatially supervised configuration improves both. That control argues against "RL just improves instruction following" and points at the spatial reward doing real work. Two backbones move in the same direction on the two main benchmarks, and the paper ships code, models, and unusually candid limitations: crop overlap doesn't score interpretation, Table 4 characterizes supplied context rather than learned behavior, and the appendix admits protocol adherence isn't evidence of learned inspection.
The soft spots are real, and they are exactly where the reader puts them. Every delta is a single evaluation pass of a single run: no seeds, no error bars, no significance tests. The stress-test arithmetic holds. HR-Bench +2.2 and +3.1 pp are 4-6 items out of 200, inside the ~3.4 pp binomial standard deviation; those two deltas should not be doing evidential work. PerceptionBench's +3.0 pp on 3,000 records survives item-level noise, but the judgment passes through one unvalidated LLM judge with no human agreement sample, so stage-dependent judge bias can't be excluded. The Qwen3.8 numbers (+6.13 source, +7.10 transfer) are the most robust-looking in the paper. There is also no non-GUI training control, so "GUI grounding specifically" isn't cleanly separated from "spatially supervised tool-use RL on any data" — a strengthening that is missing, not a refutation, given the Table 3 control already rules out the crudest version.
Circularity is not a concern: transfer benchmarks are external, target boxes are training-only, and the Section 3.1 leakage statement is plausible given the domain gap between synthetic interfaces and natural images.
For whom: anyone building crop-and-inspect agents or using RL for multimodal perception. It deserves a serious referee. I'd send it to review, with a firm request for multi-seed results, judge validation, and ideally a non-GUI training control before the transfer conclusion is leaned on.
The strongest objection
The paper's central quantitative claim is that agentic RL exceeds cold-start SFT on out-of-domain benchmarks: +3.0 and +7.10 pp on PerceptionBench, and +2.2/+3.1/+3.2 pp on HR-Bench 4K/8K and ChartQAPro (Table 2; Section 4.4). Every one of these deltas is a single point estimate from a single evaluation pass of a single training run; no variance estimate, repeated seed, error bar, or significance test appears anywhere in the paper. The smallest deltas are fragile on item-level grounds alone: HR-Bench contains 200 items per resolution (Appendix A), so +2.2 and +3.1 pp are 4-6 items, well within the roughly 3.4 pp binomial standard deviation at the reported accuracy level. ChartQAPro's +3.2 pp (about 60 items of 1,948) and PerceptionBench's +3.0 pp on 3,000 records are larger relative to item-level noise (about 1.1 pp and 0.86 pp), but both remain point estimates with no bound on the dominant unknown: the seed-to-seed spread of the RL training itself, which at G=8 rollouts over one epoch can plausibly reach several percentage points. The headline PerceptionBench numbers additionally pass through a single unvalidated LLM judge (DeepSeek-V4-Flash; Section 4.1), so stage-dependent judge bias toward the RL model's answer style cannot be excluded, and no human agreement sample is reported. The paper's internal controls strengthen the mechanism story - outcome-only RL with the same GUI data and crop tool transfers worse than SFT (40.87 vs 42.10, Table 3), arguing against generic RL protocol-following as the explanation - but they do not bound the deltas. As written, the claim that LBS models 'consistently improve over their corresponding baselines' rests on unquantified point estimates.
The reader's weakest assumption is the absence of a matched non-GUI training control. I agree that this is a real gap, but the paper already contains a partially matched control that weakens the reader's main alternative: outcome-only RL runs the same GUI data, the same crop-and-inspect tool, and the same GRPO machinery with only correctness and format rewards, and it transfers worse than SFT (40.87 vs 42.10 on PerceptionBench, Table 3) despite improving ScreenSpot-Pro. That contrast argues against generic RL-induced instruction following or tool handling as the source of transfer and points to the spatial process reward as the causal ingredient. A non-GUI spatial-supervision control would refine whether GUI content specifically is needed, but a negative finding there would not refute the practical claim that GUI grounding is a workable supervision source; it would relocate the credit to spatial grounding in general. The more load-bearing link is upstream: whether the reported RL-vs-SFT deltas are real at their stated magnitudes. Every delta is a single point estimate; there are no seeds, no error bars, and no significance tests in the manuscript. HR-Bench deltas of 2.2 and 3.1 pp correspond to four to six items on 200-item benchmarks and are within binomial noise; the PerceptionBench deltas (3.0 and 7.10 pp) are large relative to item-level noise but unquantified against training-seed variance, which typically dominates for one-epoch G=8 GRPO runs. The PerceptionBench evaluation also relies on a single LLM judge with no human validation, and Table 3's configuration selection adds a mild multiple-comparison optimism to the Qwen3.5 numbers. These issues do not overturn the reader's CONDITIONAL verdict; they sharpen its conditions. Credit is due where the paper is strong: the outcome-only ablation is a genuinely informative control, the Qwen3.8-27B PerceptionBench gain of 7.10 pp is large, the qualitative interaction traces and the Table 4 reconstruction analysis are consistent with the stated mechanism, limitation statements are honest (e.g., Sections 4.6 and Appendix C disclaim crop-feedback isolation), and the code and models are released. The verdict remains CONDITIONAL, with the added condition that the authors report seed-level variance or item-level significance and a human-validated judge sample.
The referee's report
If the quantitative claims survive scrutiny, this is a useful and non-obvious contribution: it would demonstrate that a purely GUI-sourced spatial supervision signal can improve non-GUI visual perception, which is practically attractive because GUI grounding annotations are cheap to synthesize at scale. The paper ships public code and model weights, and Appendix B specifies the GRPO variant, masking, and crop construction in checkable form. I credit explicitly: (i) the outcome-only RL control in Table 3, which is the right internal experiment to argue against generic-RL confounds; (ii) the reconstructed-observation-history design in Section 4.6 and Appendix C, which cleanly separates crop-image availability and upsampling scale from online crop selection; and (iii) the paper's honest scoping statements, including the acknowledgment that protocol adherence alone is not evidence of learned inspection utility. The main risk is statistical: the headline transfer deltas are single-run point estimates, and the gap between 'transferable gains' and 'a learned domain-general observation strategy' is not yet closed by process-level evidence.
Pith's reading of the paper (a machine's summary, not the authors')
This paper asks whether spatial supervision from GUI grounding — teaching a vision-language model to click targets on interface screenshots — can produce an observation strategy that transfers to non-GUI visual tasks. The authors answer yes: after supervised fine-tuning on teacher-generated grounding trajectories followed by reinforcement learning over multi-turn crop-and-inspect decisions, two Qwen-based models improve on GUI grounding and also on out-of-domain benchmarks including PerceptionBench, HR-Bench, and ChartQAPro. The RL stage contributes gains beyond supervised fine-tuning alone, roughly 3.0 and 7.1 percentage points on PerceptionBench for the two backbones. If the interpretation holds, interface screenshots become a cheap, scalable training signal for active visual perception, a capability that usually requires expensive domain-specific trajectory data.
What carries the argument. The carrying mechanism is the two-stage pipeline: cold-start SFT imitates teacher-generated multi-turn trajectories (reason, crop, observe, then click a point inside the target box), and RL then optimizes the same protocol with a trajectory-level reward that gates spatial feedback on terminal grounding correctness. The spatial component scores the crop sequence by the best IoU between crop regions and the training-only target box, penalizes repeated crops, declining overlap, and tool-call count, and adds a small-target bonus for crops containing the target center; a response mask suppresses direct policy updates on intermediate turns whose crops have near-zero target relevance. The effect is that the policy is rewarded for inspection sequences that gather useful evidence, not merely for producing a correct final coordinate.
Core claim. The paper's central claim is that GUI grounding can serve as a scalable source of spatial supervision for learning active visual observation, and that the resulting observation policy transfers to non-GUI perception. Concretely, cold-start SFT on rejection-sampled teacher trajectories followed by multi-turn RL with outcome-gated spatial rewards moves Qwen3.5-35B-A3B from 78.56 to 80.39 on ScreenSpot-Pro and from 42.1 to 45.1 on PerceptionBench, and Qwen3.8-27B from 75.02 to 81.15 and from 52.57 to 59.67, with the Qwen3.5-based policy also improving HR-Bench 4K/8K and ChartQAPro. The authors further claim that the reward design carries the transfer: outcome-only RL improves GUI grounding but lowers PerceptionBench accuracy relative to SFT, whereas the spatially supervised configuration improves both, showing that better source-domain performance does not by itself imply better non-GUI perception.
The premise it rests on. The interpretation of the transfer gains as a learned, domain-general observation strategy rests on the assumption that GUI spatial supervision is the active ingredient, but all training data and rewards are GUI-specific, no control runs the same recipe on non-GUI supervision or with a different action space, and the only leakage check is the statement that reference screenshots exclude evaluation images, so generic RL-induced instruction-following or format adherence could in principle explain the improvements.
If the paper is right (Pith's inferences, not the authors')
Synthetic GUI data becomes a practical, scalable supervision source for active visual perception, potentially reducing the need to build specialized observation trajectories for each downstream domain.
Observation-policy gains on OCR, counting, and comparison tasks suggest RL-trained inspection helps most where local detail and evidence gathering matter, so similar recipes may help other fine-grained perception tasks.
The outcome-only versus spatial-reward comparison implies that optimizing final correctness on a source domain can be insufficient or harmful for transfer, making process-level spatial supervision a design choice worth adopting in other grounding settings.
Because crop presentation scale (4x upsampling) affects both GUI and non-GUI accuracy independent of crop selection, inference-time rendering decisions are part of the effective observation policy and should be tuned alongside it.
An untested corollary: if the transfer is carried by the spatial-supervision mechanism rather than by GUI data per se, the same recipe should work with non-GUI spatial annotations, such as referring-expression boxes on natural images — a control the paper does not run.
The paper's account predicts a specific behavioral signature: the RL policy should show larger gains on images with distractors or small targets than on images where a single glance suffices, which could be tested by partitioning PerceptionBench items by target size and clutter.
The credit-assignment scheme — reward inspections whose geometry indicates target relevance, mask turns that are merely redundant — reads as a general principle for sequential perception, and the authors' own discussion points to video frame selection as the natural next domain to port it to.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-28
The authors' main claim, as the classifier picked it out 1.00
“These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning.”
How the main claim is stated: asserted 0.68
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.12 · human participants 0.08 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.10 · compares with prior results 0.95 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Visual perception is evolving from passive recognition toward active, task-oriented observation. | background 1.00 | 0.06 | 0.04 | asserted 1.00 |
| We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. | contribution 0.90 | 0.56 | 0.08 | asserted 0.78 |
| We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. | method 0.80 | 0.95 | 0.07 | asserted 1.00 |
| Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. | result 1.00 | 0.93 | 0.09 | asserted 1.00 |
| Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. | result 0.97 | 0.95 | 0.23 | asserted 1.00 |
| Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. | interpretation 0.99 | 0.87 | 0.10 | hedged possibility 0.99 |
| These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. | interpretation 1.00 | 0.76 | 0.84 | asserted 0.68 |
| Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively. | contribution 0.84 | 0.50 | 0.02 | asserted 0.98 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 6,290 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (12)
- [1] Before returning the final coordinate, you SHOULD make at least one successful crop_image call to inspect the target more precisely. A direct first-turn coordinate is discouraged. Skip cropping only when the task is clearly infeasible or cropping cannot reasonably improve localization on Pith
- [2] You may make at most {max_tool_calls} crop_image calls in total. Never exceed this budget; use each crop purposefully and return the final coordinate as soon as the target is localized. A single assistant turn may contain one or two crop_image actions, and every action counts ... on Pith
- [3] Tool arguments are local absolute pixels in the image_path passed to that call. Write x1, y1, x2, and y2 with exactly two digits after the decimal point, including trailing zeros (for example, x1=100.00). Two actions emitted in the same turn must refer only to image paths alre... on Pith
- [4] Each result states the crop's region both in original-image pixels and original-image normalized [0,1000] coordinates. Use that mapping to convert any point found in a crop back to the ORIGINAL image's normalized [0,1000] coordinate system on Pith
- [5] After crop_image actions, the following single user message contains all result metadata and all successful crop images in action order. 25 Learning to Look Beyond Screens Agentic RL system promptpRL (continued) Final-answer rule: When ready, output exactly <think>...</think> ... on Pith
- [6] Annotation alignment: against the reference annotation (GT bbox or infeasibility flag), whether the cropped regions and coordinate directions are progressively approaching the correct location; for refusal samples, evaluate whether the justification for judging the task as inf... on Pith
- [7] Goal orientation: whether each turn's thinking revolves around "how to locate the target of the task instruction" on Pith
- [8] Feedback utilization: whether subsequent thinking leverages the results of previous crop executions (successful region ranges / failure reasons) on Pith
- [9] Thought-action consistency: whether the regions and directions described in the thinking match the coordinate parameters in the actions on Pith
- [10] Incremental progress: whether each turn brings new information or new decisions, without repeating the same crop or reasoning on Pith
- [11] Conciseness: no meaningless filler or parroting. Scoring anchors: - 0.9~1.0: The thinking stays focused on locating the target throughout, fully leverages the crop feedback from each turn, is consistent with the action parameters, converges step by step, and is concise without... on Pith
- [2017] with learning rate10−6, a cosine schedule without warmup, weight decay0.1, (β 1,β 2) = (0.9, 0.95), and gradient clipping at norm1.0. The global batch contains 512 trajectories (64 tasks with 8 rollouts each) across64 NPUs, with a per-device micro-batch size of1 and 8 gradient... on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment