{"id":"ed8764c5-6aaa-46c5-b2e4-78fd83df3d46","arxiv_id":"2607.11598","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Grounded interaction with external instruments is a third test-time compute axis that escapes the internal ceiling of reasoning and sampling, but only when both feedback and evaluation observe the actual defect.","lead":"Spending more compute at test time by letting a model interact with external tools (run tests, measure layouts) beats longer reasoning or more sampling, but only when both the feedback and the scoring metric actually observe the real defect. The usual screenshot-based AI judge rates broken figures as perfect, hiding the gains.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The visual half of the central claim rests on an unvalidated mapping from hand-tuned DOM defect counts to human quality, while the same instrument both drives and scores the loop.","rationale":"The reader's weakest_assumption correctly isolates the softest load-bearing point: the unvalidated DOM-to-human mapping under a keep-best loop that optimizes the same instrument. Code results (matched-budget saturation past oracle best-of-N, 100% with zero seed variance across three families, held-out recovery) are tightly controlled and do not depend on that mapping; they alone establish that grounded interaction is distinct from reasoning and sampling. The visual results and the structural-blindness argument for screenshot judges are directionally strong (VLM rates 14/15 broken figures perfect; geometry reviewer vs. VLM reviewer opposite signs) but remain partly circular until human preference confirms that the measured defects are the ones people care about. No stronger internal inconsistency appears; the information-theoretic ceiling argument is only motivational and the authors treat measurement as the evidence. Therefore the reader's CONDITIONAL verdict (accept-shaped for code/framework, conditional on human validation for visual) is the right one; no adjustment is needed. The concrete test above is exactly the study the authors already list as future work and would settle the concern.","tokens_in":20325,"tokens_out":695,"duration_ms":6722,"concrete_test":"Run a blinded pairwise preference study on the exact single-shot vs. reviewed pairs already scored in Table 6 / Figure 12 (figures, slides, web, animations; ≥20 decisive pairs per modality). Present full-page renders (not screenshots) to human raters; compute preference rate for the reviewed artifact and correlation of DOM-defect delta with preference. If preference for reviewed is ≤55% or correlation with defect reduction is near zero on any modality, the visual half of the claim (and Prediction 3) weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest claim is two-sided: interaction escapes the internal ceiling on code (well-supported by oracle best-of-N, cross-family zero-variance 100% pass rates, and held-out recovery) and the same grounded loop removes 40–74% of layout defects on four visual modalities, with the gain invisible to (and sometimes reversed by) a screenshot VLM. The visual half is load-bearing for the two-sided grounding thesis and Prediction 3. Yet the DOM-geometry instrument (Section 3: ≥6 px text-on-text, >16 px document overflow, box-group misalignment rules) both supplies the defect list that drives revision and is the sole reported scorer; the harness keeps the best-scoring iteration, so some reduction is mechanically guaranteed. Authors correctly flag that a quantitative human-preference study is future work (Section 6 caveats; Section 9). Without that mapping, the reported defect reductions could be metric optimization rather than human-visible quality gains, and the claim that ungrounded VLM reviewers regress quality (Figure 10) is only decisive under the same unvalidated metric. The code axis stands independently; the visual axis and the full 'both sides of the loop' framing do not until the mapping is checked.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that test-time compute has a third axis, interaction, distinct from internal reasoning and sampling: a frozen model proposes an artifact, an external instrument observes its actual behavior, and the model revises. The governing variable is two-sided grounding—both the feedback that drives revision and the metric that scores the result must come from an instrument that observes the relevant defect (the coverage principle). On hard coding tasks at matched token budgets, reasoning-only and oracle best-of-N plateau while a proposer–reviewer harness reaches 100% pass rate with zero seed variance across three model families and a held-out suite. On rendered visual artifacts, a screenshot VLM rates most broken figures perfect and can regress slide layouts as a reviewer, whereas a deterministic DOM-geometry instrument reports 40–74% defect reductions across four modalities. Two modalities (video, deep research) are reported as honest negatives with no single-shot headroom.","tokens_in":20730,"tokens_out":1476,"duration_ms":22409,"significance":"If the results hold, the paper cleanly separates interaction from reasoning and sampling under a matched budget, with an information-theoretic motivation (Appendix A) and a falsifiable coverage principle that organizes the ablations. The code evidence is unusually strong for this literature: fixed budgets, oracle best-of-N as an internal ceiling, three seeds, cross-family replication (Sonnet 4, Qwen3-235B, GPT-5), held-out recovery with no regressions, sign tests and bootstrap CIs, and released code. The structural-blindness critique of screenshot VLM judges is practically important and well illustrated. The distillation section’s caution that RFT can spend the variance best-of-N needs is a useful secondary finding. These strengths make the work a serious contribution to test-time compute, contingent on tightening the visual evaluation half of the two-sided claim.","major_comments":[{"comment":"Section 6 and Limitations (Section 9): the visual half of the central claim (Prediction 3; Figs. 10–12; Table 6) rests on DOM-geometry defect counts that both drive revision and score the result, with keep-best guaranteeing non-worsening. Thresholds (≥6 px text overlap, >16 px document overflow, box-group misalignment rules) are free parameters. The authors correctly flag that a quantitative human-preference study is future work, but without it the reported 40–74% reductions and the claim that a VLM reviewer regresses quality are only decisive under an unvalidated metric. This is load-bearing for the two-sided grounding thesis. A modest preference study (or at least a blinded human defect audit on a stratified sample of single-shot vs. reviewed pairs) is needed before the visual results can carry equal weight with the code results.","section":"Section 6; Section 9; Figure 12; Table 6"},{"comment":"Section 3 and Table 6: the geometry instrument’s defect definitions and thresholds are not accompanied by a sensitivity analysis. Because the same instrument is the sole reported scorer, it is unclear how much of the effect size depends on the particular cutoffs versus robust layout failures. Reporting defect reductions under a small grid of thresholds (or an ablated subset of defect classes) would show that the qualitative claim is not an artifact of one hand-chosen rule set.","section":"Section 3; Table 6"},{"comment":"Section 4 / Table 2 and the visual suites: several headline numbers rest on modest N (15 hard code tasks; 12 dense slides; 15–20 per visual modality). The code path is partly rescued by three seeds, cross-family replication, and a 32-task held-out suite (Table 8), but the visual effect sizes and the VLM-reviewer regression (Figure 10) would be more convincing with larger or more diverse task sets, or at least explicit power discussion for the paired sign tests. This is secondary to the human-validation issue but still load-bearing for generalizing Prediction 3.","section":"Section 4; Figure 10; Table 2; Table 6"}],"minor_comments":[{"comment":"Figure 5 and Table 2: the small non-monotonic dip in best-of-N from B=1K to B=5K is explained as seed noise, but a one-sentence note in the main text (not only the table caption) would prevent misreading of the internal-ceiling plot.","section":"Figure 5; Table 2"},{"comment":"Section 2.2 / Figure 4: classifying a screenshot-fed VLM as ungrounded is justified in the text, but the figure’s orange lane could more explicitly mark that the lossy channel, not the pixels alone, is what places it in the ungrounded category.","section":"Section 2.2; Figure 4"},{"comment":"Appendix A: the DPI argument correctly bounds information channels rather than achievable quality; the main text already notes this, but a single forward pointer in Section 2.3 to the two cautions at the end of Appendix A would help readers who stop at the informal one-sentence claim.","section":"Section 2.3; Appendix A"},{"comment":"Table 3 vs. Table 2: the single-seed reasoning-only numbers and the 3-seed aggregate can be misread as the same run; a clearer cross-reference in the caption of Table 3 would reduce confusion.","section":"Table 3"},{"comment":"Section 7 / Figure 14: the RFT variance finding is interesting but somewhat orthogonal; consider shortening or moving more of the distillation tables to the appendix so the main narrative stays on the three predictions.","section":"Section 7"},{"comment":"Minor polish: arXiv date line and a few long figure captions (e.g., Figure 1) could be tightened; ensure all modality N values in Table 1 match the statistics tables.","section":null}],"recommendation":"major_revision","confidential_remarks":"The code axis alone is close to a solid contribution; the risk is that the two-sided framing and the visual claims are currently carrying more rhetorical weight than the unvalidated DOM metric can support. I would not reject on novelty grounds—the matched-budget oracle best-of-N comparison and the structural-blindness isolation of screenshot judges are real increments over Reflexion/CRITIC/tool-use work—but I would hold the visual half to a human check before treating Prediction 3 as established. Fit for a serious AI/ML journal is good if the revision lands."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that the coding results are unusually clean for this area. At matched token budgets on hard tasks, reasoning-only and even oracle best-of-N (selecting against the same pytest suite) both flatten, while every execution-feedback loop keeps climbing; their proposer-reviewer harness reaches strict 100% with zero seed variance across three model families and recovers every first-shot failure on a held-out suite. That is not just another Reflexion demo.\n\nWhat is actually new is the joint framing: interaction as a third axis of test-time compute, governed by two-sided grounding (the instrument must observe the defect for both the revision signal and the score) plus a coverage principle that predicts when a grounded channel helps or hurts. The controls land: linter buys nothing on runtime bugs, screenshot VLM reviewer makes slide geometry worse while the DOM instrument repairs it, and the default VLM-on-screenshot judge rates 14/15 visibly broken figures perfect. They ship code, report honest negatives on saturated modalities, and the DPI appendix is a clean formalization of the internal ceiling rather than hand-waving.\n\nSoft spots are real but proportionate. On visuals the same DOM-geometry instrument (hand thresholds for overlap, overflow, misalignment) both drives revision and scores, and keep-best guarantees non-worsening, so some of the 40-74% reduction is mechanical; they correctly call the missing human-preference study future work. Custom suites and free parameters (caps, splits, design prompt) exist, but the code path shares its oracle with the baseline so circularity is low there. Distillation is secondary and the RFT variance caution is useful but not load-bearing.\n\nThis is for people building coding agents, visual generators, or eval pipelines who need a crisp reason why tool loops beat longer CoT or more samples, and why screenshot judges hide the gain. The math is light but sound, data and citations are solid, and the thinking is honest. I would bring it to reading group, cite the framework and code results, and a serious editor should send it to referees rather than desk-reject; the core is ready even if visual needs the human map or tighter caveats.","headline":"Code axis is clean and strong: grounded interaction hits 100% with zero variance where reasoning and oracle best-of-N plateau; visual half is real but partly optimizes its own scorer, so the full two-sided claim needs the human check they flag.","tokens_in":21330,"tokens_out":572,"would_cite":true,"duration_ms":13520,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Interaction with grounded instruments is a third axis of test-time compute that breaks the ceiling reasoning and sampling hit.","keywords":["test-time compute","interaction scaling","grounding","proposer-reviewer harness","execution feedback","layout geometry","model-as-judge","coverage principle"],"falsifier":"A human preference study on the same figure, slide, web, and animation suites showing that lower measured DOM-defect counts do not predict human preference, or a matched-budget experiment in which oracle best-of-N equals or exceeds the grounded interaction harness once human raters replace the instrument score.","tokens_in":21181,"feed_emoji":"🔁","tokens_out":971,"duration_ms":25473,"temperature":0.7,"pith_summary":"The paper argues that test-time compute has three axes, not two. Reasoning longer and sampling more attempts are both internal: every extra token still comes from the same frozen weights and the same prompt, so neither can tell the model anything it does not already know. Interaction—proposing an artifact, letting an external instrument observe how it actually behaves, then revising—imports a real observation each cycle and can keep improving past that internal ceiling. The single governing variable is grounding, and it must hold on both sides of the loop: the feedback that drives revision must come from an instrument that actually sees the flaw, and so must the metric that scores the result. On hard coding tasks the interaction harness reaches a perfect pass rate with no variance while reasoning and even oracle best-of-N plateau; on visual artifacts the usual screenshot judge is structurally blind, and only a geometry-measuring tool reveals the large defect reductions the loop actually achieves.","feed_headline":"Grounded interaction hits 100% where sampling plateaus","feed_subtitle":"Both feedback and the score must come from real instruments, not model opinions of screenshots.","key_machinery":"Grounding on both sides of the interaction loop, plus the coverage principle: feedback is grounded only when an external instrument observes the artifact’s real form or behavior, and it helps exactly as far as that instrument can observe the defects present. The same observation must also serve as the evaluation metric, or the gain remains invisible.","core_discovery":"At a fixed token budget on hard coding tasks, reasoning-only and best-of-N sampling (even with an oracle verifier) both saturate, while every strategy that iterates on grounded execution feedback keeps climbing; a proposer–reviewer harness reaches 100% pass rate with zero seed variance across three model families. On rendered visual artifacts the default vision-language model judge rates 14 of 15 visibly broken figures “perfect” because screenshots hide the flaws; a deterministic layout instrument instead shows the same loop removing 40–74% of defects, and that same VLM used as reviewer makes slides worse where the measuring tool repairs them.","pith_inferences":["Results in self-improving visual generation that rely solely on VLM-on-screenshot judges may systematically overstate quality or miss real layout failures.","The two-sided grounding requirement extends naturally to audio, 3D, tabular, and interactive UI artifacts where model judges are similarly blind to physical properties.","Industrial coding harness gains already seen in large deployments are predicted once the instrument’s coverage is matched to the defect class.","Post-training that collapses output variance can trade away the very resource best-of-N sampling converts into solved tasks."],"forward_implications":["Reasoning and sampling alone cannot produce candidates outside the high-probability region of the model’s own distribution, even with perfect selection.","A frozen frontier model wrapped in a proposer–reviewer harness that feeds instrument observations reaches higher quality at lower token cost and with zero run-to-run variance.","For layout-sensitive visual artifacts, screenshots must be replaced by direct geometry measurement on both the reviewer and the scorer.","Using an ungrounded model as reviewer can actively regress quality on defects its channel cannot see.","Distilling teacher interaction trajectories into a small student recovers a substantial fraction of teacher quality at much lower deployment cost, provided sampling variance is retained."],"fun_headline_variants":["Grounded interaction hits 100% as sampling saturates","Proposer-reviewer reaches perfect code pass with zero variance","Real instruments keep climbing where reasoning and sampling plateau","Layout tools cut 40-74% defects that VLM judges miss","Both feedback and metric need grounded instruments not model opinions"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The paper treats reductions in its deterministic DOM-geometry defect counts—using fixed pixel thresholds for overlap, overflow, and misalignment—as faithful measures of quality a human would prefer.","fun_headline_variants_meta":{"raw":{"variants":["Grounded interaction hits 100% as sampling saturates","Proposer-reviewer reaches perfect code pass with zero variance","Real instruments keep climbing where reasoning and sampling plateau","Layout tools cut 40-74% defects that VLM judges miss","Both feedback and metric need grounded instruments not model opinions"]},"model":"grok-4.5","effort":"low","cost_usd":0.003556,"raw_usage":{"total_tokens":1274,"prompt_tokens":926,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":35560000,"prompt_tokens_details":{"text_tokens":926,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":264,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":926,"tokens_out":84,"duration_ms":3265,"temperature":1.0,"reasoning_tokens":264,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T04:30:23.820024+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A human preference study on the same figure, slide, web, and animation suites showing that lower measured DOM-defect counts do not predict human preference, or a matched-budget experiment in which oracle best-of-N equals or exceeds the grounded interaction harness once human raters replace the instrument score.","supporting_citations":[],"review_version":1}