REVIEW 3 major objections 6 minor 37 references
Interaction Scaling: Grounding the Third Axis of Test-Time Compute
T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read Interaction with grounded instruments is a third axis of test-time compute that breaks the ceiling reasoning and sampling hit.
desk verdict Code axis is clean and strong: grounded interaction hits 100% with zero variance where reasoning and oracle best-of-N plateau; visual half is real but partly optimizes its own scorer, so the full two-sided claim needs the human check they flag. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Grounding on both sides of the interaction loop, plus the coverage principle: feedback is grounded only when an external instrument observes the artifact’s real form or behavior, and it helps exactly as far as that instrument can observe the defects present. The same observation must also serve as the evaluation metric, or the gain remains invisible.
What would settle it
A human preference study on the same figure, slide, web, and animation suites showing that lower measured DOM-defect counts do not predict human preference, or a matched-budget experiment in which oracle best-of-N equals or exceeds the grounded interaction harness once human raters replace the instrument score.
Extended reading notes
Core claim
At a fixed token budget on hard coding tasks, reasoning-only and best-of-N sampling (even with an oracle verifier) both saturate, while every strategy that iterates on grounded execution feedback keeps climbing; a proposer–reviewer harness reaches 100% pass rate with zero seed variance across three model families. On rendered visual artifacts the default vision-language model judge rates 14 of 15 visibly broken figures “perfect” because screenshots hide the flaws; a deterministic layout instrument instead shows the same loop removing 40–74% of defects, and that same VLM used as reviewer makes slides worse where the measuring tool repairs them.
Load-bearing premise
The paper treats reductions in its deterministic DOM-geometry defect counts—using fixed pixel thresholds for overlap, overflow, and misalignment—as faithful measures of quality a human would prefer.
Editorial extensions
If this is right
- Reasoning and sampling alone cannot produce candidates outside the high-probability region of the model’s own distribution, even with perfect selection.
- A frozen frontier model wrapped in a proposer–reviewer harness that feeds instrument observations reaches higher quality at lower token cost and with zero run-to-run variance.
- For layout-sensitive visual artifacts, screenshots must be replaced by direct geometry measurement on both the reviewer and the scorer.
- Using an ungrounded model as reviewer can actively regress quality on defects its channel cannot see.
- Distilling teacher interaction trajectories into a small student recovers a substantial fraction of teacher quality at much lower deployment cost, provided sampling variance is retained.
Reading between the lines
- Results in self-improving visual generation that rely solely on VLM-on-screenshot judges may systematically overstate quality or miss real layout failures.
- The two-sided grounding requirement extends naturally to audio, 3D, tabular, and interactive UI artifacts where model judges are similarly blind to physical properties.
- Industrial coding harness gains already seen in large deployments are predicted once the instrument’s coverage is matched to the defect class.
- Post-training that collapses output variance can trade away the very resource best-of-N sampling converts into solved tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that test-time compute has a third axis, interaction, distinct from internal reasoning and sampling: a frozen model proposes an artifact, an external instrument observes its actual behavior, and the model revises. The governing variable is two-sided grounding—both the feedback that drives revision and the metric that scores the result must come from an instrument that observes the relevant defect (the coverage principle). On hard coding tasks at matched token budgets, reasoning-only and oracle best-of-N plateau while a proposer–reviewer harness reaches 100% pass rate with zero seed variance across three model families and a held-out suite. On rendered visual artifacts, a screenshot VLM rates most broken figures perfect and can regress slide layouts as a reviewer, whereas a deterministic DOM-geometry instrument reports 40–74% defect reductions across four modalities. Two modalities (video, deep research) are reported as honest negatives with no single-shot headroom.
Significance. If the results hold, the paper cleanly separates interaction from reasoning and sampling under a matched budget, with an information-theoretic motivation (Appendix A) and a falsifiable coverage principle that organizes the ablations. The code evidence is unusually strong for this literature: fixed budgets, oracle best-of-N as an internal ceiling, three seeds, cross-family replication (Sonnet 4, Qwen3-235B, GPT-5), held-out recovery with no regressions, sign tests and bootstrap CIs, and released code. The structural-blindness critique of screenshot VLM judges is practically important and well illustrated. The distillation section’s caution that RFT can spend the variance best-of-N needs is a useful secondary finding. These strengths make the work a serious contribution to test-time compute, contingent on tightening the visual evaluation half of the two-sided claim.
major comments (3)
- [Section 6; Section 9; Figure 12; Table 6] Section 6 and Limitations (Section 9): the visual half of the central claim (Prediction 3; Figs. 10–12; Table 6) rests on DOM-geometry defect counts that both drive revision and score the result, with keep-best guaranteeing non-worsening. Thresholds (≥6 px text overlap, >16 px document overflow, box-group misalignment rules) are free parameters. The authors correctly flag that a quantitative human-preference study is future work, but without it the reported 40–74% reductions and the claim that a VLM reviewer regresses quality are only decisive under an unvalidated metric. This is load-bearing for the two-sided grounding thesis. A modest preference study (or at least a blinded human defect audit on a stratified sample of single-shot vs. reviewed pairs) is needed before the visual results can carry equal weight with the code results.
- [Section 3; Table 6] Section 3 and Table 6: the geometry instrument’s defect definitions and thresholds are not accompanied by a sensitivity analysis. Because the same instrument is the sole reported scorer, it is unclear how much of the effect size depends on the particular cutoffs versus robust layout failures. Reporting defect reductions under a small grid of thresholds (or an ablated subset of defect classes) would show that the qualitative claim is not an artifact of one hand-chosen rule set.
- [Section 4; Figure 10; Table 2; Table 6] Section 4 / Table 2 and the visual suites: several headline numbers rest on modest N (15 hard code tasks; 12 dense slides; 15–20 per visual modality). The code path is partly rescued by three seeds, cross-family replication, and a 32-task held-out suite (Table 8), but the visual effect sizes and the VLM-reviewer regression (Figure 10) would be more convincing with larger or more diverse task sets, or at least explicit power discussion for the paired sign tests. This is secondary to the human-validation issue but still load-bearing for generalizing Prediction 3.
minor comments (6)
- [Figure 5; Table 2] Figure 5 and Table 2: the small non-monotonic dip in best-of-N from B=1K to B=5K is explained as seed noise, but a one-sentence note in the main text (not only the table caption) would prevent misreading of the internal-ceiling plot.
- [Section 2.2; Figure 4] Section 2.2 / Figure 4: classifying a screenshot-fed VLM as ungrounded is justified in the text, but the figure’s orange lane could more explicitly mark that the lossy channel, not the pixels alone, is what places it in the ungrounded category.
- [Section 2.3; Appendix A] Appendix A: the DPI argument correctly bounds information channels rather than achievable quality; the main text already notes this, but a single forward pointer in Section 2.3 to the two cautions at the end of Appendix A would help readers who stop at the informal one-sentence claim.
- [Table 3] Table 3 vs. Table 2: the single-seed reasoning-only numbers and the 3-seed aggregate can be misread as the same run; a clearer cross-reference in the caption of Table 3 would reduce confusion.
- [Section 7] Section 7 / Figure 14: the RFT variance finding is interesting but somewhat orthogonal; consider shortening or moving more of the distillation tables to the appendix so the main narrative stays on the three predictions.
- Minor polish: arXiv date line and a few long figure captions (e.g., Figure 1) could be tightened; ensure all modality N values in Table 1 match the statistics tables.
Circularity Check
Code path is non-circular (external pytest oracle shared with best-of-N); visual path has real circular content because the same DOM-geometry instrument both drives revision and scores, and keep-best guarantees non-worsening under that metric.
-
self definitional
[Section 3 (Architecture / Budget and protocol) + Section 6 (Is the reduction circular?)]
"A reviewer (the same model in a diagnostic role) turns that signal into a structured list of defects, and the proposer revises. The loop repeats up to an iteration cap and keeps the best-scoring iteration, so the reviewed result can never score below single-shot under the same metric. ... Is the reduction circular? The instrument both drives the revision and scores it, and the harness keeps the best-scoring iteration, so some reduction is mechanically guaranteed."
The DOM-geometry instrument is defined as both the feedback channel and the evaluation metric. Combined with keep-best, non-worsening (and some positive reduction) under that metric is true by construction of the harness, not an independent empirical discovery. The paper acknowledges the mechanical guarantee; the remaining claim is only about magnitude and human relevance, which the authors defer to future human-preference work.
-
fitted input called prediction
[Section 3 (Rendered geometry instrument) + Section 6 / Figure 12 / Table 6]
"We compute, exactly: text-on-text overlap (≥ 6 px), out-of-bounds/clipping, container overflow, document overflow (> 16 px), and box-group misalignment (rows or columns of card-like boxes flagged for unequal size, misaligned far edges, or uneven gutters). ... Grounded geometric feedback removes real layout defects on all four visual modalities. Mean defect reduction ... −74% / −73% / −47% / −40% ... The principal remaining caveat is a quantitative human-preference study confirming that DOM-defect reduction maps onto perceived quality across the full suite."
The defect counts that are reported as the main visual result (40–74% reductions) are produced by hand-chosen thresholds and alignment rules that the same instrument optimizes. Without an external human-preference or independent quality mapping, the 'prediction' that grounded interaction improves visual quality reduces to optimization of the authors' own defect definition. The ungrounded-VLM ablation (Figure 10) is a useful control under the same metric but does not break the circle relative to human quality.
full rationale
The paper's central thesis is two-sided grounding: interaction escapes the internal ceiling, but only when both feedback and metric are grounded. On code this is cleanly non-circular. The instrument is pytest pass/fail + traceback; the oracle best-of-N baseline selects against the same tests (Section 3, Section 4, Table 2), so the 100% ceiling and the gap past oracle best-of-N are not defined by a model opinion or by a fitted parameter. Cross-family zero-variance 100% and held-out recovery further show the result is not a tautology of the harness. The visual half is different. Section 3 defines a deterministic DOM-geometry instrument (text-on-text overlap ≥6 px, document overflow >16 px, box-group misalignment rules) that both feeds the defect list into revision and is the sole reported scorer for figures, slides, web, and animations. The harness 'keeps the best-scoring iteration, so the reviewed result can never score below single-shot under the same metric' (Section 3). Section 6 itself asks 'Is the reduction circular?' and answers with three mitigations (real defects, non-trivial magnitude, ungrounded-VLM-reviewer ablation that moves the same metric the wrong way), while Section 9 and the Section 6 caveats correctly flag that a quantitative human-preference study remains future work. Those mitigations reduce but do not eliminate the construction: some reduction is mechanically guaranteed by keep-best under the instrument that also drives the loop, and the claim that ungrounded VLM reviewers regress quality (Figure 10) is decisive only under that same unvalidated metric. No self-citation chain or uniqueness theorem is load-bearing; the circularity is the fitted-input / self-definitional pattern on the visual metric. Overall score 4: partial circularity confined to the visual half of Prediction 3 and the two-sided framing; the code axis and the information-theoretic internal-ceiling argument stand independently.
Assumptions & free parameters
free parameters (4)
- geometry defect thresholds (overlap ≥6 px, document overflow >16 px, box-group misalignment rules)
- iteration caps (≤3 geometry, ≤5 code) and keep-best-iteration rule
- token budget B and proposer/reviewer split
- design-principle generation prompt and temperature settings
assumptions (5)
- standard math Data-processing inequality: ungrounded post-processing of model outputs cannot increase information about the correct artifact beyond what is in weights and prompt (Appendix A).
- domain assumption An instrument observation E = g(A, world) can carry information about defects not already in (Θ, X), so conditioning generation on E can produce candidates the model would not sample unaided.
- ad hoc to paper Coverage principle: a feedback channel helps exactly as far as its instrument’s observational reach.
- ad hoc to paper Screenshot-fed VLM signal is classified as ungrounded because the model’s reading of a lossy view is what enters the loop.
- domain assumption pytest multi-assertion suites and DOM bounding boxes are faithful external ground truth for behavioral correctness and layout quality respectively.
invented entities (3)
-
internal ceiling (of reasoning and sampling)
independent evidence
-
two-sided grounding (feedback + evaluation)
independent evidence
-
coverage principle
independent evidence
Cite this review
Pith. "Pith review of Interaction Scaling: Grounding the Third Axis of Test-Time Compute." pith.science (2026). https://pith.science/paper/JZMXIHO5
@misc{pith2026260711598,
author = {Pith},
title = {Pith review of: Interaction Scaling: Grounding the Third Axis of Test-Time Compute},
year = {2026},
howpublished = {\url{https://pith.science/paper/JZMXIHO5}},
note = {Machine review of arXiv:2607.11598}
}
read the original abstract
There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one. Both share a hidden limit: they are internal. Every extra token comes from the same frozen weights and the same prompt, so neither can tell the model anything it does not already know. We study a third way, interaction: the model proposes an artifact, an external instrument observes how it actually behaves, and the model revises. Each cycle imports a real observation, so interaction breaks through the ceiling the other two hit. We argue that a single variable governs this third axis, grounding, and that it must hold on both sides of the loop. The feedback that drives revision must come from an instrument that actually observes the flaw, and so must the metric that scores the result. On hard coding tasks at a fixed token budget, reasoning-only and best-of-N sampling both plateau (the latter even when an oracle picks the best sample), while every interaction strategy keeps improving; our proposer-reviewer harness reaches a perfect 100% pass rate with no run-to-run variance, and the gain holds across three model families. On rendered visual artifacts, the usual judge (a vision-language model, or VLM, reading a screenshot) rates 14 of 15 visibly broken figures "perfect," because the screenshot hides the flaws before the judge can see them. A tool that measures the real layout instead shows the loop removing 40-74% of defects across four modalities; and that same VLM, used as the reviewer, makes slide layouts worse where the measuring tool repairs them. Interaction scaling is real and distinct from reasoning and sampling, but only visible when both the feedback and the metric are grounded.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Processing Systems , year =
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , year =
-
[2]
International Conference on Learning Representations , year =
Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. International Conference on Learning Representations , year =
-
[3]
Advances in Neural Information Processing Systems , year =
Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author =. Advances in Neural Information Processing Systems , year =
-
[4]
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =
-
[5]
Advances in Neural Information Processing Systems , year =
Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , year =
-
[6]
Advances in Neural Information Processing Systems , year =
Self-Refine: Iterative Refinement with Self-Feedback , author =. Advances in Neural Information Processing Systems , year =
-
[7]
NeurIPS Foundation Models for Decision Making Workshop , year =
Voyager: An Open-Ended Embodied Agent with Large Language Models , author =. NeurIPS Foundation Models for Decision Making Workshop , year =
-
[8]
Annual Meeting of the Association for Computational Linguistics (Short Papers) , year =
Teaching Small Language Models to Reason , author =. Annual Meeting of the Association for Computational Linguistics (Short Papers) , year =
Show all 37 references
-
[9]
Annual Meeting of the Association for Computational Linguistics , year =
Large Language Models Are Reasoning Teachers , author =. Annual Meeting of the Association for Computational Linguistics , year =
-
[10]
International Conference on Machine Learning , year =
Specializing Smaller Language Models towards Multi-Step Reasoning , author =. International Conference on Machine Learning , year =
-
[11]
Advances in Neural Information Processing Systems , year =
Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems , year =
-
[12]
and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E
Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E. , booktitle =. Gorilla: Large Language Model Connected with Massive
-
[13]
2021 , eprint =
Evaluating Large Language Models Trained on Code , author =. 2021 , eprint =
2021
-
[14]
2024 , eprint =
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling , author =. 2024 , eprint =
2024
-
[15]
Understanding the Effects of
Kirk, Robert and Mediratta, Ishita and Nalmpantis, Christoforos and Luketina, Jelena and Hambro, Eric and Grefenstette, Edward and Raileanu, Roberta , booktitle =. Understanding the Effects of
-
[16]
International Conference on Learning Representations , year =
Does Writing with Language Models Reduce Content Diversity? , author =. International Conference on Learning Representations , year =
-
[17]
Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , booktitle =. Scaling
- [18]
-
[19]
Kimi K1.5: Scaling Reinforcement Learning with
Kimi Team , year =. Kimi K1.5: Scaling Reinforcement Learning with. 2501.12599 , archivePrefix =
-
[20]
Advances in Neural Information Processing Systems , year =
Learning to Summarize with Human Feedback , author =. Advances in Neural Information Processing Systems , year =
-
[21]
2023 , eprint =
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback , author =. 2023 , eprint =
2023
-
[22]
2024 , howpublished =
Scaling Test-Time Compute with Open Models , author =. 2024 , howpublished =
2024
-
[23]
International Conference on Learning Representations , year =
Large Language Models Cannot Self-Correct Reasoning Yet , author =. International Conference on Learning Representations , year =
-
[24]
When Can
Kamoi, Ryo and Zhang, Yusen and Zhang, Nan and Han, Jiawei and Zhang, Rui , journal =. When Can
-
[25]
Gou, Zhibin and Shao, Zhihong and Gong, Yeyun and Shen, Yelong and Yang, Yujiu and Duan, Nan and Chen, Weizhu , booktitle =
- [26]
-
[27]
The Practice and Exploration of
Hong, Dingkun , year =. The Practice and Exploration of
-
[28]
2023 , eprint =
Improving Factuality and Reasoning in Language Models through Multiagent Debate , author =. 2023 , eprint =
2023
-
[29]
Advances in Neural Information Processing Systems , year =
Thinking vs.\ Doing: Agents that Reason by Scaling Test-Time Interaction , author =. Advances in Neural Information Processing Systems , year =
-
[30]
Gehring, Jonas and Zheng, Kunhao and Copet, Jade and Mella, Vegard and Cohen, Taco and Synnaeve, Gabriel , booktitle =
-
[31]
2006 , publisher =
Elements of Information Theory , author =. 2006 , publisher =
2006
-
[32]
2509.22644 , archivePrefix =
Lu, Zimu and Ren, Houxing and Yang, Yunqiao and Wang, Ke and Zong, Zhuofan and Pan, Junting and Zhan, Mingjie and Li, Hongsheng , year =. 2509.22644 , archivePrefix =
-
[33]
Iterative Agent Decoding for Detecting Compounding Errors in
Ruan, Yangjun and Briakou, Eleftheria and Han, Charles Xie and Chen, Yu and Jiao, Yufan and others , year =. Iterative Agent Decoding for Detecting Compounding Errors in. 2504.01931 , archivePrefix =
-
[34]
and Zhang, Hao and Gonzalez, Joseph E
Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging
-
[35]
2026 , howpublished =
Loopcraft: The Art of Stacking Loops , author =. 2026 , howpublished =
2026
-
[36]
2026 , howpublished =
The Art of Loop Engineering , author =. 2026 , howpublished =
2026
-
[37]
2026 , howpublished =
2026
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.