{"id":"f2556b46-9815-4b62-a2dc-e2ace3fdbb0a","arxiv_id":"2506.23678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Interactive Reasoning, instantiated as Hippo, lets users view and edit an LLM's chain-of-thought as a tree, and a 16-person study reports improved perceived control, sense-making, and assumption awareness.","lead":"This paper introduces Hippo, a system that turns a language model's chain-of-thought reasoning into an editable tree, letting users add, trim, or revise topics before the model composes its final answer. A 16-participant study found that people reported more control, sense-making, and awareness of assumptions with Hippo than with an editable text baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The steering claim rests on an untested causal assumption: edited CoT text inside <think> tags may not actually change the final output, and §8 concedes no systematic test was run.","rationale":"The paper is a solid HCI contribution with a well-motivated interface and statistically significant self-report results. However, its headline claim of 'steering' requires that edited CoT text causally influences the final response. The evidence for this is anecdotal and self-report only; the study never measures whether the final output actually changes in the intended direction. The authors' own Limitations section (§8) concedes the lack of systematic analysis and even disclaims a causal claim, creating a contradiction with the Abstract. This is the single most load-bearing issue because the interaction design is specifically about control and steering; if the CoT edit is not causally effective, users' heightened sense of control and personalization is a placebo effect, and the rationale for revealing and editing CoT collapses. The proposed ablation test would settle this directly. I agree with the reader's weakest_assumption and with the CONDITIONAL verdict: the paper is not fatally flawed (perceived sense-making and awareness are still valuable; the design could still support oversight even without perfect steering), but the steering claim must be validated or substantially weakened. No adjustment to the reader's verdict is needed.","tokens_in":24164,"tokens_out":4674,"duration_ms":47310,"concrete_test":"Run a controlled ablation: select N=50 DailyDilemma-style queries; for each, generate the initial DeepSeek-R1 CoT and response. Apply three scripted edits to the CoT: (a) delete a specific topic subtree, (b) insert a concrete user preference via a Feedback node, (c) make a length-matched control edit that does not change meaning. Regenerate responses via the §4.2.3 <think> pipeline (same model, same temperature). Have two independent, condition-blind raters judge whether the response reflects the edit: deleted topic absent, inserted preference present. Pre-register a threshold (e.g., ≥80% agreement with intended direction and ≥30% above control). If editing does not steer output relative to control, the steering claim is unsupported and the perceived-control findings are likely placebo.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that users can \"efficiently steer the model towards customized responses\" (Abstract) depends on the assumption that feeding edited chain-of-thought text back inside <think> tags (§4.2.3) causes the final output to move in the direction of the user's edits. The user study in §5 provides only self-report Likert measures (control p=0.003, sense-making p=0.004, layout p=0.009, awareness p=0.012) and qualitative anecdotes (P8, P10); it contains no objective measure of whether responses actually changed in line with edits. The authors themselves acknowledge in §8: \"our current approach does not systematically analyze the model behaviors given users' feedback\" and disclaim claiming that feedback led to more personalized outputs—yet the abstract and contribution list assert efficient steering. This is an internal tension. If, as [10] shows, visible CoT text does not reliably condition a model's reasoning, then perceived control may be illusory, and the system's primary value proposition collapses. The concern is load-bearing because the design's purpose is control, not just display.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Interactive Reasoning, an interaction design that converts an LLM's chain-of-thought into a hierarchical tree of topics, allows users to add, edit, delete, and regenerate nodes, and generates a final answer conditioned on the edited thinking. This is implemented in Hippo, a prototype for decision-making under trade-offs. The authors report a within-subjects user study (N=16) comparing Hippo to an editable linear-text baseline, finding significant self-reported improvements in control, sense-making, layout, awareness of assumptions, and confidence, plus qualitative evidence of personalization and repurposing of the final response. Two follow-up case studies illustrate use for information seeking and financial planning. The paper claims that interactive reasoning lets users 'efficiently steer the model towards customized responses' and proposes a new paradigm for human oversight during test-time scaling.","tokens_in":24395,"tokens_out":4236,"duration_ms":50325,"significance":"If the steering claim holds, this is a timely and useful contribution to human-AI interaction: it directly addresses a real usability problem with long, unstructured chain-of-thought output and offers a concrete design space for test-time scaling control. The work's strengths include a thoughtful derivation of design goals from prior HCI/XAI literature, a fair baseline that is also editable, rich qualitative analysis, and unusually candid limitations. I see no circularity in the empirical evaluation, which is an external user study. However, the paper's most attention-grabbing claim, that edited chain-of-thought text inside <think> tags steers the final output, is not tested objectively, and the headline statistical results are based on a small number of unadjusted Likert comparisons. These issues are load-bearing because the system is framed as a control mechanism rather than a visualization only. The paper is worth publishing after the causal and statistical concerns are addressed.","major_comments":[{"comment":"The abstract states that Hippo lets users 'efficiently steer the model towards customized responses', and contribution (2) asserts that users reported 'increased confidence in making the final decision' with Hippo. Yet §8 explicitly says that 'our current approach does not systematically analyze the model behaviors given users’ feedback' and that the paper 'does not claim that user feedback led to more personalized or accurate reasoning chains or final model outputs'. These statements are in tension. The mechanism in §4.2.3, which places edited reasoning inside <think>...</think> tags, is a causal assumption: the final response should change in the direction of the user's edits. The user study provides only self-reported Likert scales and qualitative anecdotes (e.g., P8, P10), with no objective measure of whether the final output actually changed in the intended direction. This is not merely a presentation issue; the central value proposition of Hippo is control and steering, not just display. I recommend either softening the abstract and contribution claims to describe perceived control, or adding a systematic evaluation of edit-to-output alignment (e.g., a small controlled experiment with blind raters or automated entailment/similarity metrics comparing responses with and without a given edit).","section":"Abstract; §4.2.3; §8"},{"comment":"The headline quantitative findings rest on nine Likert items tested with Wilcoxon signed-rank tests without any correction for multiple comparisons. With a conservative Bonferroni threshold of α=0.0056 for nine tests, control (p=0.003) and sense-making (p=0.004) remain significant, but layout (p=0.009), awareness (p=0.012), and confidence (p=0.049) would not survive. At minimum, the authors should report effect sizes and confidence intervals for each comparison, use a mixed-effects model or otherwise account for the multiple outcomes, and temper the wording in the abstract and Section 5.3 that presents all five results as equally robust. The lack of correction is especially relevant because the measures are correlated (e.g., layout and sense-making) and the sample is only N=16.","section":"§5.3.1; §5.3.2"},{"comment":"RQ3 concerns how users interact with the reasoning tree, but the paper provides no quantitative interaction logs. Statements such as 'we observed that participants rarely made edits', 'only four users typed their own experiences and opinions', and 'zero participants used the collapse function during tree generation' are based on informal observation. Since the system was implemented with a logging backend (Next.js/Flask), the authors can easily report counts of add/edit/delete/regenerate operations, time-to-first-edit, and usage of the collapse feature. Such logs would also help distinguish perceptual effects in RQ1–RQ2 from actual behavioral changes, and would strengthen the design implications in Section 7.","section":"§5.3.3; §5.1"}],"minor_comments":[{"comment":"The phrase 'long reasonig chain' contains a typo; it should read 'long reasoning chain'.","section":"§2.2"},{"comment":"The section heading 'A wareness' should be 'Awareness'.","section":"§5.3.1"},{"comment":"The figure caption says 'Review and edit the reaching chain below!' but should say 'reasoning chain'.","section":"Figure 4 caption"},{"comment":"The literal placeholder '[More of this example]' appears inside the few-shot prompt; this should be removed or clearly marked as an abridged prompt rather than part of the actual prompt text.","section":"Appendix B.1.1"},{"comment":"The duplicate-clarification threshold (cosine similarity > 0.8 with all-MiniLM-L6-v2) and the group-thoughts paragraph limit (8 paragraphs) are introduced without any sensitivity analysis or justification; a brief ablation or note on their influence would improve reproducibility.","section":"§4.2.2; Appendix B.1.2"},{"comment":"The power analysis is reported as Cohen's d = 0.8, α = 0.05, N = 16, but the assumed effect size and the paired versus independent design are not justified; please specify the target power and whether the test is two-sided.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid HCI contribution but currently oversells the steering effect. The authors are candid in the limitations, and the gap between the abstract and §8 is fixable with a modest additional analysis or careful rewording. The multiple-comparisons issue affects three of the five headline results and should be addressed before acceptance. No concerns about circularity or authorship."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a competent, honest HCI paper about a timely idea, but the headline claim that users can “efficiently steer” the model is not supported by the evidence. The study shows users feel more in control, not that they actually are.\n\nWhat’s new: applying node-link diagrams to intermediate chain-of-thought during test-time scaling, with editable nodes, a clarify step, and response-to-node linking. That’s a sensible extension of Graphologue/Sensecape-style sensemaking into the CoT regime. The baseline is fair—editable linear text rather than a non-editable control—and the within-subjects design is standard. The qualitative analysis is the paper’s real strength: the tension between full transparency and interaction fatigue, the depth-first vs. breadth-first discussion, and the observation that users sometimes treat the reasoning itself as the output. The authors also deserve credit for acknowledging in §8 that they did not systematically test how edits affect model behavior.\n\nThe soft spots: the internal tension between the abstract/contributions and §8 is real. The abstract says users can “efficiently steer the model towards customized responses,” but the study only measures perceived control, and §8 says the paper does not claim feedback led to more personalized outputs. That mismatch needs fixing—either soften the claims or add an objective test of whether edits move the output in the intended direction. The anecdotes (P8, C2) suggest the mechanism sometimes works, and feeding edited text into <think> tags is a plausible conditioning mechanism, so the stress-test’s “collapses” framing is too strong; the value proposition is simply not yet demonstrated. Also, with nine Likert comparisons and no correction, only control (p=0.003) and sense-making (p=0.004) survive a strict Bonferroni; layout and awareness at p=0.009/0.012 would not. The confidence result at p=0.049 is fragile. The sample of 16 self-selected, LLM-experienced users is typical for CHI but limits generalization. Minor: the power analysis calls d=0.8 a “medium” effect, though that’s conventionally large.\n\nWho it’s for: HCI researchers working on LLM interaction and steering, and NLP folks interested in human-in-the-loop test-time scaling. It deserves a serious referee—the idea is timely and the study is competently run—but it needs revision to align claims with evidence.\n\nRecommendation: send to peer review with a clear request for major revision, specifically to either weaken the steering language or add a small objective evaluation of edit-to-output alignment.","headline":"Timely HCI prototype for editing chain-of-thought, but the steering claim outruns the self-report evidence—worth refereeing nonetheless.","tokens_in":24923,"tokens_out":3425,"would_cite":true,"duration_ms":41734,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that turning a model's chain-of-thought into an interactive topic tree gives users a sense of control over, and understanding of, the reasoning behind an LLM's answer.","keywords":["interactive reasoning","chain-of-thought","LLM","tree visualization","human-AI interaction","test-time scaling","user study","AI decision-making"],"falsifier":"A controlled experiment where users insert a node asserting a clearly false or contradictory premise (for example, 'the user has no budget constraints' when the query is about budget travel) and the final response is compared against a baseline run without that edit; if the response does not systematically and reliably incorporate the edit, the steering mechanism fails. Repeating across multiple users and edits would show whether the effect is real or merely perceived.","tokens_in":23994,"feed_emoji":"🌳","tokens_out":3821,"duration_ms":39932,"temperature":0.7,"pith_summary":"The paper introduces Interactive Reasoning, an interaction design that turns a large language model's verbose, linear chain-of-thought into a hierarchical tree of topics that users can directly edit. It implements this in Hippo, a prototype for AI-assisted decision-making under trade-offs, and reports a within-subjects study of 16 participants. The study finds that, compared to a baseline interface that also allows editing but in plain text, Hippo significantly increases users' perceived control, sense-making, layout comprehension, and awareness of assumptions in the reasoning. The promise is that users can catch and interrupt wrong turns, add their own context and priorities, and receive a final answer that feels personalized rather than generic. The paper frames this as a new paradigm: putting human oversight inside the reasoning step, not just at the final answer.","feed_headline":"LLM chain-of-thought becomes an editable tree users can steer","feed_subtitle":"16-person study: users felt more control, sense-making, and awareness of assumptions than with plain editable text.","key_machinery":"The central object is the interactive reasoning tree, a depth-first progressively generated topic hierarchy created by a few-shot prompt that structures raw chain-of-thought text into <topic> and <branch> XML-like tags. A Clarify operator flags feedback nodes where user input would be valuable, and a Link operator uses zero-shot natural-language inference to highlight which reasoning nodes support which sentences of the final response. User edits (add, edit, delete, regenerate, or branch with a custom prompt) are collected, and the updated reasoning is placed back inside <think>...</think> tags to condition the model's final generation.","core_discovery":"On its own terms, the central claim is that representing a model's chain-of-thought as an interactive topic tree, rather than as a linear text wall, lets users feel they are steering the model, and this sense of agency extends to the final response. In the user study, perceived control rose from a mean of 4.19 to 5.75 (p = 0.003), sense-making from 5.19 to 6.44 (p = 0.004), layout comprehensibility from 3.44 to 6.00 (p = 0.009), and awareness of assumptions from 4.69 to 6.25 (p = 0.012). Participants described outputs as personalized, saying the model 'hears my voice rather than generating a generic answer,' and a case-study user called the tree 'essentially a plan for me to consider' in financial analysis. The paper is careful not to claim that editing the chain improves the factual correctness of outputs; it claims an improved sense of control, understanding, and personalization from the end user's perspective.","pith_inferences":["Editorial inference: This study measures perceived control and understanding, not whether edited chain-of-thought text actually changes the model's final output; the paper itself cites evidence that chain-of-thought may not faithfully reflect a model's real reasoning.","Editorial inference: The feedback-node mechanism may act as a cognitive forcing function, so part of the measured awareness gain could come from being asked questions rather than from the tree layout itself.","Editorial inference: A testable extension would insert a deliberately contradictory or false premise into a node and measure whether the final response systematically reflects that edit across repeated runs, which would separate real steering from perceived steering."],"forward_implications":["Reasoning-model interfaces can be redesigned around review-and-edit, letting users interrupt wrong assumptions before an answer is finalized.","In high-stakes decisions, the reasoning chain itself becomes a plan the user can refine, while the final response becomes a personalized summary rather than the main deliverable.","Tree visualization may reduce cognitive load enough that users actually read reasoning they would otherwise skim or ignore.","The Link highlight feature offers traceability from each response sentence back to the reasoning step that produced it, which can support appropriate trust calibration."],"supporting_citations":[{"why":"Supplies the DailyDilemma dataset used for the study's high-stakes decision-making tasks.","marker":"[11]"},{"why":"Provides DeepSeek-R1, the open-weight reasoning model whose chain-of-thought is visualized and edited in Hippo.","marker":"[16]"},{"why":"Cited in the Limitations section to acknowledge that chain-of-thought may not faithfully represent a model's actual reasoning, bounding the paper's steering claim.","marker":"[10]"},{"why":"Graphologue demonstrates node-link diagrams for LLM sensemaking, the approach Hippo extends to reasoning chains rather than final outputs.","marker":"[25]"},{"why":"Cognitive forcing functions motivate the Clarify feedback nodes that interrupt users during reasoning review.","marker":"[8]"},{"why":"Concept Induction motivates the Structure operator that aggregates raw reasoning text into high-level themes before topic tagging.","marker":"[37]"},{"why":"Establishes chain-of-thought prompting as a way to condition model outputs, the conceptual basis for feeding edited reasoning back inside <think> tags.","marker":"[80]"},{"why":"Supports the zero-shot prompting approach used by the Link operator to match response sentences to reasoning nodes.","marker":"[36]"}],"fun_headline_variants":["Interactive tree view lets users steer LLM chain-of-thought","Editable topic tree of LLM thoughts increases user control","LLM reasoning as an editable tree boosts user agency","Visualizing LLM reasoning as an editable tree improves oversight","Tree-shaped chain-of-thought editing gives users more agency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that editing the visible chain-of-thought text and feeding it back inside <think> tags actually steers the model's final answer in the direction the user intends; if the written chain is not a faithful or effective control handle, the claimed benefits are illusory even if users feel more confident.","fun_headline_variants_meta":{"raw":{"variants":["Interactive tree view lets users steer LLM chain-of-thought","Editable topic tree of LLM thoughts increases user control","LLM reasoning as an editable tree boosts user agency","Visualizing LLM reasoning as an editable tree improves oversight","Tree-shaped chain-of-thought editing gives users more agency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3420,"prompt_tokens":942,"completion_tokens":2478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2397}},"tokens_in":558,"tokens_out":2478,"duration_ms":17008,"temperature":1.0,"reasoning_tokens":2397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:34:01.251570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment where users insert a node asserting a clearly false or contradictory premise (for example, 'the user has no budget constraints' when the query is about budget travel) and the final response is compared against a baseline run without that edit; if the response does not systematically and reliably incorporate the edit, the steering mechanism fails. Repeating across multiple users and edits would show whether the effect is real or merely perceived.","supporting_citations":[],"review_version":1}