pith. sign in

arxiv: 2606.01182 · v1 · pith:7YEEZQN2new · submitted 2026-05-31 · 💻 cs.CL · cs.AI

CA-BED: Conversation-Aware Bayesian Experimental Design

Daniel Arnould , Rashad Aziz , Zixuan Kang , Tanav Changal , Kevin Zhu , Sunishchal Dev , Gabriel Grand , Shreyas Sunil Kulkarni This is my paper

Pith reviewed 2026-06-28 17:00 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords Bayesian experimental designdialog planninginformation gainLLM interactionentity deductionbelief updatingquestion selection
0
0 comments X

The pith

CA-BED uses Bayesian experimental design and LLM likelihoods to plan multi-turn questions by tracking belief distributions and expected information gain.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Conversation-Aware Bayesian Experimental Design to help large language models select better questions during interactive tasks where information must be gathered step by step. It maintains a probability distribution over possible hypotheses, uses the model itself to estimate how likely each possible user response is, and builds a tree of future conversation paths to choose the question that maximizes expected reduction in uncertainty. On two entity-deduction benchmarks the method raises success rates by an average of 21.8 percent relative to direct prompting while adding only about 1.8 extra turns on average. A sympathetic reader would care because many real-world uses of LLMs involve back-and-forth clarification rather than one-shot answers, and the framework supplies an explicit, inference-time mechanism for deciding what to ask next.

Core claim

CA-BED integrates Bayesian Experimental Design with LLM-based likelihood estimation to maintain a belief distribution over hypotheses, simulate possible answer sequences in a conversation tree, and select each next question by the expected information gain propagated across those future turns.

What carries the argument

A simulated conversation tree that propagates expected information gain by combining a maintained belief distribution over hypotheses with LLM-estimated probabilities of each possible user reply.

If this is right

  • Success rates rise on structured deduction tasks while the number of extra turns remains small.
  • The same planning loop can be applied at inference time without retraining the underlying language model.
  • Question selection improves over both direct prompting and other information-seeking baselines on the tested benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The tree-based planning could be extended to open-ended tasks such as medical history taking or technical troubleshooting where hypotheses are less discrete.
  • If the belief-update step is replaced by an external symbolic reasoner the method might become more robust to LLM calibration errors.
  • The computational cost of expanding the conversation tree at each turn sets a practical limit on how deep the lookahead can be in real deployments.

Load-bearing premise

LLM-based estimates of how likely a user would give each possible answer are accurate enough that the simulated tree reflects the uncertainty reduction that would actually occur with real humans.

What would settle it

Run the same entity-deduction tasks with real human respondents instead of simulated replies and measure whether the success-rate gains disappear or shrink substantially.

Figures

Figures reproduced from arXiv: 2606.01182 by Daniel Arnould, Gabriel Grand, Kevin Zhu, Rashad Aziz, Shreyas Sunil Kulkarni, Sunishchal Dev, Tanav Changal, Zixuan Kang.

Figure 1
Figure 1. Figure 1: Overview of CA-BED. The agent maintains a belief distribution over candidate hypothe [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative example of belief updating on an ambiguous exchange from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
read the original abstract

Large Language Models (LLMs) excel at static reasoning tasks, yet their performance often degrades in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in selecting questions that reduce uncertainty while incorporating responses that may be ambiguous or only partially informative. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialog planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED maintains a belief distribution over hypotheses, anticipates possible answers, and propagates expected information gain through a simulated conversation tree. Across two structured entity-deduction benchmarks, CA-BED yields an average 21.8% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It achieves these gains with an average increase of only 1.8 conversational turns compared to direct prompting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time framework that maintains a belief distribution over hypotheses, uses LLM-based estimation of P(answer | hypothesis, question), and selects questions by maximizing expected information gain propagated over a finite-depth simulated conversation tree. On two structured entity-deduction benchmarks the method reports an average 21.8% absolute improvement in success rate relative to direct prompting (and comparable gains versus other information-seeking baselines) while increasing average conversation length by only 1.8 turns.

Significance. If the reported gains prove robust to real human response distributions and the LLM likelihood model is shown to be sufficiently calibrated, CA-BED would constitute a concrete, reproducible advance in principled dialog planning for LLMs. The combination of Bayesian experimental design with LLM likelihoods is a natural and previously underexplored direction; the modest increase in turn count is also practically attractive.

major comments (3)
  1. [Abstract, §4] Abstract and §4 (experimental results): the central claim of a 21.8% success-rate improvement rests entirely on LLM-derived response likelihoods and simulated trees, yet the manuscript supplies no calibration data, no held-out human-response validation set, and no comparison of simulated versus live human trajectories. This directly undermines transferability of the reported gains.
  2. [§3.2] §3.2 (likelihood model and tree propagation): the expected-information-gain computation assumes that the LLM-estimated P(answer | hypothesis, question) accurately ranks questions under real conversational uncertainty; no ablation on likelihood calibration error or on tree depth is reported, leaving open the possibility that the observed ranking improvements are artifacts of the simulation.
  3. [§4] §4 (evaluation protocol): the abstract states empirical improvements but the provided text gives no information on the number of independent runs, statistical significance tests, variance across seeds, or exact baseline implementations, making it impossible to judge whether the 21.8% figure is load-bearing or reproducible.
minor comments (2)
  1. [§3] Notation for the belief distribution and the information-gain objective should be introduced with explicit equations rather than prose descriptions.
  2. [§4] The two entity-deduction benchmarks are referenced only by name; a short description of their hypothesis spaces and answer formats would improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate whether revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract, §4] Abstract and §4 (experimental results): the central claim of a 21.8% success-rate improvement rests entirely on LLM-derived response likelihoods and simulated trees, yet the manuscript supplies no calibration data, no held-out human-response validation set, and no comparison of simulated versus live human trajectories. This directly undermines transferability of the reported gains.

    Authors: We agree that the evaluation relies exclusively on LLM-simulated responses and does not include human calibration or live trajectories. This design choice prioritizes reproducibility and controlled comparison on the structured benchmarks. In the revised manuscript we have expanded the limitations discussion in §4 and the conclusion to explicitly address the assumptions of the simulation and the implications for transfer to human interactions. No human validation experiments are added, as they are outside the scope of the current work. revision: partial

  2. Referee: [§3.2] §3.2 (likelihood model and tree propagation): the expected-information-gain computation assumes that the LLM-estimated P(answer | hypothesis, question) accurately ranks questions under real conversational uncertainty; no ablation on likelihood calibration error or on tree depth is reported, leaving open the possibility that the observed ranking improvements are artifacts of the simulation.

    Authors: We have added an ablation study on tree depth in the revised §4, confirming that performance plateaus at the depth used in the main experiments. For likelihood calibration, we included a sensitivity analysis demonstrating that the question-ranking decisions remain stable under moderate perturbations to the estimated probabilities, because the information-gain objective depends primarily on relative ordering rather than absolute values. revision: yes

  3. Referee: [§4] §4 (evaluation protocol): the abstract states empirical improvements but the provided text gives no information on the number of independent runs, statistical significance tests, variance across seeds, or exact baseline implementations, making it impossible to judge whether the 21.8% figure is load-bearing or reproducible.

    Authors: We have revised §4 to report that all results are averaged over five independent runs with different random seeds, include standard deviations, apply paired t-tests for significance (p < 0.05 on the primary comparisons), and provide complete hyperparameter settings and implementation details for every baseline. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical method and benchmark results

full rationale

The paper introduces CA-BED as an inference-time framework combining Bayesian experimental design with LLM likelihoods and reports measured success-rate gains (21.8% average) on two entity-deduction benchmarks. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text; the central claim is an observed performance delta under the stated simulation, not a quantity forced by construction from its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are described in sufficient detail to enumerate.

pith-pipeline@v0.9.1-grok · 5704 in / 1030 out tokens · 29074 ms · 2026-06-28T17:00:53.120489+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 5 canonical work pages

  1. [1]

    Warm up cold-start advertisements: Improving ctr predictions via learning to learn id embeddings

    doi: 10.1145/3331184.3331265. URLhttp: //dx.doi.org/10.1145/3331184.3331265. Chinmaya Andukuri, Jan-Philipp Fr ¨anken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions,

    work page doi:10.1145/3331184.3331265

  2. [2]

    Donald A

    URLhttps://arxiv.org/ abs/2403.19154. Donald A. Berry. Bayesian clinical trials.Nature Reviews Drug Discovery, 5(1):27–36, January

    arXiv

  3. [3]

    doi: 10.1038/nrd1927

    ISSN 1474-1784. doi: 10.1038/nrd1927. URLhttp://dx.doi.org/10.1038/ nrd1927. Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. ChatGPT’s in- formation seeking strategy: Insights from the 20-questions game. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.),Proceedings of the 16th International Natural Language Genera- tion...

    work page doi:10.1038/nrd1927

  4. [4]

    Agentic AI in the Wild

    Association for Computational Linguistics. doi: 10.18653/v1/2023.inlg-main.11. URLhttps://aclanthology.org/ 2023.inlg-main.11. 9 Published as a workshop paper at the workshop of “Agentic AI in the Wild” in ICLR 2026 Kwan Ho Ryan Chan, Yuyan Ge, Edgar Dobriban, Hamed Hassani, and Ren ´e Vidal. Conformal information pursuit for interactively guiding large l...

    work page doi:10.18653/v1/2023.inlg-main.11 2023

  5. [5]

    Harshita Chopra and Chirag Shah

    URLhttps:// arxiv.org/abs/2507.03279. Harshita Chopra and Chirag Shah. Feedback-aware monte carlo tree search for efficient informa- tion seeking in goal-oriented conversations,

    arXiv

  6. [6]

    Michael Cooper, Rohan Wadhawan, John Michael Giorgi, Chenhao Tan, and Davis Liang

    URLhttps://arxiv.org/abs/ 2508.21184. Michael Cooper, Rohan Wadhawan, John Michael Giorgi, Chenhao Tan, and Davis Liang. The curious language model: Strategic test-time information acquisition,

    Pith/arXiv arXiv

  7. [7]

    Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua

    URLhttps:// arxiv.org/abs/2506.09173. Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration,

    arXiv

  8. [8]

    URLhttps://arxiv.org/abs/2305.13626. J. Denzler and C.M. Brown. Information theoretic sensor data selection for active object recognition and state estimation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(2): 145–157,

    arXiv

  9. [9]

    Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio

    doi: 10.1109/34.982896. Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. Clarq-llm: A benchmark for models clarifying and requesting information in task-oriented dialog,

    work page doi:10.1109/34.982896

  10. [10]

    Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Be- linda Z

    URLhttps://arxiv.org/abs/2409.06097. Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Be- linda Z. Li. Bayesian preference elicitation with language models,

    arXiv

  11. [11]

    Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi

    URLhttps: //arxiv.org/abs/2403.05534. Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models,

    arXiv

  12. [12]

    Yichen Huang and Lin F

    URLhttps://arxiv.org/abs/ 2402.03271. Yichen Huang and Lin F. Yang. Winning gold at imo 2025 with a model-agnostic verification-and- refinement pipeline,

    arXiv 2025

  13. [13]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim

    URLhttps://arxiv.org/abs/2507.15855. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation,

    arXiv

  14. [14]

    Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S

    URLhttps://arxiv.org/abs/2406.00515. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning,

    Pith/arXiv arXiv

  15. [15]

    Shuyue Stella Li, Jimin Mun, Faeze Brahman, Pedram Hosseini, Bryceton G

    URLhttps://arxiv.org/abs/2406.00922. Shuyue Stella Li, Jimin Mun, Faeze Brahman, Pedram Hosseini, Bryceton G. Thomas, Jessica M. Sin, Bing Ren, Jonathan S. Ilgen, Yulia Tsvetkov, and Maarten Sap. Alfa: Aligning llms to ask good questions a case study in clinical reasoning,

    arXiv

  16. [16]

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi

    URLhttps://arxiv.org/abs/ 2502.14860. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning,

    arXiv

  17. [17]

    Yang Nan, Qihao Wen, Jiahao Wang, Pengfei He, Ravi Tandon, Yong Ge, and Han Xu

    URLhttps://arxiv.org/abs/2502.01100. Yang Nan, Qihao Wen, Jiahao Wang, Pengfei He, Ravi Tandon, Yong Ge, and Han Xu. Interpretable probability estimation with llms via shapley reconstruction,

    arXiv

  18. [18]

    Agentic AI in the Wild

    URLhttps://arxiv. org/abs/2601.09151. 10 Published as a workshop paper at the workshop of “Agentic AI in the Wild” in ICLR 2026 Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agen- tic evaluation of large language models....

    arXiv 2026

  19. [19]

    Irina Rish

    URLhttps://arxiv.org/abs/2302.14545. Irina Rish. An empirical study of the na ¨ıve bayes classifier.IJCAI 2001 Work Empir Methods Artif Intell, 3, 01

    arXiv 2001

  20. [20]

    Jimmy Wang, Thomas Zollo, Richard Zemel, and Hongseok Namkoong

    URLhttps://arxiv.org/abs/2505.12543. Jimmy Wang, Thomas Zollo, Richard Zemel, and Hongseok Namkoong. Adaptive elicitation of latent information using natural language, 2025a. URLhttps://arxiv.org/abs/2504. 04204. Peng-Yuan Wang, Tian-Shuo Liu, Chenyang Wang, Yi-Di Wang, Shu Yan, Cheng-Xing Jia, Xu-Hui Liu, Xin-Wei Chen, Jia-Cheng Xu, Ziniu Li, and Yang Yu...

    arXiv

  21. [21]

    Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025a

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025a. URL h...

    Pith/arXiv arXiv

  22. [22]

    Michael J

    URLhttps://arxiv.org/abs/2305.13660. Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi. Modeling future conversation turns to teach llms to ask clarifying questions,

    arXiv

  23. [23]

    doi: 10.18653/v1/2024.acl-long.82

    Association for Computational Lin- guistics. doi: 10.18653/v1/2024.acl-long.82. URLhttps://aclanthology.org/2024. acl-long.82/. Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information?,

    work page doi:10.18653/v1/2024.acl-long.82 2024

  24. [24]

    Agentic AI in the Wild

    URLhttps://arxiv.org/abs/2506.08295. A QUESTIONERMODELABLATION To assess whether the gains reported in the main paper depend on the specific Questioner model, we repeated the20 Questionsevaluation withGPT-5.4-nanoandGemini-3.1-flash-liteas alternative Questioners. We report these runs in the appendix, rather than the main tables, because these models 11 P...

    arXiv 2026