Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

This paper seeks to establish that uncertainty can act as a control signal for LLM agents, improving factual summarization over biomedical multi-table databases and enabling abstention when confidence is low.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An LLM agent using retrieval and summary uncertainty as training rewards and inference filters produces more factual, useful multi-omics summaries and better downstream survival predictions.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful integration of uncertainty control for table agents, but the headline factuality numbers are measured by the same judge family used in training—so the central claim is not yet independently verified. the 4 major comments →

arxiv 2509.02401 v1 pith:CT6T5OCB submitted 2025-09-02 cs.AI

Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning

Josefa Lia Stoisser , Marc Boubnovski Martell , Lawrence Phillips , Gianluca Mazzoni , Lea M{\o}rch Harder , Philip Torr , Jesper Ferkinghoff-Borg , Kaspar Martens
show 1 more author
Julien Fauqueur
This is my paper
classification cs.AI
keywords uncertainty quantificationLLM agentmulti-table summarizationreinforcement learningGRPOcalibrationabstentionmulti-omics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model agents that summarize biomedical multi-omics databases often produce fluent but overconfident claims. This paper argues that uncertainty should be an active control signal, not a post-hoc diagnostic: retrieval uncertainty (entropy over table-selection rollouts) and summary uncertainty (self-consistency combined with perplexity) are fed into reinforcement learning rewards and into inference-time filtering and abstention. On two multi-omics benchmarks, the resulting agent nearly triples the number of correct and useful claims per summary (3.0 to 8.4 internal; 3.6 to 9.9 cancer), improves calibration as measured by prediction rejection ratio, and lifts downstream survival prediction concordance from 0.32 to 0.63. A reader should care because this offers a concrete path toward agents that know when not to answer, which matters in high-stakes biomedical settings.

Core claim

The central claim is that uncertainty is a first-class control signal for an LLM agent operating over structured multi-table data, not just a diagnostic to report. The agent runs multiple rollout episodes per query; retrieval uncertainty is the normalized binary entropy of table-selection frequencies across episodes, and summary uncertainty is the CoCoA score, a perplexity-weighted semantic self-consistency measure. During GRPO training, a reward component based on inverse perplexity steers the policy toward confident summaries while a judge-based reward encourages exploration. At inference, the agent abstains when the combined uncertainty exceeds a threshold and otherwise returns the lowest

What carries the argument

The carrying mechanism is 'uncertainty as control,' built from two complementary signals. Retrieval uncertainty is the normalized binary entropy, averaged over tables, of how often each table is selected across K rollout episodes, so it rises when evidence acquisition is unstable. Summary uncertainty, CoCoA, multiplies the perplexity of the candidate summary by one minus the mean semantic similarity to other sampled summaries, flagging confident but inconsistent outputs. During training, perplexity alone serves as a cheap reward proxy within GRPO; at inference, the two signals set an abstention threshold and select the lowest-perplexity candidate. The same signals also filter high-uncertaint

Load-bearing premise

The factual-quality numbers assume that the LLM judge used to split summaries into claims and check them against the database is a reliable proxy for ground truth; only 40 human queries were used to validate this judge.

What would settle it

Have human domain experts fact-check every claim in the best configuration (adaptive reward plus filtering) and in the baseline SQL agent on both datasets. If the human-verified count of correct, useful claims no longer shows roughly a threefold gap (about 8.4–9.9 versus 3.0–3.6), the paper's central claim fails. A sharper check is to replace the scoring judge with an independently implemented fact-checker; a large drop would indicate judge-specific reward hacking.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Agents can be trained to abstain: applying the uncertainty threshold raises the correctness ratio of emitted summaries (0.82 to 0.94 on cancer; 0.84 to 0.90 internal).
  • Uncertainty estimates become meaningful control signals: the prediction rejection ratio roughly doubles, reaching 0.45–0.47 for CoCoA, meaning confidence tracks factual reliability.
  • Uncertainty-aware training transfers to downstream tasks: survival prediction C-index improves from 0.32 before training to 0.55–0.64 depending on reward schedule, above the 0.22 baseline.
  • Reward scheduling matters: an adaptive exploitation schedule avoids early training collapse and yields the best useful-claims ratio, while omitting uncertainty rewards produces mostly hallucinated claims (correct ratio 0.27).
  • Filtering high-uncertainty summaries also improves the quality of synthetic corpora, making uncertainty a data-curation tool rather than only an evaluation metric.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The factuality gains are measured by an LLM judge that is also used in the training reward, with only 40 human queries validating that judge; a reader should treat the headline numbers as contingent on judge–human agreement until a larger human-annotated claim set confirms them.
  • Because the same judge family participates in both reward and evaluation, part of the improvement could reflect optimizing the judge's preferences rather than database-grounded truth; re-scoring with a judge from a different family or with rule-based verification would test this.
  • The abstention mechanism implies a coverage–precision tradeoff, and thresholds tuned on a validation split may not transfer across database schemas, so schema-specific recalibration is likely needed in practice.
  • The framework's cost is dominated by multiple rollouts and CoCoA's semantic-similarity computation; the paper's own ablations suggest lighter signals such as perplexity or entropy retain most of the benefit, pointing to cheaper deployment variants.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an uncertainty-aware LLM agent for query-conditioned multi-table summarization in biomedical multi-omics databases. The method combines retrieval uncertainty (entropy over table-selection rollouts) and summary uncertainty (perplexity plus self-consistency, using the CoCoA framework). Summary uncertainty is used as a reward signal in GRPO training, while both retrieval and summary uncertainty are used at inference for filtering/abstention. Experiments on an internal proprietary dataset and the public MLOmics benchmark report large improvements in correct/useful claims per summary, improved PRR calibration, and improved downstream survival prediction C-index. The central claim is that uncertainty can serve as a control signal for LLM agents, improving factuality, calibration, and downstream utility.

Significance. If the results hold, the paper makes a useful contribution: it integrates uncertainty into both training and inference for table-reasoning agents and demonstrates gains on a realistic biomedical task. The framework is modular and the reward-schedule ablations are informative. However, the headline factuality claims are measured by an LLM judge from the same model family used as the training reward, with only a 40-query human validation, and the survival experiment is under-specified. These issues currently prevent the central claim from being independently verified, so the significance is conditional on additional validation.

major comments (4)
  1. [§4.3 vs §3.4/Appendix A] The central factuality numbers (Q2, Q3) are computed by an 'o4 mini' judge that decomposes summaries into claims and validates them against the database. The same o4-mini model family is used as the RJudge component of the GRPO reward (Appendix A). Thus the policy is trained against the same judge family used for the headline evaluation. The only human check is a 40-query holdout (Appendix E.3, Table A7) reporting r=0.64 and preserved rankings. Rank preservation does not rule out systematically inflated absolute claim counts, and 40 queries is too small to estimate claim-level precision/recall, especially for the proprietary 2,000-table internal schema where the five task-specific workflows are not released. Please provide (i) claim-level human agreement with the judge, (ii) evaluation using an independent judge family not used in training, and (iii) release of the validation workflows o
  2. [§4.7, Table 3] The survival prediction experiment is the only independent outcome measure, but it is under-specified. The text says the agent is prompted to estimate survival times 'leveraging in-context knowledge from summaries,' but no prompts, inputs, held-out patient count, or confidence intervals are given. Table 3 reports C-index values without error bars. The abstract highlights C-index 0.32→0.63, but the reader cannot tell how much of this is due to the summarization improvement versus the survival-prediction prompting protocol, nor whether the differences are statistically meaningful. Please specify the protocol, report N and uncertainty intervals, and describe how summaries are converted into survival-time predictions.
  3. [§3.3, §3.4, §4.5] There is a partial circularity in the calibration claim. During training, Rconf directly rewards low perplexity. At inference and evaluation, uCoCoA is defined as perplexity multiplied by semantic consistency (Appendix A). Therefore the reported decreases in uCoCoA and the reported PRR improvements partly reflect direct optimization of the perplexity component of the measured uncertainty. To show that the uncertainty signal is meaningful beyond this mechanical effect, please report PRR for consistency-only uncertainty (without the perplexity term) or otherwise demonstrate calibration improvements on uncertainty quantities not used as training rewards.
  4. [Tables 1–2, §4.5] The inference-time filtering improvement is a key part of the 'uncertainty as control' claim, but the main tables do not report the abstention rate or coverage for the specific filtering used. Table A8 gives coverage only for the internal dataset at three κ values, while Tables 1–2 report results 'after filtering' without stating which κ was used or how many summaries were abstained. Without coverage information, higher precision after filtering is expected by construction. Please report coverage/abstention rates alongside the filtered results in the main tables.
minor comments (4)
  1. [Abstract and §1] Typos and inconsistencies: 'ulti-omics' in the abstract; 'just less than tripling' vs 'nearly tripling' used inconsistently; in the Introduction, 'while demonstrate multi-agent orchestration' is missing a subject; 'advatage' in §3.5; 'reliabilty' in §3.6.
  2. [§4.2 / Appendix D] The statement that datasets are split '70:30 based on patient samples' is ambiguous for a task-level train/test split of 80/20 tasks. Clarify whether the 70:30 patient split was applied before defining tasks, and whether any patient overlap occurs between train and test tasks.
  3. [Appendix E.2, Table A6] The 'Entropy' and 'Retrieval variance' reward signals are not defined precisely. It is unclear what entropy is computed over when used as a training reward signal. Define these in the text.
  4. [Appendix E.3, Table A7] The judge robustness table reports only correlation coefficients. Reporting calibration (e.g., precision/recall of claim correctness) and a larger human sample would substantially strengthen the argument.

Circularity Check

2 steps flagged

Headline factuality gains are measured by the same o4-mini judge family used as the training reward, and the reported uncertainty reduction partly re-measures the perplexity term that was explicitly optimized.

specific steps
  1. fitted input called prediction [Section 3.4 / Appendix A (Exploration Judge Reward); Section 4.3 (Summary Quality); Tables 1-2]
    "An external o4-mini LLM counts the number c(τ ) of grounded, non-overlapping atomic facts in the trajectory τ that are relevant to the user's topic. The reward is: RJudge(τ ) = min(c(τ )/20, 1) ... an o4 mini judge decomposes s into atomic claims, validates them against D using a set of five task-specific workflows ..., and assigns correctness and utility labels to each claim."

    The GRPO terminal reward explicitly contains RJudge(τ), an o4-mini count of grounded atomic facts in the trajectory. The headline evaluation (Q1-Q3) is produced by 'an o4 mini judge' that decomposes the final summary into atomic claims and validates correctness/usefulness against the database. The policy is therefore trained to maximize the same proxy family that later generates the reported factuality numbers (e.g., 3.6->9.9 correct claims on cancer; 3.0->8.4 internal). The 40-query human holdout preserves rankings but does not calibrate absolute claim counts, and the five task-specific validation workflows are not released. The improvement is thus partly a measure of fit to the judge, not purely to ground truth.

  2. self definitional [Section 3.3 / Appendix A (Summary Confidence Reward); Tables 1-2 uCoCoA rows]
    "By construction, CoCoA already integrates perplexity, so no separate perplexity term is calculated at inference; during training, we use perplexity uPerp alone as a cheaper proxy. ... Rconf(τ ) = 1/uPerp(s(τ ))."

    Training uses Rconf(τ) = 1/uPerp(s(τ)) as a terminal reward, directly minimizing the perplexity of the emitted summary. The paper's summary-uncertainty evaluation uses uCoCoA, which is defined as perplexity multiplied by a consistency term. Therefore the reported decreases in uCoCoA (e.g., 0.47->0.19 on cancer; 0.45->0.20 internal) are partly forced by the training objective: the model is rewarded for lowering one of the multiplicative factors of the evaluation metric. The consistency component and the PRR alignment are not directly optimized, so the circularity is partial rather than total.

full rationale

The two flagged steps are genuine overlaps between training signals and evaluation metrics, but they are not complete reductions. First, the factuality judge (o4-mini) is the same model family as the RJudge reward, so the headline 'correct claims per summary' gains are partially an evaluation with the training proxy; the paper itself notes in Section 4.6 that 'Optimizing a single judge invites reward hacking' and offers only a 40-query human holdout with preserved rankings, which does not validate absolute counts. Second, the summary-uncertainty metric uCoCoA contains perplexity by construction, and perplexity is the training reward Rconf, so the measured drop in uCoCoA is partly a direct consequence of the optimization objective. However, retrieval uncertainty is inference-only and independent of the training reward; the PRR alignment and the consistency component of CoCoA are not directly optimized; and the downstream survival prediction (C-index 0.32->0.63) is an external, judge-free benchmark, even if under-specified. The paper's self-citations are background only and not load-bearing. These independent elements keep the circularity partial, so a score of 6 is appropriate rather than 8 or 10.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical entities or forces. The main free parameters are reward shaping weights and the inference threshold, which are tuned on validation data. The central claim rests on the reliability of LLM-based evaluation and on the environmental tool execution.

free parameters (4)
  • Reward weight α_conf schedule = Rbase: 1/3; Rstep: 2 every 10 steps; Radapt: 2α with α = exp(-50(RJudge - 1/2)^2)
    Weights chosen empirically to balance exploration and exploitation; paper admits 'Constants are empirically chosen' in Appendix A.
  • Inference threshold κ = 0.5 default, with 0.2 and 0.8 explored
    Threshold tuned on validation split through human inspection, as stated in Section 3.6.
  • Learning rate = 5e-5
    Searched over range 1e-7 to 1e-4, as reported in Appendix D.
  • Max tool calls = 6
    Search over 4-10 tool calls; marginal improvements beyond 6, per Appendix D.
axioms (3)
  • domain assumption LLM judge (o4-mini) provides valid atomic claim decomposition and fact-checking
    Used for both training reward (RJudge) and evaluation (Q1-Q3). Only 40 human queries validate this, so all correctness figures rely on this assumption.
  • domain assumption The C-index from the unstated survival prediction protocol is meaningful
    Section 4.7 reports C-index without describing the prediction procedure, patient sample, ground truth, or how summaries are converted to survival times.
  • domain assumption The multi-omics database environment executes SQL and Python tools faithfully
    The agent's actions are assumed to be executed correctly by the environment; no verification or error analysis of tool calls is provided.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning." pith.science (2026). https://pith.science/paper/CT6T5OCB

@misc{pith2026250902401,
  author       = {Pith},
  title        = {Pith review of: Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CT6T5OCB}},
  note         = {Machine review of arXiv:2509.02401}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Large language model (LLM) agents are increasingly deployed in structured biomedical data environments, yet they often produce fluent but overconfident outputs when reasoning over complex multi-table data. We introduce an uncertainty-aware agent for query-conditioned multi-table summarization that leverages two complementary signals: (i) retrieval uncertainty--entropy over multiple table-selection rollouts--and (ii) summary uncertainty--combining self-consistency and perplexity. Summary uncertainty is incorporated into reinforcement learning (RL) with Group Relative Policy Optimization (GRPO), while both retrieval and summary uncertainty guide inference-time filtering and support the construction of higher-quality synthetic datasets. On multi-omics benchmarks, our approach improves factuality and calibration, nearly tripling correct and useful claims per summary (3.0\(\rightarrow\)8.4 internal; 3.6\(\rightarrow\)9.9 cancer multi-omics) and substantially improving downstream survival prediction (C-index 0.32\(\rightarrow\)0.63). These results demonstrate that uncertainty can serve as a control signal--enabling agents to abstain, communicate confidence, and become more reliable tools for complex structured-data environments.

Figures

Figures reproduced from arXiv: 2509.02401 by Gianluca Mazzoni, Jesper Ferkinghoff-Borg, Josefa Lia Stoisser, Julien Fauqueur, Kaspar Martens, Lawrence Phillips, Lea M{\o}rch Harder, Marc Boubnovski Martell, Philip Torr.

Figure 1
Figure 1. Figure 1: The Uncertainty-Aware Agent Framework. This diagram shows the two phases of our agent: (a) training with reinforcement learning, and (b) inference. In training, the agent’s policy is refined using a reward signal informed by summary uncertainty (perplexity). During inference, multiple roll￾outs generate candidate summaries, which are then filtered based on a combined score of retrieval and summary uncertai… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier

    cs.AI 2026-06 unverdicted novelty 6.0

    Agent systems lose uncertainty at decision handoffs, causing downstream over-trust; the paper proposes latent uncertainty as a carrier to preserve pre-commitment fragility across interfaces.

  2. Calibration-Aware Policy Optimization for Reasoning LLMs

    cs.LG 2026-04 unverdicted novelty 6.0

    CAPO improves LLM calibration by up to 15% while matching or exceeding GRPO accuracy through logistic AUC loss and noise masking, enabling better abstention and scaling performance.

Reference graph

Works this paper leans on

90 extracted references · 44 canonical work pages · cited by 2 Pith papers · 7 internal anchors

  1. [1]

    Exploring LLM agents for cleaning tabular machine learning datasets

    Tommaso Bendinelli, Artur Dox, and Christian Holz. Exploring LLM agents for cleaning tabular machine learning datasets. arXiv preprint arXiv:2503.06664, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Rambla: a framework for evalu- ating the reliability of LLMs as assistants in the biomedical domain

    William James Bolton, Rafael Poyiadzi, Edward R Morrell, Gabriela van Bergen Gonzalez Bueno, and Lea Goetz. Rambla: a framework for evalu- ating the reliability of LLMs as assistants in the biomedical domain. arXiv preprint arXiv:2403.14578, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Uqlm: A python package for uncertainty quantification in large language models

    Dylan Bouchard, Mohit Singh Chauhan, David Skarbrevik, Ho-Kyeong Ra, Viren Bajaj, and Zeya Ahmad. Uqlm: A python package for uncertainty quantification in large language models. arXiv preprint arXiv:2507.06196 , 2025. 14

    work page arXiv 2025

  4. [4]

    The uk biobank resource with deep phenotyping and genomic data

    Clare Bycroft, Colin Freeman, Desislava Petkova, Gavin Band, Lloyd T Elliott, Kevin Sharp, Allan Motyer, Damjan Vukcevic, Olivier Delaneau, Jared O’Connell, et al. The uk biobank resource with deep phenotyping and genomic data. Nature, 562(7726):203–209, 2018

    2018

  5. [5]

    Multi2: Multi-agent test-time scalable frame- work for multi-document processing

    Juntai Cao, Xiang Zhang, Raymond Li, Chuyuan Li, Chenyu You, Shafiq Joty, and Giuseppe Carenini. Multi2: Multi-agent test-time scalable frame- work for multi-document processing. arXiv preprint arXiv:2502.20592 , 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Spe- cia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 , 2017

    Pith/arXiv arXiv 2017

  7. [7]

    The gtex consortium atlas of genetic regulatory effects across human tissues

    GTEx Consortium. The gtex consortium atlas of genetic regulatory effects across human tissues. Science, 369(6509):1318–1330, 2020

    2020

  8. [8]

    The entropy mechanism of reinforcement learning for reasoning language models

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Multi-LLM debate: Framework, princi- pals, and interventions

    Andrew Estornell and Yang Liu. Multi-LLM debate: Framework, princi- pals, and interventions. Advances in Neural Information Processing Sys- tems, 37:28938–28964, 2024

    2024

  10. [10]

    Faithfulness-aware uncertainty quan- tification for fact-checking the output of retrieval augmented generation

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Roman Vashurin, Shehzaad Dhuliawala, Artem Shelmanov, Timothy Baldwin, Preslav Nakov, Mrin- maya Sachan, and Maxim Panov. Faithfulness-aware uncertainty quan- tification for fact-checking the output of retrieval augmented generation. arXiv preprint arXiv:2505.21072 , 2025

    Pith/arXiv arXiv 2025

  11. [11]

    LM-polygraph: Uncertainty estimation for language models

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. LM-polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:2311.07383 , 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Large language models (LLMs) on tabular data: Prediction, generation, and understanding–a survey

    Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (LLMs) on tabular data: Prediction, generation, and understanding–a survey. arXiv preprint arXiv:2402.17944, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  14. [14]

    Robust gymnasium: A uni- fied modular benchmark for robust reinforcement learning

    Shangding Gu, Laixi Shi, Muning Wen, Ming Jin, Eric Mazumdar, Yuejie Chi, Adam Wierman, and Costas Spanos. Robust gymnasium: A uni- fied modular benchmark for robust reinforcement learning. arXiv preprint arXiv:2502.19652, 2025. 15

    Pith/arXiv arXiv 2025

  15. [15]

    Riche- lieu: Self-evolving LLM-based agents for ai diplomacy

    Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Riche- lieu: Self-evolving LLM-based agents for ai diplomacy. Advances in Neural Information Processing Systems, 37:123471–123497, 2024

    2024

  16. [16]

    eC-Tab2Text: Aspect-Based Text Generation from e-Commerce Product Tables

    Luis Antonio Guti´ errez Guanilo, Mir Tafseer Nayeem, Cristian L´ opez, and Davood Rafiei. ec-tab2text: Aspect-based text generation from e-commerce product tables. arXiv preprint arXiv:2502.14820 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Deepseek- r1: Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek- r1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    Pith/arXiv arXiv 2025

  18. [18]

    The need for guardrails with large language models in medical safety-critical settings: An artificial intelligence application in the pharma- covigilance ecosystem

    Joe B Hakim, Jeffery L Painter, Darmendra Ramcharran, Vijay Kara, Greg Powell, Paulina Sobczak, Chiho Sato, Andrew Bate, and Andrew Beam. The need for guardrails with large language models in medical safety-critical settings: An artificial intelligence application in the pharma- covigilance ecosystem. arXiv preprint arXiv:2407.18322 , 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Towards uncertainty- aware language agent

    Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. Towards uncertainty- aware language agent. arXiv preprint arXiv:2401.14016 , 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Uncertainty-aware gui agent: Adaptive perception through component recommendation and human-in- the-loop refinement

    Chao Hao, Shuai Wang, and Kaiwen Zhou. Uncertainty-aware gui agent: Adaptive perception through component recommendation and human-in- the-loop refinement. arXiv preprint arXiv:2508.04025 , 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Agentgen: Enhancing plan- ning abilities for large language model based agent via environment and task generation

    Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian-Guang Lou, Qing- wei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing plan- ning abilities for large language model based agent via environment and task generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 , pages 496–507, 2025

    2025

  22. [22]

    Un- certainty of thoughts: Uncertainty-aware planning enhances information seeking in LLMs

    Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei W Koh, and Bryan Hooi. Un- certainty of thoughts: Uncertainty-aware planning enhances information seeking in LLMs. Advances in Neural Information Processing Systems , 37:24181–24215, 2024

    2024

  23. [23]

    On the importance of exploration for generalization in reinforcement learning

    Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generalization in reinforcement learning. Advances in Neural Information Processing Systems , 36:12951–12986, 2023

    2023

  24. [24]

    Genegpt: Aug- menting large language models with domain tools for improved access to biomedical information

    Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: Aug- menting large language models with domain tools for improved access to biomedical information. Bioinformatics, 40(2):btae075, 2024

    2024

  25. [25]

    A roadmap for multi- omics data integration using deep learning

    Mingon Kang, Euiseong Ko, and Tesfaye B Mersha. A roadmap for multi- omics data integration using deep learning. Briefings in Bioinformatics , 23(1):bbab454, 2022. 16

    2022

  26. [26]

    Structtext: A synthetic table-to-text approach for benchmark generation with multi-dimensional evaluation

    Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, and Horst Samulowitz. Structtext: A synthetic table-to-text approach for benchmark generation with multi-dimensional evaluation. arXiv preprint arXiv:2507.21340, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Synthetic data production for biomedical research

    Yun Gyeong Lee, Mi-Sook Kwak, Jeong Eun Kim, Min Sun Kim, Dong Un No, and Hee Youl Chai. Synthetic data production for biomedical research. Osong Public Health and Research Perspectives , 16(2):94, 2025

    work page 2025

  28. [28]

    A Conceptual Framework for Human-AI Collaborative Genome Annotation

    Xiaomei Li, Alex Whan, Meredith McNeil, David Starns, Jessica Irons, Samuel C Andrew, and Rad Suchecki. A conceptual framework for human- ai collaborative genome annotation. arXiv preprint arXiv:2503.23691 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    Pith/arXiv arXiv 1907

  30. [30]

    The truth becomes clearer through debate! multi-agent systems with large lan- guage models unmask fake news

    Yuhan Liu, Yuxuan Liu, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. The truth becomes clearer through debate! multi-agent systems with large lan- guage models unmask fake news. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Re- trieval, pages 504–514, 2025

    work page 2025

  31. [31]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 , 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Large language model for table processing: A survey

    Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large language model for table processing: A survey. Frontiers of Computer Science, 19(2):192350, 2025

    work page 2025

  33. [33]

    Confidence Estimation for Text-to-SQL in Large Language Models

    Sepideh Entezari Maleki, Mohammadreza Pourreza, and Davood Rafiei. Confidence estimation for text-to-sql in large language models. arXiv preprint arXiv:2508.14056, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    A scalable llm framework for therapeutic biomarker discovery: Grounding q/a generation in knowledge graphs and literature

    Marc Boubnovski Martell, Kaspar M¨ artens, Lawrence Phillips, Daniel Keit- ley, Maria Dermit, and Julien Fauqueur. A scalable llm framework for therapeutic biomarker discovery: Grounding q/a generation in knowledge graphs and literature. In ICLR 2025 Workshop on Machine Learning for Genomics Explorations

    work page 2025

  35. [35]

    Matsa: Multi- agent table structure attribution

    Puneet Mathur, Alexa Siu, Nedim Lipka, and Tong Sun. Matsa: Multi- agent table structure attribution. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 250–258, 2024. 17

    work page 2024

  36. [36]

    Deep bayesian active learning for preference modeling in large language models

    Luckeciano C Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling in large language models. Advances in Neural Information Processing Systems , 37:118052– 118085, 2024

    work page 2024

  37. [37]

    Benchmarking the confidence of large language models in answering clinical questions: cross-sectional evaluation study

    Mahmud Omar, Reem Agbareia, Benjamin S Glicksberg, Girish N Nad- karni, and Eyal Klang. Benchmarking the confidence of large language models in answering clinical questions: cross-sectional evaluation study. JMIR Medical Informatics , 13:e66917, 2025

    work page 2025

  38. [38]

    Visualization of very large high- dimensional data sets as minimum spanning trees

    Daniel Probst and Jean-Louis Reymond. Visualization of very large high- dimensional data sets as minimum spanning trees. Journal of Cheminfor- matics, 12(1):12, 2020

    work page 2020

  39. [39]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 , 2019

    Pith/arXiv arXiv 1908

  40. [40]

    Self-reflection in LLM agents: Effects on problem-solving performance

    Matthew Renze and Erhan Guven. Self-reflection in LLM agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682 , 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Mag-v: A multi-agent framework for synthetic data generation and verification

    Saptarshi Sengupta, Harsh Vashistha, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, and Liang Gou. Mag-v: A multi-agent framework for synthetic data generation and verification. arXiv preprint arXiv:2412.04494, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024

    Pith/arXiv arXiv 2024

  43. [43]

    A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions

    Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Ma- jumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Comput- ing Surveys , 2025

    work page 2025

  44. [44]

    Agen- tic reasoning and tool integration for LLMs via reinforcement learning

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agen- tic reasoning and tool integration for LLMs via reinforcement learning. arXiv preprint arXiv:2505.01441 , 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Confidence estimation for error de- tection in text-to-sql systems

    Oleg Somov and Elena Tutubalina. Confidence estimation for error de- tection in text-to-sql systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25137–25145, 2025

    work page 2025

  46. [46]

    Finesure: Fine-grained summarization evaluation using LLMs

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Man- sour. Finesure: Fine-grained summarization evaluation using LLMs. arXiv preprint arXiv:2407.00908, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Why uncer- tainty estimation methods fall short in RAG: An axiomatic analysis

    Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. Why uncer- tainty estimation methods fall short in RAG: An axiomatic analysis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher 18 Pilehvar, editors, Findings of the Association for Computational Linguis- tics: ACL 2025 , July 2025

    work page 2025

  48. [48]

    Sparks of Tabular Reasoning via Text2SQL Reinforcement Learning

    Josefa Lia Stoisser, Marc Boubnovski Martell, and Julien Fauqueur. Sparks of tabular reasoning via text2sql reinforcement learning. arXiv preprint arXiv:2505.00016, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Query, don’t train: Privacy-preserving tabular prediction from ehr data via sql queries

    Josefa Lia Stoisser, Marc Boubnovski Martell, Kaspar M ˜A¯Irtens, Lawrence Phillips, Stephen Michael Town, Rory Donovan-Maiye, and Julien Fauqueur. Query, don’t train: Privacy-preserving tabular prediction from ehr data via sql queries. arXiv preprint arXiv:2505.21801 , 2025

    arXiv 2025

  50. [50]

    Struct-llm: Unifying tabular and graph reasoning with reinforcement learning for semantic parsing

    Josefa Lia Stoisser, Marc Boubnovski Martell, Lawrence Phillips, Casper Hansen, and Julien Fauqueur. Struct-llm: Unifying tabular and graph reasoning with reinforcement learning for semantic parsing. arXiv preprint arXiv:2506.21575, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Evaluating model ro- bustness and stability to dataset shift

    Adarsh Subbaswamy, Roy Adams, and Suchi Saria. Evaluating model ro- bustness and stability to dataset shift. In International conference on ar- tificial intelligence and statistics , pages 2611–2619. PMLR, 2021

    work page 2021

  52. [52]

    Chain-of-query: Unleashing the power of LLMs in sql- aided table understanding via multi-agent collaboration

    Songyuan Sui, Hongyi Liu, Serena Liu, Li Li, Soo-Hyun Choi, Rui Chen, and Xia Hu. Chain-of-query: Unleashing the power of LLMs in sql- aided table understanding via multi-agent collaboration. arXiv preprint arXiv:2508.15809, 2025

    arXiv 2025

  53. [53]

    On Generalization Across Environments In Multi-Objective Reinforcement Learning

    Jayden Teoh, Pradeep Varakantham, and Peter Vamplew. On generaliza- tion across environments in multi-objective reinforcement learning. arXiv preprint arXiv:2503.00799, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence

    Jiaming Tian, Liyao Li, Wentao Ye, Haobo Wang, Lingxin Wang, Lihua Yu, Zujie Ren, Gang Chen, and Junbo Zhao. Toward real-world table agents: Capabilities, workflows, and design principles for LLM-based table intelligence. arXiv preprint arXiv:2507.10281 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    Uncertainty quan- tification for LLMs through minimum bayes risk: Bridging confidence and consistency

    Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shelmanov, and Maxim Panov. Uncertainty quan- tification for LLMs through minimum bayes risk: Bridging confidence and consistency. arXiv preprint arXiv:2502.04964 , 2025

    arXiv 2025

  56. [56]

    Uncertainty-aware attention heads: Efficient unsupervised uncertainty quantification for LLMs

    Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Mrinmaya Sachan, Preslav Nakov, et al. Uncertainty-aware attention heads: Efficient unsupervised uncertainty quantification for LLMs. arXiv preprint arXiv:2505.20045, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Tabular data understanding with LLMs: A survey of recent advances and challenges

    Xiaofeng Wu, Alan Ritter, and Wei Xu. Tabular data understanding with LLMs: A survey of recent advances and challenges. arXiv preprint arXiv:2508.00217, 2025. 19

    Pith/arXiv arXiv 2025

  58. [58]

    An empirical analysis of uncertainty in large language model evalu- ations

    Qiujie Xie, Qingqiu Li, Zhuohao Yu, Yuejie Zhang, Yue Zhang, and Linyi Yang. An empirical analysis of uncertainty in large language model evalu- ations. arXiv preprint arXiv:2502.10709 , 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Hallucination is in- evitable: An innate limitation of large language models

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is in- evitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Mlomics: Can- cer multi-omics database for machine learning

    Ziwei Yang, Rikuto Kotoge, Xihao Piao, Zheng Chen, Lingwei Zhu, Peng Gao, Yasuko Matsubara, Yasushi Sakurai, and Jimeng Sun. Mlomics: Can- cer multi-omics database for machine learning. Scientific Data , 12(1):913, 2025

    work page 2025

  61. [61]

    Uncertainty-aware step-wise verification with generative reward models

    Zihuiwen Ye, Luckeciano Carvalho Melo, Younesse Kaddar, Phil Blunsom, Sam Staton, and Yarin Gal. Uncertainty-aware step-wise verification with generative reward models. arXiv preprint arXiv:2502.11250 , 2025

    Pith/arXiv arXiv 2025

  62. [62]

    Enhancing scientific table understanding with type-guided chain-of-thought

    Zhen Yin and Shenghua Wang. Enhancing scientific table understanding with type-guided chain-of-thought. Information Processing & Management, 62(4):104159, 2025

    work page 2025

  63. [63]

    Tablerag: A retrieval aug- mented generation framework for heterogeneous document reasoning.arXiv preprint arXiv:2506.10380, 2025

    Xiaohan Yu, Pu Jian, and Chong Chen. Tablerag: A retrieval aug- mented generation framework for heterogeneous document reasoning.arXiv preprint arXiv:2506.10380, 2025

    arXiv 2025

  64. [64]

    No free lunch: Rethinking internal feedback for LLM reasoning

    Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for LLM reasoning. arXiv preprint arXiv:2506.17219, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Uncertainty propagation on LLM agent

    Qiwei Zhao, Dong Li, Yanchi Liu, Wei Cheng, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Huaxiu Yao, Chen Zhao, Haifeng Chen, and Xujiang Zhao. Uncertainty propagation on LLM agent. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Com- putational...

    work page 2025

  66. [66]

    Saup: Situation awareness uncertainty propagation on LLM agent

    Qiwei Zhao, Xujiang Zhao, Yanchi Liu, Wei Cheng, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Huaxiu Yao, and Haifeng Chen. Saup: Situation awareness uncertainty propagation on LLM agent. arXiv preprint arXiv:2412.01033, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    QTSumm: Query-Focused Summarization over Tabular Data

    Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, et al. Qt- summ: Query-focused summarization over tabular data. arXiv preprint arXiv:2305.14303, 2023. 20

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  68. [68]

    Evaluating judges as evaluators: The jetts benchmark of LLM-as-judges as test-time scaling evaluators

    Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of LLM-as-judges as test-time scaling evaluators. arXiv preprint arXiv:2504.15253 , 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019. A Additional Methods Details This section collects additional details about our setup that were omitted from the main text for clarity. Summ...

    Pith/arXiv arXiv 1909

  70. [70]

    How many patients are in the training set?

    work page

  71. [71]

    What is the median survival time?

    work page

  72. [72]

    What is the event rate (percentage of deaths)?

    work page

  73. [73]

    Describe the survival distribution

    work page

  74. [74]

    24 Task 2: Molecular Data Profile Objective: For a specified cancer type CANCER TYPE, analyze each omic layer:

    Compare this cancer’s survival patterns to other cancers in the database. 24 Task 2: Molecular Data Profile Objective: For a specified cancer type CANCER TYPE, analyze each omic layer:

    work page

  75. [75]

    Data distribution characteristics for each omic type

    work page

  76. [76]

    Missing value analysis

    work page

  77. [77]

    Task 3: Cancer-Specific Biomarkers Objective: For a specified cancer type CANCER TYPE, identify and ana- lyze biomarkers:

    Create a molecular profile summary specific to this cancer type. Task 3: Cancer-Specific Biomarkers Objective: For a specified cancer type CANCER TYPE, identify and ana- lyze biomarkers:

    work page

  78. [78]

    Identify top survival-associated features from each omic type: • Top 20 mRNA features • Top 20 miRNA features • Top 20 methylation sites • Top 20 CNV regions

    work page

  79. [79]

    Analyze their biological relevance

    work page

  80. [80]

    Compare with known markers for this cancer type

    work page

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.