{"id":"efe44f92-3f32-4e2b-be82-19db6713837f","arxiv_id":"2412.14860","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Think&Cite uses self-guided Monte Carlo tree search and progress reward modeling to improve attributed text generation, reporting gains on ASQA, QAMPARI, and ELI5.","lead":"This paper introduces Think&Cite, a framework that combines Monte Carlo tree search with a progress-based reward signal to make large language models generate text with supporting citations. Across three question-answering benchmarks, the method reports higher correctness and citation quality than prompting and fine-tuning baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.3.1's generation progress reward is ill-defined: w_0 = 1/|y_{1:0}| divides by zero, and the arbitrary weights are absent from the DPO telescoping-sum derivation, so the published reward cannot be what was evaluated.","rationale":"The reader's weakest assumption correctly identifies the arbitrary weights in the generation progress reward. I sharpen this to a concrete internal inconsistency: the published formula has w_0 = 1/|y_{1:0}| = 1/0, making the reward undefined for every node. This is more specific than 'arbitrary weights' because it is a direct mathematical error, not merely a choice. The concern is load-bearing for the central claim because the ablation study shows that removing the GP reward lowers ASQA EM Recall from 50.1 to 47.1, so the component materially contributes to the reported improvement. If the formula is ill-defined, either the implementation differs from the paper (making the paper not self-contained) or the method literally computes an infinite reward (which is impossible). The empirical results may still be valid if the code uses a corrected weighting, but the theoretical grounding and reproducibility are compromised. I agree with the reader's overall CONDITIONAL verdict; the concern reinforces the need for a code check and a corrected derivation, but does not invalidate the strong empirical gains. The verdict should remain CONDITIONAL, hence UNCHANGED.","tokens_in":18312,"tokens_out":6090,"duration_ms":47500,"concrete_test":"Inspect the released code in the linked repository (github.com/nusnlp/Think-Cite), specifically the progress reward computation, and check whether w_k = 1/|y_{1:k}| is literally implemented for k=0. If the implementation uses a different weighting (e.g., w_k = 1/|y_{k+1}| or no weights), then the published formula is not the evaluated reward. If the implementation matches the paper, compute Rg for a one-sentence node: the k=0 term divides by zero, so the reward is undefined. Re-run the ASQA 'w/o GP Reward' ablation with the unweighted telescoping sum; if the gap to the full model changes materially, the reweighting is responsible for the reported gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The derivation of the generation progress reward Rg in Section 4.3.1 is internally inconsistent. The paper derives a telescoping sum Σ_{k=0}^t β log(π*(y_{k+1}|x,y1:k)/π_ref(...)) = V*(s_{t+1}) - V*(s_0), then defines Rg as the same sum but with arbitrary weights w_k = 1/|y_{1:k}|. These weights do not appear in the derivation, and they make the reward ill-defined: for k=0, y_{1:0} is the empty prefix, so |y_{1:0}| = 0 and w_0 = 1/0. Since Rg includes the k=0 term for every partial response, a literal implementation of the published equation would produce an infinite reward for every node. The paper does not state that k=0 is handled specially, nor does it justify why the telescoping sum should be reweighted. This is load-bearing because the ablation in Table 2 attributes 3 points of EM Recall on ASQA to the GP reward, so the central claim depends on this component functioning as a bounded, meaningful progress signal. If the released code does not implement the formula as written, the paper's theoretical grounding for SG-MCTS's generation reward does not match the evaluated system; if the code does implement it, the reward is undefined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Think&Cite, a framework for attributed text generation that combines a think-verbalize-cite language agent with Self-Guided Monte Carlo Tree Search (SG-MCTS) and Progress Reward Modeling (PRM). SG-MCTS uses LLM self-reflection to guide tree expansion, and PRM provides two reward signals: a generation progress reward based on DPO log-ratios and an attribution progress reward based on NLI citation recall/precision F1. The method is evaluated on ASQA, QAMPARI, and ELI5 with LLaMA-3.1-8B and GPT-4o, reporting gains over several prompting-based and fine-tuning baselines. The paper also includes ablation studies, hyperparameter analyses, cost comparisons, and a released code/data repository.","tokens_in":18627,"tokens_out":6177,"duration_ms":44397,"significance":"If the reported results hold, the paper makes a useful contribution: it is among the first to apply tree search to attributed text generation, and the self-reflection-guided expansion is a reasonable way to reduce search cost. The code and data release, the inclusion of multiple datasets, and the component-level ablations are strengths that help the community reproduce and build on the work. The main weakness is the theoretical grounding of the generation progress reward, which contains a derivation gap and an undefined term; this is fixable but should be addressed carefully, as the ablation attributes nontrivial gains to this reward.","major_comments":[{"comment":"The generation progress reward R_g is defined as a weighted sum of DPO log-ratios with weights w_k = 1/|y_{1:k}|, but the preceding derivation establishes an unweighted telescoping sum: Σ_{k=0}^t β log(π*(a_k|s_k)/π_ref(a_k|s_k)) = V*(s_{t+1}) - V*(s_0). The weights are introduced without any justification and do not follow from the derivation. Moreover, for k=0 the weight w_0 = 1/|y_{1:0}| is undefined because |y_{1:0}| = 0 (the empty prefix); the paper does not state a special case. Since R_g is evaluated at every partial response, a literal implementation of Eq. (9) would involve division by zero. This is load-bearing: the ablation in Table 2 attributes a 3-point EM Recall drop on ASQA to removing the GP reward, and the claimed theoretical grounding of SG-MCTS rests on this reward. The authors should either derive the weights from the MDP/DPO formulation, remove them, or explicitly define w_0 and explain why the reweighting is valid; they should also confirm that the released code implements the corrected formula.","section":"Section 4.3.1, Eq. (9)"},{"comment":"The statement that 'our approach outperforms all baselines significantly across all three datasets' is not supported by any statistical significance test. The table reports point estimates only; no confidence intervals, p-values, or multiple-comparison corrections are provided. Given that the proposed method is substantially more computationally expensive than the prompting baselines (Appendix B), the claim of significance should be backed by a bootstrap or paired test to rule out that the differences are within the noise of the evaluation sets.","section":"Section 5.2, Table 1"},{"comment":"The attribution progress reward R_a is computed as the F1 of citation recall and precision, which are exactly the citation-quality metrics used in the evaluation (Section 5.1). This is not circular reasoning, but it does mean that the reported citation gains arise in part because the search directly optimizes the evaluation objective. The paper should state this overlap explicitly and provide a robustness check, for example by measuring citation quality with a different NLI model or with human judgments, to show that the gains are not an artifact of optimizing the specific NLI model used both as reward and evaluator.","section":"Section 4.3.2 and Section 5.1"},{"comment":"The derivation identifies the DPO-aligned model's log-ratio with the optimal policy π* for the sentence-level MDP, but the chosen DPO model (Llama-3-8B-SFR-Iterative-DPO-R2) was not trained for attributed text generation and no evidence is given that its log-ratio is a meaningful progress signal for this task. A concrete test would be to compute the correlation between R_g and final answer quality (e.g., EM Recall or citation F1) across partial prefixes; without such evidence, the 'generation progress' interpretation of R_g remains an assumption.","section":"Section 4.3.1"}],"minor_comments":[{"comment":"The notation is inconsistent: the final displayed equation for R_g uses π* in the log-ratio, but the text says the reward is computed using a DPO policy. Please clarify that the DPO model serves as a proxy for π* in practice.","section":"Section 4.3.1"},{"comment":"The boundary conditions of the sentence-level MDP are under-specified: the terminal value V* is not defined, and the reward r(y|x) appears without definition. Please state the terminal value and how T (the number of sentences) is determined.","section":"Section 4.3.1"},{"comment":"The claim that this is 'the first to apply tree search algorithms to the task of attributed text generation' should be qualified, since Self-RAG, discussed in Section 2, explicitly supports tree-decoding with critique tokens. Please clarify the precise sense in which this is the first application.","section":"Section 1 and Section 2"},{"comment":"The prompt description refers to blue and green text colors, but the tables in the manuscript are monochrome. Either include the colored prompts as supplementary material or remove the color reference.","section":"Appendix C, Table 5"},{"comment":"The figure shows reward values (e.g., R_g = 0.25, R_a = 1.0) but does not explain how these numbers are computed or how they relate to Eq. (9). Adding a brief notation explanation would improve clarity.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about Eq. (9) is well-founded and corresponds to a real gap in the manuscript. I recommend requesting the authors to verify the exact formula implemented in their released code; if the code uses a different (e.g., unweighted or length-normalized) reward, the theory section must be updated accordingly. The overall empirical contribution is strong, and the issues are fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. This paper does something new and useful: it is the first to apply Monte Carlo tree search to attributed text generation, and the system—think-verbalize-cite agent with self-guided reflection in expansion and dual progress rewards—beats the prior prompting and fine-tuning baselines on ASQA, QAMPARI, and ELI5 by healthy margins. The ablations in Table 2 support the design choices, especially the removal of SG-MCTS. The authors also release code and data and credit prior tree-search and citation work properly. The attribution progress reward using the same NLI metric as evaluation is not circular; it is objective alignment, and the paper is upfront about it.\n\nThe soft spot is the generation progress reward in Section 4.3.1. The DPO telescoping-sum derivation is a clean motivation, but the move to the weighted sum R_g introduces weights w_k = 1/|y_{1:k}| that do not follow from the derivation, and the k=0 term divides by zero because the empty prefix has length zero. As printed, the formula is undefined for every partial response. I suspect the released code special-cases k=0 or weights by something like the length of the new sentence, but the paper does not say so. Since the ablation attributes about 3 EM points on ASQA to the GP reward, this is load-bearing. The theoretical grounding of SG-MCTS currently does not match the evaluated system.\n\nThe evaluation has a few smaller soft spots. There are no significance tests or variance estimates despite the word 'significantly'. Hyperparameters are tuned on the test sets. And the LLaMA comparison is not same-base: FG-Reward and APO are LLaMA-2-7B/13B, while the policy is LLaMA-3.1-8B, so that particular comparison is a system-level contrast, not a controlled one. None of these is fatal to the empirical story, which is consistent across three datasets and two policy models.\n\nMy take: this paper deserves a serious referee. It is for anyone working on attributed generation, RAG, or inference-time search. The empirical contribution is real and the framework is clearly described. The reward-derivation issue is fixable—rewrite the weights with a justification, or drop the derivation and call R_g a heuristic—and the evaluation could be tightened. I would send it out.","headline":"The first MCTS-for-attributed-generation paper with real gains; solid empirically, but the Section 4.3.1 reward derivation has a genuine formal hole (division by zero) that needs fixing before I'd trust the theory section.","tokens_in":19129,"tokens_out":3049,"would_cite":true,"duration_ms":27924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attributed text generation improves when a language model searches over partial answers, reflects on retrieval before each expansion, and is guided by progress rewards.","keywords":["attributed text generation","Monte Carlo tree search","self-reflection","progress reward modeling","direct preference optimization","citation quality","retrieval-augmented generation","long-form question answering"],"falsifier":"If a faithful re-implementation on ASQA, QAMPARI, and ELI5 fails to beat the best non-search baseline on both correctness and citation metrics, the central claim is falsified; a more surgical check is to ablate the generation-progress reward and see whether the reported citation scores on ASQA survive.","tokens_in":18110,"feed_emoji":"🔎","tokens_out":10819,"duration_ms":69438,"temperature":0.7,"pith_summary":"Think&Cite tries to establish that attributed text generation, producing long-form answers with in-text citations, is best solved as a multi-step search problem rather than a single auto-regressive pass. The paper builds a Monte Carlo tree whose nodes are partial answer states, a sentence together with its search query, retrieved passages, and citations, and lets the language model reflect on intermediate retrieval states before expanding the tree. A progress reward scores each partial state from two directions: how much closer the text is to a complete answer, and how well the cited passages support the sentences written so far. On the ASQA, QAMPARI, and ELI5 datasets, the resulting system outperforms prompting-based and fine-tuning baselines on both correctness and citation quality. That claim matters because it identifies search and intermediate reflection, rather than more training data or larger models, as a path to more verifiable long-form generation.","feed_headline":"Self-guided tree search lifts citation quality on three QA benchmarks","feed_subtitle":"An agent that reflects on retrieval before each sentence, guided by progress rewards, beats prompting and fine-tuning baselines","key_machinery":"The engine is Self-Guided Monte Carlo Tree Search (SG-MCTS). A node is a state $s_t = [q_t, \\mathcal{D}_t, y_t, \\mathcal{C}_t]$ recording the retrieval query, the retrieved passages, the sentence generated, and the passages cited for that sentence. Selection follows the UCT formula $UCT(s_t)=V(s_t)+w\\sqrt{\\ln N(p)/N(s_t)}$; expansion runs the think-verbalize-cite loop with an added reflection step that critiques and reformulates the initial query based on the question and the retrieved passages. New nodes are scored by Progress Reward Modeling: $R(s_{t+1}) = R_g + R_a$, where $R_g$ is a weighted sum of sentence-level DPO log-ratios, $\\sum_{k=0}^t w_k \\log(\\pi_*(y_{k+1}\\mid x,y_{1:k})/\\pi_{\\mathrm{ref}}(y_{k+1}\\mid x,y_{1:k}))$, and $R_a$ is the F1 of NLI-based citation recall and precision. That reward is backpropagated to update each ancestor's value and visit count. The mechanism is what allows the model to abandon weak evidence paths early rather than letting one bad retrieval contaminate every later sentence.","core_discovery":"On the paper's own terms, the central discovery is that a think-verbalize-cite agent, when embedded in a self-guided tree search, produces answers that are both more factually complete and better supported by citations than standard retrieval-augmented generation. The search tree is built sentence by sentence; before a node is expanded, the model reflects on the proposed retrieval query and the passages it returned, and reformulates the query if the evidence looks weak. Each expanded node is scored by a progress reward that combines a generation-progress term derived from preference-optimized log-ratios with an attribution-progress term based on NLI citation recall and precision. The strongest configuration reaches 50.1 exact-match recall and 89.5 citation recall on ASQA, with gains on QAMPARI and ELI5, and the ablations show that removing the tree search, the reflection step, or either reward component all hurt performance.","pith_inferences":["Beyond the paper: the theoretical derivation of $R_g$ yields a telescoping sum that needs no per-sentence weights; the weights $w_k=1/|y_{1:k}|$ are an extra modeling choice, and comparing weighted versus unweighted variants would test whether the grounding or the heuristic is doing the work.","The same reward decomposition could plausibly transfer to other evidence-constrained generation tasks such as summarization, fact-checking, or dialogue grounding, because neither the DPO reward model nor the NLI model is task-specific.","The reflection step is currently applied to retrieval queries; a natural extension is to reflect on the drafted sentence before finalizing citations, which would create a second chance to catch unsupported claims before they enter the tree.","The paper's own limitations note the computational cost of tree search; a practical extension is to test whether an adaptive stopping rule based on progress rewards can recover most of the quality gain at a fraction of the fixed iteration budget."],"forward_implications":["Base models can gain attribution ability at inference time: with a DPO-aligned reward model and an NLI model, the policy model itself does not need fine-tuning for citation quality.","Reflection on intermediate retrieval states should stop error cascades: a bad initial query is corrected before a sentence is written, so downstream sentences do not inherit the bad evidence.","Search budget can be traded for quality: increasing MCTS iterations and reflection steps improves correctness and citation scores up to a point, after which over-reflection begins to hurt.","Generation and attribution checks are complementary: ablating either reward component degrades results, so both text quality and citation consistency contribute to the final score."],"supporting_citations":[{"why":"Supplies the ALCE benchmark, the three datasets, the 100-word passage setup, the citation recall/precision metrics, and the prompting baselines that Think&Cite must beat.","marker":"Gao et al., 2023b"},{"why":"Provides the direct preference optimization framework whose token-level log-ratio is treated as an implicit reward, grounding the generation progress reward.","marker":"Rafailov et al., 2023"},{"why":"Supplies the UCT selection rule used to choose which node of the search tree to expand.","marker":"Kocsis and Szepesvári, 2006"},{"why":"Supplies the NLI model used to decide whether cited passages entail each generated sentence for the attribution progress reward.","marker":"Honovich et al., 2022"},{"why":"Provides prior reflection-on-final-outcome work that motivates the paper's shift to reflection on intermediate MCTS states.","marker":"Shinn et al., 2023"},{"why":"Defines process reward models, the predecessor the paper contrasts with its progress-based reward view.","marker":"Lightman et al., 2024"},{"why":"A baseline approach that attributes first and then generates, which the paper's iterative search is designed to outperform.","marker":"Slobodkin et al., 2024"},{"why":"Supplies the VTG baseline with verifiers and evolving memory that Think&Cite must beat on citation quality.","marker":"Sun et al., 2024"}],"fun_headline_variants":["Self-guided tree search boosts citation recall","Think&Cite: planning citations via tree search","Progress rewards improve LLM attribution quality","Search-guided agent cites better than RAG baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's search signal rests on the assumption that a preference-tuned language model's sentence-level scores really measure how much closer the partial answer is to a good final answer; if that proxy fails, the generation-progress reward is an ad hoc heuristic rather than a grounded value estimate.","fun_headline_variants_meta":{"raw":{"variants":["Self-guided tree search boosts citation recall","Think&Cite: planning citations via tree search","Progress rewards improve LLM attribution quality","Search-guided agent cites better than RAG baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":1103,"prompt_tokens":881,"completion_tokens":222,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":166}},"tokens_in":497,"tokens_out":222,"duration_ms":2274,"temperature":1.0,"reasoning_tokens":166,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:50:26.682793+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a faithful re-implementation on ASQA, QAMPARI, and ELI5 fails to beat the best non-search baseline on both correctness and citation metrics, the central claim is falsified; a more surgical check is to ablate the generation-progress reward and see whether the reported citation scores on ASQA survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UCT selection rule used to choose which node of the search tree to expand."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NLI model used to decide whether cited passages entail each generated sentence for the attribution progress reward."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides prior reflection-on-final-outcome work that motivates the paper's shift to reflection on intermediate MCTS states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A baseline approach that attributes first and then generates, which the paper's iterative search is designed to outperform."}],"review_version":1}