{"id":"d3623213-7b6e-44e5-9738-8ddccdb4fe8d","arxiv_id":"2607.21961","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AI-generated podcasts often add unsupported claims; a turn-level detector plus rewrite pass improves measured faithfulness across five models and in- and out-of-domain documents.","lead":"This paper measures how often AI-generated podcast transcripts drift from the documents they are based on, and proposes a two-step fix that catches and rewrites unfaithful conversational turns. It provides one of the first systematic benchmarks for faithfulness in long-form, multi-speaker podcast generation, which matters as tools like NotebookLM turn documents into audio.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No human validation of catch-n-repair outputs: LLM-judge improvements in Table 3 may not reflect human-perceived faithfulness.","rationale":"The reader's weakest assumption — that LLM-judge scores are a valid proxy for human-judged faithfulness on repaired outputs — is the most load-bearing point. The paper validates the judge on only 328 vanilla turns, and the repaired turns are generated under a different prompt distribution, so the judge's behavior on them is unknown. The training/evaluation loop is also saturated with LLM judgments, making the absence of human evaluation of CnR outputs a real gap. Because the reader already recommended CONDITIONAL with this concern as the core condition, my read does not change the verdict; I agree with the reader's framing and would uphold the CONDITIONAL. My concrete test would settle whether the concern lands by directly measuring human-perceived deltas for the repair pipeline.","tokens_in":14094,"tokens_out":5056,"duration_ms":53751,"concrete_test":"Run a human annotation study on catch-n-repair outputs: for a random sample of at least 10 held-out documents and two generator models (e.g., Qwen2.5-14B and GPT-4o), collect both vanilla and CnR transcripts; have at least 3 annotators per transcript score every turn using the same 1-5 grounding rubric as Sec. A.8, with condition blinded; compute the mean human-scored delta (CnR − vanilla). If the human delta is not significantly >0, or is significantly lower than the corresponding LLM-judge delta in Table 3, the claim that CnR improves human-perceived faithfulness fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of Sec. 5.5 — 'Across all models and both ID and OOD settings, CnR improves faithfulness scores' — rests on the validity of the LLM-as-a-judge metric for repaired turns. Sec. 5.3 validates the judge only on 328 original (vanilla) turns, with Pearson r=0.63. Repaired turns differ systematically: Sec. 4.3 instructs the generator to rewrite turns using only facts explicitly stated in the source document, which likely shifts output style (more conservative, more explicitly source-aligned). LLM judges are known to be sensitive to style; without human scores on CnR transcripts, the reported Δ in Table 3 conflates 'improvement according to GPT-4o/Qwen' with 'improvement in actual grounding.' This is exacerbated by the fact that the catch model itself is trained on LLM-judge labels (Sec. 4.1), so the entire pipeline is optimized toward LLM preferences. The abstract and conclusion claim improvements in faithfulness simpliciter, but human grounding gains are not demonstrated. The absence of significance tests in Sec. 5.5 is a secondary weakening, but the judge-validity gap is primary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first systematic study of faithfulness in document-grounded podcast generation. The authors construct Doc-to-Podcast, a dataset of 1520 documents across five domains, and generate transcripts with five LLMs (GPT-4o, LLaMA3.3-70B, and Qwen2.5 14B/32B/72B). They introduce a turn-level LLM-as-a-judge protocol assigning Likert 1-5 grounding scores to each conversational turn, and validate this protocol against human annotations on 328 turns (Pearson r=0.63, Krippendorff α=0.69). They then propose catch-n-repair (CnR), a model-agnostic mitigation framework: a LoRA-fine-tuned LLaMA-3.1-8B binary classifier detects unfaithful turns, and the generator is re-prompted to rewrite flagged turns using only facts explicitly stated in the source. Experiments report consistent faithfulness gains with CnR across in-domain (ID) and out-of-domain (OOD) sets for all generators, as measured by GPT-4o and Qwen judges (Tables 3-4), with minimal impact on coverage (Table 5). The paper claims to be the first to study faithfulness in document-grounded podcast generation and to provide an effective turn-level grounding intervention.","tokens_in":14332,"tokens_out":5116,"duration_ms":55078,"significance":"The work addresses an important and underexplored problem: grounding in long-form, multi-speaker conversational generation. The Doc-to-Podcast dataset and the human annotation study for the judge are useful community resources. The proposed catch-n-repair is simple, model-agnostic, and black-box compatible, which makes it practically appealing. If the reported gains were shown to correspond to human-perceived faithfulness, the paper would make a solid contribution to trustworthy generative AI. However, as presented, the central claim that CnR 'improves faithfulness' is not fully established because the evaluation metric is validated only on original (vanilla) turns, while the repaired turns are scored exclusively by LLM judges. The whole pipeline is optimized toward LLM preferences: training labels come from LLaMA-3.1-70B as both generator and judge, the catch model is trained to imitate that judge, and the final scores are produced by GPT-4o or Qwen judges. The single external anchor—a moderate 0.63 correlation on vanilla turns—does not cover the distribution shift introduced by the repair step. This gap, combined with the lack of significance testing on the reported deltas, preve","major_comments":[{"comment":"The central claim that CnR improves faithfulness rests on LLM-as-a-judge scores for repaired turns. However, the human validation of the judge is limited to 328 original turns (Sec. 5.3). Repaired turns differ systematically: the repair prompt (Sec. 4.3) instructs the generator to rewrite using only facts explicitly stated in the source, which likely makes the turns more conservative and more lexically/semantically similar to the document. LLM judges are known to be sensitive to style and surface overlap; without human scores on CnR transcripts, the reported Δ of +0.1 to +0.5 in Table 3 may reflect judge preference rather than actual grounding gains. Please provide a human evaluation of a sample of CnR outputs, or at least a judge–human correlation computed separately on repaired turns. Until then, the paper should temper its claim of 'improving faithfulness.' This is the primary gap and","section":"Sec. 5.3 vs. Sec. 5.5 (Table 3)"},{"comment":"There is a potential circularity in the evaluation pipeline: the catch model is trained on labels produced by LLaMA-3.1-70B acting as both generator and evaluator (Sec. 4.1), and the final improvements are measured using GPT-4o and Qwen LLM judges (Tables 3-5). While the use of a different judge model for the final evaluation mitigates the concern of judge self-preference, all judges are LLMs and may share systematic biases toward conservative, source-aligned text. The only external anchor is a 0.63 correlation with humans on original turns. The paper does not demonstrate that this correlation holds for repaired turns. Concretely, if the LLM judge rewards the verbatim repetition of source phrases, the repair strategy would inflate scores without improving human-perceived grounding. A simple and decisive test would be to compute the judge–human correlation on repaired turns; if it is sign","section":"Sec. 4.1 and Sec. 5.5 (Tables 3-5)"},{"comment":"The reported improvements are not accompanied by statistical significance tests. Several deltas are small relative to the reported standard deviations—e.g., Qwen2.5-14B ID Δ=+0.1 with SDs of 0.5/0.6, and LLaMA3.3-70B ID Δ=+0.1 with SDs of 0.5. The claim that 'Across all models and both ID and OOD settings, CnR improves faithfulness scores' is therefore not supported by the data as presented. Please report per-document paired analyses (e.g., two-sided bootstrap or signed-rank tests with effect-size confidence intervals) and qualify any improvements that are not statistically reliable. This does not require a huge new experiment, but it is essential for a quantitative claim of consistent gains.","section":"Sec. 5.5, Table 3"},{"comment":"The catch model achieves an F1-score of only 67.3% on the unfaithful class (Sec. 5.4). Since the repair mechanism only corrects turns that are caught, the headline improvement may be driven by a small and potentially biased subset of turns that the detector happens to identify. The paper does not report the percentage of turns that were flagged and repaired, nor the per-turn outcomes for those cases. Reporting the repair rate and the judge-assigned improvements among repaired turns would clarify the mechanism. Moreover, the synthetic training data are generated by prompting LLaMA-3.1-70B to intentionally hallucinate, which may produce a narrow distribution of unfaithfulness; the detector's behavior on naturally occurring ungrounded content—which may be more subtle—remains uncharacterized.","section":"Sec. 4.2 and Sec. 5.4"}],"minor_comments":[{"comment":"The Figure 1 example uses 'Attention Is All You Need' and notes that ChatGPT is mentioned anachronistically. This is a good illustrative example, but the temporal framing (the paper is from 2017) should be stated explicitly to avoid confusion about the difference between factual correctness and grounding.","section":"Abstract / Intro"},{"comment":"Typo: 'Appendex' should be 'Appendix' in the pointer to A.2.","section":"Sec. 5.5"},{"comment":"The 'Avg.Δ' column is computed as the simple mean of ID and OOD deltas. Please clarify whether the per-setting sample sizes are equal, and consider reporting the weighted mean or the per-setting confidence intervals.","section":"Table 3 and Table 4"},{"comment":"The financial-domain link (annualreports.com) is a commercial site and may be volatile; please provide an access date or a more stable snapshot.","section":"Appendix A.1"},{"comment":"The limitations section acknowledges dependence on the catch classifier but does not mention that the LLM-judge evaluation is validated only on original turns. Adding a sentence about this would be appropriate and consistent with the paper's apparent honesty about scope.","section":"Sec. 7 Limitations"},{"comment":"The definition of coverage is vague: 'important information' is not operationalized. Consider specifying a rubric or using a small set of key points extracted from the source, so that the coverage metric is reproducible.","section":"Sec. 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and practical problem, and the dataset plus mitigation idea are credible. The main obstacle is the evaluation gap for repaired turns: the absolute necessity of human validation of CnR outputs, and the lack of significance tests on the headline gains. Both are addressable within a revision, and I would be willing to review a revised version. I would not recommend rejection, because the core approach is sound and the reported trends are promising, but the current evidence does not yet support the strong claim of improved faithfulness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuine first: nobody has systematically benchmarked faithfulness in document-to-podcast generation at turn level, and the paper does that carefully. The Doc-to-Podcast dataset (1520 docs, five domains), the turn-level Likert judge, and the cross-model analysis are useful contributions. The catch-n-repair method is simple and model-agnostic, and the apparent gains are consistent across five generators and two judges, with coverage roughly preserved. The human validation of the judge on 328 turns (r=0.63, alpha=0.69) is credible and better than existing automatic metrics. I would send this to a serious referee.\n\nThe soft spot is exactly where the stress-test lands: the judge is validated only on vanilla turns, and the repaired turns are never scored by humans. That matters because repair changes output style toward conservative, source-aligned language, and LLM judges are known to favor that style. The catch model is trained on LLM-judge labels too, so the whole pipeline is optimized toward what LLMs call faithful. The paper is honest about this in Section 5.3 and Limitations, but the conclusion still claims faithfulness improvements simpliciter, when the evidence supports \"LLM-judged faithfulness improvements.\" Also missing: significance tests on the deltas (several are +0.1 with overlapping standard deviations) and comparison to prior black-box mitigation baselines like SelfCheckGPT or RARR. Those are not fatal, but they weaken the contribution.\n\nThe abstract and conclusion slightly overclaim. The data and prompts are not released, which limits reproducibility, though the dataset sources are documented. The limitations section is candid, which I respect.\n\nFor whom: anyone building product pipelines for grounded long-form generation, especially in industry. The benchmark alone justifies reading it. The mitigation is plausible but not yet established as human-meaningful. I would cite it for the benchmark and the judge protocol, but not yet for the repair gains. It deserves peer review, with a request for human evaluation of catch-n-repair outputs and significance tests.","headline":"A solid first benchmark for faithfulness in document-to-podcast generation, but the headline gains rest on an LLM judge that is only validated on vanilla turns, not on catch-n-repair outputs.","tokens_in":14869,"tokens_out":758,"would_cite":true,"duration_ms":10562,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a turn-level catch-and-repair loop — a small classifier that flags ungrounded conversational turns plus a re-prompting step that rewrites them — improves faithfulness scores by roughly 0.1 to 0.5 points on a 5-point sc","keywords":["faithfulness","podcast generation","document grounding","hallucination mitigation","LLM-as-a-judge","catch-n-repair","long-form dialogue","turn-level evaluation"],"falsifier":"Have human annotators rate catch-n-repair transcripts from the same 40 test documents using the paper's own grounding instructions. If human-judge agreement on repaired turns is no better than chance, or if any domain shows human-rated faithfulness dropping while judge-rated faithfulness rises, the reported 0.1–0.5 gains would not establish real grounding.","tokens_in":13934,"feed_emoji":"🎙️","tokens_out":4291,"duration_ms":39142,"temperature":0.7,"pith_summary":"AI-generated podcasts that are supposed to summarize a document often add plausible but unsupported claims. This paper argues that faithfulness can be measured turn by turn, and that unfaithful turns can be caught and rewritten before they contaminate the rest of the conversation. The authors build a dataset of more than 1,500 documents across five domains, show that even strong models produce ungrounded turns, and propose catch-n-repair: a lightweight detector that flags unfaithful turns plus a re-prompting step that has the generator rewrite them. Across five generator models and both in-domain and out-of-domain documents, the method raises faithfulness scores by roughly 0.1 to 0.5 points on a 5-point scale while leaving coverage essentially unchanged. If the judge scores are trusted, the result shows that long-form conversational grounding can be improved without retraining the generator.","feed_headline":"Catch-and-repair loop keeps AI podcasts grounded in the source","feed_subtitle":"Turn-level rewriting lifts faithfulness scores by up to 0.5 points across five models, in and out of domain, without hurting coverage.","key_machinery":"The load-bearing object is the turn-level 'catch' classifier: a LoRA fine-tuned 8-billion-parameter instruction model that, given a source document and a single conversational turn, outputs faithful or unfaithful. It is trained on synthetic turns generated and labeled by a 70-billion-parameter model using a 1–5 Likert grounding prompt, with scores of 4 or above treated as faithful. During sequential generation, flagged turns are re-prompted for repair; because repair happens before subsequent turns are generated, errors do not propagate. The judge that measures outcomes is a separate LLM prompted to rate each turn's grounding on the same 1–5 scale.","core_discovery":"The central claim is that turn-level grounding, not transcript-level quality, is the right unit for faithfulness in document-grounded podcast generation, and that a model-agnostic detect-and-repair loop reliably improves judged faithfulness. The loop uses a fine-tuned classifier to flag each conversational turn as faithful or unfaithful; the generator is then re-prompted to rewrite flagged turns using only facts from the source document. The paper validates a turn-level LLM judge against human ratings on 328 turns (Pearson r=0.63, Krippendorff α=0.69), then applies catch-n-repair to five generators. It reports consistent improvements in both in-domain and out-of-domain settings, with larger","pith_inferences":["The same catch-and-repair pattern could transfer to other long-form grounded generation tasks — lecture scripts, meeting summaries, narrated articles — wherever turns can be judged against a source; the paper only demonstrates podcasts.","Because the judge is both the training-signal source and the outcome measure, part of the reported gain may reflect the judge's preference for repaired turns rather than human-perceived grounding; a human evaluation of catch-n-repair transcripts would settle this.","Detector accuracy on unfaithful turns (67.3% F1) sets a ceiling; improving recall on subtle hallucinations, or training on human-labeled rather than model-labeled data, should amplify the repair gains.","The judge could be replaced by a human-in-the-loop or a retrieval-verified scorer to make the loop work in high-stakes domains where a mislabeled turn is costly."],"forward_implications":["Faithfulness gains appear across all five generators tested, including a state-of-the-art closed model, so the method is not tied to a particular generator.","Out-of-domain gains are at least as large as in-domain gains, suggesting the detector generalizes beyond the five training domains.","Coverage is largely preserved; the largest drops are about 0.2 points on a 5-point scale, and in some cases coverage rises.","The improvement pattern holds when a different evaluator model replaces the primary judge, reducing the chance that gains are an artifact of one judge.","Turn-level repair is cheap: it requires only a small classifier and a re-prompt, not access to model internals or external retrieval."],"fun_headline_variants":["Turn-by-turn repair boosts faithfulness in AI podcasts","Detect-and-rewrite loop grounds podcast AI output","Catch-and-repair targets unfaithful podcast lines","Turn-level fix grounds AI podcast generation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole result rests on the assumption that the LLM judge's 1–5 faithfulness scores, which are used to train the detector, select repairs, and measure success, agree with human judgment on the repaired transcripts; the paper only validates the judge against humans on 328 original turns, with a Pearson correlation of 0.63.","fun_headline_variants_meta":{"raw":{"variants":["Turn-by-turn repair boosts faithfulness in AI podcasts","Detect-and-rewrite loop grounds podcast AI output","Catch-and-repair targets unfaithful podcast lines","Turn-level fix grounds AI podcast generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000443,"raw_usage":{"total_tokens":2063,"prompt_tokens":709,"completion_tokens":1354,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":1305}},"tokens_in":453,"tokens_out":1354,"duration_ms":10716,"temperature":1.0,"reasoning_tokens":1305,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:12:36.438651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators rate catch-n-repair transcripts from the same 40 test documents using the paper's own grounding instructions. If human-judge agreement on repaired turns is no better than chance, or if any domain shows human-rated faithfulness dropping while judge-rated faithfulness rises, the reported 0.1–0.5 gains would not establish real grounding.","supporting_citations":[],"review_version":1}