{"id":"50d92459-85c3-4b33-99dd-26015040eb26","arxiv_id":"2506.14580","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GenerationPrograms, a program-then-execute generation framework, substantially improves document- and sentence-level attribution quality in long-form QA and multi-document summarization compared with direct citation generation.","lead":"This paper introduces GenerationPrograms, a two-stage framework that first writes an executable program of text operations and then runs it to generate answers, making every output sentence carry citations to the source sentences it was built from. The approach reports large gains in fine-grained attribution accuracy over direct-citation baselines on QA and summarization tasks, at the cost of some answer-quality as measured by traditional metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution gains may partly reflect NLI-judge leniency toward source-overlapping GenerationPrograms outputs; the 50-example human check does not cover LFQA/MDS.","rationale":"The reader's weakest assumption was that NLI-based attribution metrics are a valid oracle, citing the small human validation (50 examples, 78.1%) and small LFQA test set. I agree that the oracle validity is the most load-bearing assumption, but I sharpen the concern in a specific way: GenerationPrograms' modular outputs are constructed by prompting GPT-4o to paraphrase, compress, or fuse the very sentences that will later be cited, producing high lexical overlap with the cited sources. An entailment judge with a lexical-overlap prior will therefore favor GenerationPrograms independent of whether the method provides genuinely better support. Additionally, the only reported human validation does not cover the GPT-4o-judged LFQA and MDS settings, which are also the settings with the largest claimed gains relative to ALCE. This is not a claim of intentional misconduct; it is a plausible measurement confound that the paper does not fully control for. The proposed concrete test is a human-annotation study on the GPT-4o-judged datasets, which would directly establish whether the judge bias changes the method ranking. I would keep the reader's CONDITIONAL verdict: the method and its qualitative contributions are credible, but the quantitative magnitude of the attribution improvement should not be accepted until the judge validity is demonstrated on LFQA and MDS.","tokens_in":24904,"tokens_out":3994,"duration_ms":46295,"concrete_test":"Sample 100 LFQA and 100 MDS attribution decisions, stratified by method and sentence, and have two independent annotators judge entailment against the cited source sentences using the same binary protocol as Appendix C.8. Recompute method-level attribution precision, recall, and F1 from the human labels. If the human-judged gap between GenerationPrograms and ALCE is materially smaller than the GPT-4o-judged gap, the headline improvement is partly an artifact of the NLI judge; if the gap persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on model-based attribution metrics: AutoAIS for ASQA and GPT-4o NLI for LFQA and MDS (§3.2). GenerationPrograms executes paraphrase/compression/fusion modules directly on source sentences, so its outputs retain more source wording than ALCE's directly generated citations. NLI judges are known to be sensitive to lexical overlap, and a judge that rewards such overlap would inflate GenerationPrograms' attribution F1 even when the underlying support relation is no better than ALCE's. The only human validation of the attribution metric (Appendix C.8) is 50 binary examples at 78.1% agreement and appears to be conducted on the ASQA/AutoAIS setting; no human validation is reported for GPT-4o on LFQA or MDS. In those settings the evaluator is the same model family used to generate the program and execute the modules, so the possibility of a self-consistency bias is not excluded. Since the LFQA improvement is 39.0 document-level F1 points and 28.8 sentence-level F1 points, while MDS improves by 30.6 document-level points, a moderate judge bias could account for a substantial fraction of the headline gain. This is a measurement concern rather than an internal inconsistency, but it directly affects whether the claimed superiority over direct-citation baselines generalizes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenerationPrograms, a two-stage framework for attributed long-form generation: a planner LLM first produces an executable program composed of modular text operations (paraphrase, compression, fusion, extract) over retrieved source sentences, and the program is then executed to produce an answer whose sentence-level citations are automatically the module input sentences. The authors evaluate the framework on ASQA, LFQA, and MultiNews MDS, reporting large attribution F1 improvements over the ALCE baseline at both document and sentence level (e.g., document-level F1 of 87.1 vs 62.7 on ASQA and 94.4 vs 55.4 on LFQA in Table 1). They also present a post-hoc attribution experiment on LFQA using human gold sentence annotations, and a module-level refinement study that reranks module outputs. The paper additionally reports answer-correctness trade-offs with EM/ROUGE-L, which they argue are reduced when evaluated with an LLM-based correctness metric or human annotation.","tokens_in":25089,"tokens_out":6682,"duration_ms":62179,"significance":"If the headline attribution gains hold, the paper makes a strong practical contribution: it offers an interpretable, executable program trace that provides contributive rather than merely corroborative attribution, and the framework naturally supports localized refinement at low computational cost. The post-hoc attribution experiment is evaluated against human-annotated sentence IDs rather than an NLI judge, which is a notable strength, and the authors provide an open-source implementation, deterministic decoding settings, a replication with an open-source model, and a human evaluation of answer correctness. These elements support the credibility of the core mechanism. The main risks are the reliance on model-based NLI judges for the central comparisons and the small LFQA test set, which the major comments address.","major_comments":[{"comment":"The central attribution comparisons are evaluated with model-based NLI judges (AutoAIS for ASQA, GPT-4o for LFQA and MDS), but the only human validation of an NLI judge is 50 ASQA/AutoAIS examples with 78.1% agreement; no human validation is reported for the GPT-4o judge on LFQA or MDS, and the LFQA test set contains only 45 examples (Table 4), while the MDS test-set size is not stated. Because GenerationPrograms executes paraphrase/compression/fusion directly on source sentences, its outputs likely have higher lexical overlap with the source than ALCE's free-form generations, and an NLI judge sensitive to lexical overlap could inflate GenerationPrograms' F1 without a genuine improvement in support. The authors should report human agreement with the GPT-4o judge on a stratified sample from LFQA and MDS, provide a lexical-overlap or judge-bias analysis, or conduct a human re-evaluation of attributions on a subset of all three datasets, together with error bars.","section":"§3.2, Table 4, Appendix C.8"},{"comment":"The module-level refinement experiment uses AutoAIS both to select the 'entailed' candidate among five sampled module outputs and to compute the reported attribution F1, so the refinement procedure directly optimizes the metric on which the +4.0 (ASQA) and +2.7 (LFQA) F1 improvements are reported; the same coupling applies to the ALCE+reranking comparison. This selection-evaluation overlap can inflate the apparent refinement gains. The authors should evaluate refined outputs with an independent metric or human annotations rather than the same AutoAIS judge used for selection, or clearly justify why the coupling does not affect the comparison.","section":"§4.3, Table 3"},{"comment":"No confidence intervals or significance tests are reported anywhere in the paper, while the text repeatedly uses 'significantly' to describe improvements. Given that the LFQA evaluation is based on only 45 examples (Table 4) and the MDS sample size is not stated, a few examples could shift F1 by several points; the paper should report bootstrap confidence intervals or significance tests for the main comparisons, and should state the MDS test-set size.","section":"Tables 1, 2, 3, 5, 7, 8; Table 4"},{"comment":"The ALCE baseline is prompted to cite at most three documents per sentence and to cite only a minimum sufficient subset (Figure 5), whereas GenerationPrograms cites every source sentence consumed by the modules with no explicit upper bound. This asymmetry in the citation instruction means part of the large recall/F1 gap may reflect a difference in citation policy rather than attribution quality. The authors should either adopt a comparable citation policy for ALCE (e.g., an ALCE variant without the three-citation cap) or report a citation-budget-controlled analysis.","section":"Figure 5 vs §2.2/Table 1"}],"minor_comments":[{"comment":"The table header labels the MDS columns as 'Correct Attribution (AutoAIS)', but §3.2 states that GPT-4o is used for LFQA and MDS; the caption or header should be corrected.","section":"Table 1"},{"comment":"Section 4.1 reports document-level F1 improvements of 20.4% on ASQA and 39.0% on LFQA, but Table 1 shows absolute differences of 24.4 and 39.0 points; the abstract and introduction report gains of 31.6% and 27%, which are inconsistent with both §4.1 and Table 1.","section":"§4.1 and Abstract/Introduction"},{"comment":"The name 'GENERATIONPROGRAMS' appears without the space used elsewhere in the paper; please make the name consistent.","section":"Table 3"},{"comment":"The text states that extractive summarization enhances accuracy with Llama 3.3 70B, but Table 8 shows ASQA correctness actually decreases from 46.1 to 45.0 for GENPROG and from 51.8 to 48.7 for ALCE; the claim should be qualified to LFQA or to the average.","section":"Appendix C.4"},{"comment":"The table omits statistics for the multi-document summarization dataset even though MDS results are part of the central claims; please include the number of examples, average source tokens, and average sentences for that dataset.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a genuinely useful idea, and the post-hoc attribution experiment using human gold annotations is a strong element. The main risk is that the headline attribution gains rest on model-based NLI judges and very small test sets, and the refinement experiment has a selection-evaluation coupling. If the authors can provide the requested human validation, an ALCE-without-cap control, and confidence intervals, I would be comfortable with the claims. No concerns about novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know about this paper: the core design is genuinely useful. GenerationPrograms decomposes RAG answers into a program of modular text operations (paraphrase, compression, fusion) with explicit instructions, then executes it, so each output sentence is traceable to its source sentences by construction. That's a real step beyond the direct-prompt-with-citations approach, and it's an honest extension of SummarizationPrograms to long-form QA and multi-document summarization. The qualitative examples look right, and the code is public.\n\nWhat's new: applying program-then-execute to attribution-heavy tasks, adding instruction-conditioned modules, showing it can be used post hoc, and showing that the program trace enables targeted, cheap refinement. The post-hoc attribution result is interesting even if the baselines are not the strongest available. The modular refinement is a nice bonus, not the main event.\n\nWhere I'd push back: the quantitative claims rest on model-based NLI judges. The authors validate AutoAIS on 50 ASQA examples and get 78.1% agreement, which is reasonable but thin, and they never validate GPT-4o as a judge on LFQA or MDS. Since GenerationPrograms outputs share more surface wording with the sources (because modules operate on the source sentences), an NLI judge that rewards lexical overlap could inflate the attribution F1 advantage even if the true support relation is only modestly better. That's a real measurement concern, not a fatal one. The LFQA test set has 45 examples, so a 39-point document-level F1 gain is fragile. No error bars anywhere. The weaknesses don't undercut the qualitative claim—the mechanism is sound, and the Llama-3.3 results show the trend holds with a different generator—but they should temper how the numbers are read and reported.\n\nAlso, for the 'post-hoc attribution' table, they compare against Semantic Similarity and a simple LLM prompt, and they don't include ContextCite, which is the obvious state-of-the-art comparison they already cite. That should be fixed. And in the refinement section, the same AutoAIS metric is used both to select refinements and to evaluate them; that's a circularity, but it's confined to that one experiment and the gains are small.\n\nBottom line: this is a solid, well-engineered paper that deserves a serious referee. The method is a real contribution, the code is out, and the central mechanism is transparent. It needs a stronger evaluation section—error bars, larger or at least acknowledged test sets, a human-validated judge for the non-ASQA settings, and stronger baselines—before the magnitude of the gains should be taken at face value. I'd engage with it, and I'd probably cite it for the program-trace idea.\n\nRecommendation: send to reviewers, with a request that they focus on the metric validity and baseline choices.","headline":"A credible program-then-execute framework for attribution that likely generalizes, but the headline numbers rest on thin metric validation and small test sets.","tokens_in":25724,"tokens_out":3799,"would_cite":true,"duration_ms":33568,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GENERATION PROGRAMS claims that splitting LLM generation into an executable program plan and a modular execution stage yields accurate, exhaustive, and interpretable citations, and reports large attribution-F1 gains over direct-citation…","keywords":["attributed generation","fine-grained citation","executable programs","modular text operations","long-form question answering","multi-document summarization","contributive attribution","post-hoc attribution"],"falsifier":"Run a human adjudication on a larger sample—say 150 outputs per task from both ALCE and GenerationPrograms, with two independent annotators judging, for each sentence, whether the cited sources entail it—and compare human-judged attribution F1. If the gap between methods shrinks to a few points or reverses on LFQA or MDS, the headline improvement is an artifact of the NLI oracle rather than a real gain. A second check: on nested programs, verify locally that each fusion/paraphrase output is entailed by its inputs; if even a few percent of module outputs break entailment, the transitive attribution chain the method relies on fails for those sentences.","tokens_in":24610,"feed_emoji":"🧩","tokens_out":8839,"duration_ms":76809,"temperature":0.7,"pith_summary":"The paper claims that LLMs can get much better at citing their sources if they generate text differently: first write an executable 'program' that selects source sentences and specifies text operations (paraphrase, compression, fusion, extraction), then run that program to produce the answer. Because each operation records which sentences it consumed, every output sentence carries an exact citation to its inputs, and the program itself explains why the model wrote what it wrote. Across two long-form question-answering tasks and a multi-document summarization task, the method reports large attribution-F1 gains over a strong generate-with-citations baseline—up to 39 points at document level and 29 at sentence level—while reducing missing citations to zero. The same program structure also serves as a post-hoc attribution tool and enables module-level refinement that further improves attribution quality, at a small cost in standard answer-correctness metrics that the paper argues is mostly stylistic.","feed_headline":"Plan-then-execute generation lifts citation F1 by up to 39 points","feed_subtitle":"Every answer sentence is built from explicit text operations on cited sources, so missing citations fall to zero.","key_machinery":"The executable program itself is the load-bearing object: a collection of trees, one per output sentence, whose leaves are sentences extracted from source documents, whose internal nodes are outputs of neural modules, and whose root is the final generated sentence. Edges are module applications carrying optional natural-language instructions (for example, 'Highlight the current trend of robots replacing humans'). The mechanism that guarantees attribution is the principle of local attribution: every module's output must be entailed by its inputs, so that the program trace is transitive—a claim can be verified by checking each step against its named sources. Because the trace records exactly which sentences fed each operation, document- and sentence-level citations are read off the program rather than generated heuristically, and the program doubles as an explicit explanation of the model's reasoning.","core_discovery":"GENERATION PROGRAMS claims that the central obstacle to verifiable LLM output is not a lack of citation skill but a collapsed process: generating text and citations at once conflates deciding what to say with tracking where it came from. The paper's solution is to separate planning from execution. In the planning stage, a language model reads the query and retrieved documents and writes a Python-style program built from modular text operations, where each step names the exact source sentences it operates on and an optional instruction describing what to emphasize. In the execution stage, dedicated neural modules (paraphrase, compression, fusion, extract) apply those operations; the output is the program's result, and the citations are the operation inputs, recovered directly from the program trace. The paper reports that this design raises attribution F1 on ASQA from 62.7 to 87.1 at document level and from 54.2 to 79.4 at sentence level; on LFQA from 55.4 to 94.4 and from 54.0 to 82.8; and on a MultiNews-based MDS task from 63.8 to 94.4 document-level and 55.2 to 90.0 sentence-level, while eliminating sentences without citations entirely. It further shows the program trace can reconstruct attributions for already-written text in a post-hoc setting, beating a prompting baseline by a large margin, and that checking each module's output for entailment of its inputs, then reranking only the failing steps, adds further attribution gains at low latency.","pith_inferences":["The program trace is an inverse map from output to context, so it could be used to prune retrieved documents: removing sources that never appear in any tree and re-generating should preserve the answer while cutting context length, a use the paper mentions only as future direction.","The reported correctness gap between ALCE and GenerationPrograms (e.g., ROUGE-L 39.9 vs 32.3 on LFQA) may be largely a length and style artifact: the paper's own LLM-based correctness judge and a 25-example human rating found the methods close or reversed, which suggests standard metrics understate the method's answer quality.","If local attribution holds reliably across modules, the same tree structure could support adversarial-robustness checks—detecting when a small change to a cited source alters a downstream output sentence—since every node names its upstream sources.","A testable extension: applying GenerationPrograms to tasks with finer granularity (phrase-level or paragraph-level operations, with pointers to sentences) should preserve attribution fidelity; the paper asserts this extension is straightforward but does not evaluate it."],"forward_implications":["On all three tasks, GenerationPrograms eliminates sentences with no citation, while the ALCE baseline leaves 25.8% (ASQA docs), 40% (LFQA docs), and 25.4% (MDS docs) of sentences uncited.","The program trace provides contributive, not just corroborative, attributions: it shows which source sentences were actually used and how they were transformed, supporting per-step verification by a reader.","Because the program can be regenerated from an existing output, attribution can be added post hoc to black-box generations without access to model internals or logits, and the reconstructed output preserves content (ROUGE-L 50.7 on LFQA).","Module-level entailment checking with reranking of only failing steps raises attribution F1 by 4.0 points on ASQA and 2.7 on LFQA, at an added latency of about 1.8 seconds per example, whereas reranking entire ALCE outputs costs roughly 8x runtime.","Combining program-based generation with extractive summarization of retrieved sources balances attribution quality against answer correctness, mitigating the correctness drop the method otherwise shows on style-sensitive metrics."],"supporting_citations":[{"why":"It supplies the program-tree formalism and the modular text operations that GenerationPrograms repurposes for attributed generation.","marker":"Saha et al. (2023)"},{"why":"It provides the ALCE baseline that generates citations alongside text, the AutoAIS attribution metric, the ASQA passages, and the summarization-based context filtering the paper builds on.","marker":"Gao et al. (2023b)"},{"why":"It recommends GPT-4o NLI evaluation for attribution quality, supplies the LLM-based correctness judge, and defines the human-validation setup the paper adapts.","marker":"Zhang et al. (2024a)"},{"why":"It supplies the 'Attribute First' two-stage baseline, the curated LFQA evaluation split, and the comparison point for plan-based generation.","marker":"Slobodkin et al. (2024)"},{"why":"It provides the ASQA long-form QA dataset with gold answers used for one of the two QA evaluations.","marker":"Stelmakh et al. (2022)"},{"why":"It provides the LFQA dataset and its human sentence-level citation annotations used for post-hoc attribution evaluation.","marker":"Liu et al. (2023a)"},{"why":"It provides the processed MultiNews-based multi-document summarization dataset used for the MDS evaluation.","marker":"Ernst et al. (2024)"},{"why":"It supplies TRUE, the T5-11B NLI model behind AutoAIS for ASQA attribution scoring.","marker":"Honovich et al. (2022)"},{"why":"It defines contributive versus corroborative attribution, the conceptual distinction that motivates the program trace as an explanation mechanism.","marker":"Cohen-Wang et al. (2024)"}],"fun_headline_variants":["Plan-then-execute generation hits 94.4 citation F1","Programmed text operations eliminate missing citations","Plan first, execute later: citation F1 improves","Program traces beat post-hoc attribution methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the NLI-based oracle (AutoAIS/TRUE for ASQA, GPT-4o for LFQA and MDS) correctly decides whether a cited source supports a generated sentence—validated on only 50 examples at 78.1% agreement—and that the small test sets, notably 45 LFQA examples, represent the tasks.","fun_headline_variants_meta":{"raw":{"variants":["Plan-then-execute generation hits 94.4 citation F1","Programmed text operations eliminate missing citations","Plan first, execute later: citation F1 improves","Program traces beat post-hoc attribution methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000963,"raw_usage":{"total_tokens":4173,"prompt_tokens":1089,"completion_tokens":3084,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":3023}},"tokens_in":705,"tokens_out":3084,"duration_ms":23493,"temperature":1.0,"reasoning_tokens":3023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:51:00.345682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human adjudication on a larger sample—say 150 outputs per task from both ALCE and GenerationPrograms, with two independent annotators judging, for each sentence, whether the cited sources entail it—and compare human-judged attribution F1. If the gap between methods shrinks to a few points or reverses on LFQA or MDS, the headline improvement is an artifact of the NLI oracle rather than a real gain. A second check: on nested programs, verify locally that each fusion/paraphrase output is entailed by its inputs; if even a few percent of module outputs break entailment, the transitive attribution chain the method relies on fails for those sentences.","supporting_citations":[],"review_version":1}