{"id":"1ab97f60-ccf6-4a0f-b4b4-4de33c64bf2a","arxiv_id":"2504.14856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"INTRALIGN trains language models to produce citations from both retrieved documents and parametric memory, with confidence scores on internal citations and improved trustworthiness metrics.","lead":"This paper defines a citation-generation task in which language models must cite either retrieved documents or their own memorized knowledge, attaching a confidence score when they cite internal knowledge. It also introduces INTRALIGN, a training pipeline, and five metrics for helpfulness, citation faithfulness, and trustworthiness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal citations are not verifiably parametric: Sec. 5.4.3 shows models rewrite external documents as internal references, so the claimed transparent internal/external distinction is unenforced and the reported trustworthiness and ECE gains conflate plagiarism with parameter knowledge.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the internal versus external distinction is not enforced or measured, and the paper's own plagiarism analysis (Sec. 5.4.3) demonstrates the failure mode. My stress-test concurs and specifies the concrete avenue: the ECE and internal-recall metrics are contaminated by plagiarized internal references, so the central transparency claim is not supported by the reported numbers. This does not invalidate the helpfulness or citation-quality contributions, but it does require the authors to either enforce the distinction (e.g., by penalizing or flagging internal references that are entailed by the context) or substantially weaken the transparency claim to 'the model can be trained to report confidence on internal-style citations,' without asserting provenance. The paper is honest about not penalizing plagiarism, which is a strength, but honesty alone does not supply the missing evidence. The reader's CONDITIONAL verdict is appropriate: the paper should be accepted only if the authors address this provenance problem or explicitly reframe the claim. I therefore do not change the verdict; I recommend keeping it CONDITIONAL with this requirement.","tokens_in":24464,"tokens_out":3461,"duration_ms":33852,"concrete_test":"On the GT, PK test set, filter the internal references using the same entailment check as Sec. 5.4.3 to identify plagiarized cases (internal reference entails the provided external documents or the golden answer derivable from them). Recompute Rcin, RcO, and ECE on the remaining non-plagiarized internal references only. If ECE worsens materially (e.g., from ~0.10 to above 0.20) or Rcin drops by more than 10 absolute points, the reported trustworthiness gains depend on the conflated plagiarism category rather than on calibrated parametric knowledge. Conversely, if metrics are stable, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that INTRALIGN enables LLMs to transparently distinguish and cite internal parametric knowledge from external context (Abstract, Contributions). This requires that 'internal reference' (Sec. 2) genuinely originate from the model's parameters and be distinct from the provided context. However, Sec. 5.4.3 defines plagiarism as an internal reference that entails the answer in the GT set, reports non-trivial plagiarism rates (PR: 0.054-0.154 across settings in Table 2), and explicitly states 'we do not penalize this behavior since high-quality rewriting would be more helpful to users.' Because internal references are model-generated strings, the task definition provides no mechanism to verify their origin: any non-extracted reference is classified as internal by default. Consequently, the reported internal citation recall (Rcin) and Internal Reference ECE (Sec. 2.1.5) are computed over a mixture of genuine parametric knowledge and plagiarized external content. The ECE measures calibration of confidence against FACTSCORE factuality, but if the 'internal' reference is a rewrite of a provided document, the confidence is calibrated on provenance, not on true parametric knowledge. The transparency claim—that users can know whether a citation reflects external or internal knowledge—is therefore not established. The paper acknowledges this limitation but does not resolve it; the evaluation framework cannot enforce or measure the distinction it requires.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, Context-Prior Augmented Citation Generation, in which LLMs must generate citations that draw on both external retrieved context and internal parametric knowledge, reporting a confidence score for each internal citation. The authors define five metrics covering answer helpfulness, citation faithfulness, and reference trustworthiness, and introduce RAEL, a generation paradigm, and INTRALIGN, an alignment pipeline that combines GPT-4o-based data generation, rejection sampling, and token-type weighted fine-tuning. Experiments on CRAG, FRAMES, and SFE across several open and closed LLMs and six baselines report consistent improvements in accuracy, citation recall, convincingness, conciseness, and internal-reference ECE. The paper also analyzes model behavior under low-quality retrieval, knowledge conflicts, and what it calls dishonest internal reference generation.","tokens_in":24805,"tokens_out":8401,"duration_ms":74455,"significance":"If the central claims hold, the paper makes a useful contribution to interpretable retrieval-augmented generation: it formalizes a citation task that explicitly includes parameter knowledge, proposes a practical alignment recipe, and introduces evaluation metrics for reference quality and confidence calibration. The authors include ablations, human validation of the automatic metrics, shortcut analysis, and an extended study of plagiarism behavior. However, the significance is conditional on two unresolved issues: the evaluation cannot verify that so-called internal references actually originate from parametric knowledge rather than from rewritten context, and the trustworthiness metrics rely on a GPT-4o-mini judge that is also involved in training-data curation, with only moderate agreement against three human raters. These issues bear directly on the paper's headline transparency claim.","major_comments":[{"comment":"The task definition classifies any non-extracted reference as an internal reference, with no mechanism to ensure it comes from parametric knowledge rather than a paraphrase or rewrite of a provided document. Section 5.4.3 reports plagiarism rates between 0.054 and 0.154 (Table 2) and explicitly states that this behavior is not penalized. Consequently, the internal citation recall Rcin and the Internal Reference ECE (Section 2.1.5, Tables 1, 3, and 4) are computed over a mixture of genuine parametric knowledge and plagiarized external content. This undermines the central claim, stated in the Abstract and Introduction, that INTRALIGN transparentizes internal versus external knowledge utilization: the evaluation framework cannot enforce or measure the distinction it requires. The paper needs either a provenance check (e.g., n-gram overlap with the provided context) or a substantially more modest interpretation of the results.","section":"Section 2 and Section 5.4.3 (Table 2)"},{"comment":"The Convincingness and Conciseness metrics are assigned by GPT-4o-mini, which agrees with a single human rater only moderately (PCC 0.53 and 0.66, Table 5) based on just three raters. The same model family was used during data curation to rerank and select training responses (Section 4.2), and the same evaluator is then used to score the final outputs. The reported improvements on these two trustworthiness metrics may therefore reflect alignment to the judge's preferences rather than genuine reference quality. The Introduction's claim of a 'strong correlation' between automatic metrics and human judgments is not supported by these numbers, and the paper should either obtain a larger and more diverse human evaluation or use an independent judge for final evaluation.","section":"Section 6 (Table 5) and Section 4.2"},{"comment":"The plagiarism rate PR is defined as the proportion of internal references that entail the golden answer among questions in the GT,PK set. Because the GT setting is defined (Section 3, Appendix D) by the absence of any retrieved document that entails the golden answer, an internal reference that entails the answer cannot be a verbatim subsequence of the provided documents. The measure therefore does not directly detect copying from external documents; it largely counts any correct parametric citation as plagiarism. The conclusion that INTRALIGN achieves a 'relatively lower plagiarism rate' is not established by Table 2, and the analysis of dishonest internal reference generation should instead measure textual overlap between internal references and the provided context, or otherwise disentangle genuine parametric knowledge from rewritten context.","section":"Section 5.4.3 (equation for PR)"}],"minor_comments":[{"comment":"The golden confidence formula uses 'Pk' as a sum over NLI checks, which is easily confused with the confidence variable P_i; please rename it to avoid notation collision.","section":"Section 4.2"},{"comment":"INTRALIGN results are missing for GPT-4o, DeepSeek-R1, and o1-mini; please explain why the alignment method was not applied to these models, since the cross-scenario claim would be stronger with those results.","section":"Table 1"},{"comment":"There are typos such as 'Convinvingness' and 'desgin'; the text should be proofread before publication.","section":"Appendix B"},{"comment":"The sentence 'our pipeline first retrieves top-100 passages from a chunked Wikipedia snapshot using a GTR retriever (Wang et al., 2021) and the question' is missing a word, and 'an NLI model (Honovich et al., 2022) is applied' is grammatically awkward; please revise.","section":"Appendix D"},{"comment":"No statistical significance tests are reported; given the standard deviations shown in Table 1, some differences may overlap across methods, so significance testing or confidence intervals would strengthen the comparison.","section":"Section 5.3"},{"comment":"The paper does not state whether code and data will be released; please add a reproducibility statement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal, but the provenance problem is central and should be resolved before publication. I would encourage the authors to add a direct check of whether internal references overlap with the provided context, and to report human evaluation with more raters and inter-annotator agreement statistics. The circularity introduced by using the same model family for data generation, reranking, and final evaluation is also a concern that should be addressed, for example by using an independent evaluation model or a larger human study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it defines a task that requires LLMs to cite both retrieved documents and their own parametric knowledge, with a confidence score on the internal citations, and it proposes a sensible alignment pipeline that improves over the baselines on its own metrics. The experimental work is thorough—multiple seeds, ablations, and analyses of retrieval quality, question difficulty, and knowledge level. That part is solid.\n\nThe problem is the central claim. The task is supposed to \"transparentize\" internal versus external knowledge utilization, but the distinction is never actually enforced or measured. Any reference that isn't a verbatim extraction from the context is classified as internal. The paper's own Section 5.4.3 shows that models rewrite external documents and present them as internal references, with plagiarism rates between 0.054 and 0.154 across settings, and the authors explicitly say they do not penalize this behavior. That means the internal citation recall and the Internal Reference ECE are computed over a mixture of genuine parametric knowledge and plagiarized context. The confidence score is calibrated against factuality, not against provenance, so the transparency story doesn't hold.\n\nThis is a significant soft spot, but not a fatal one. The task and metrics are still useful as a benchmark if the plagiarism problem is addressed—for example, by adding a plagiarism penalty or by verifying internal provenance through memory probes. The alignment method itself appears to reduce plagiarism compared to baselines (Table 2), which is an interesting result. The Convincingness and Conciseness metrics are weaker: they rely on GPT-4o-mini with Pearson correlations of 0.53 and 0.66 against just three human raters. That's moderate agreement and should be presented with more caution. No code or data is released, which limits reproducibility.\n\nWho is this for? People working on citation generation, RAG evaluation, and model interpretability. It deserves a serious peer review, but the authors need to either fix the provenance loophole or reframe the contribution as \"citation generation with confidence scoring for parametric claims\" rather than \"transparentizing internal versus external knowledge.\" I'd engage with it.","headline":"Useful new task and method, but the internal/external distinction is unenforced, so the transparency headline overreaches.","tokens_in":25295,"tokens_out":3389,"would_cite":true,"duration_ms":31316,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLMs can be aligned to cite internal parametric knowledge as transparently as external documents, with calibrated confidence scores, and that its INTRALIGN method outperforms existing citation-generation baselines…","keywords":["citation generation","retrieval-augmented generation","parametric knowledge","internal citations","confidence calibration","trustworthiness","LLM alignment","interpretability"],"falsifier":"Take a test question where the provided documents contain the full answer and the model nevertheless emits an internal citation. If that internal reference is a near-verbatim substring of a provided document (measured, say, by longest-common-subsequence overlap above 80 percent), the claim that internal citations transparentize parametric knowledge would be falsified for that model. A systematic version: measure internal-citation overlap with the provided documents across the GT, PK set and check whether confidence scores still calibrate after removing all such overlapping cases.","tokens_in":24293,"feed_emoji":"📚","tokens_out":4432,"duration_ms":36510,"temperature":0.7,"pith_summary":"The paper argues that current citation-generation systems treat all cited information as coming from retrieved documents, leaving the model's use of its own memorized knowledge opaque. It proposes a new task, Context-Prior Augmented Citation Generation, in which the model must cite either an external span or an internally generated reference paired with a confidence score, and it introduces INTRALIGN, a data-generation and alignment method that teaches open-source LLMs to do this faithfully. The reported experiments indicate that INTRALIGN improves answer accuracy, citation recall, reference convincingness and conciseness, and confidence calibration relative to six baselines. If the claim holds, LLM answers could transparently separate “this comes from the documents” from “this comes from what I know,” with a number the user can check.","feed_headline":"Internal knowledge can be cited as transparently as documents","feed_subtitle":"A new task and alignment method separate “from the documents” from “from what I know” and calibrate the confidence score.","key_machinery":"The load-bearing mechanism is the RAEL (Rational Attribution and Elaboration) generation paradigm combined with INTRALIGN's token-weighted alignment loss. RAEL forces the model to first write a context review and a parameter-knowledge scrutiny, then produce extractive external references and recited internal references with confidence scores, making knowledge use explicit in the output stream. The alignment loss assigns type-dependent weights so that reference tokens, confidence tokens, and citation markers receive more gradient signal than ordinary answer tokens. The paper also uses a self-consistency-inspired golden confidence formula — the fraction of sampled documents that entail the golden answer — to supervise confidence scores, and it computes ECE against FACTSCORE to measure calibration.","core_discovery":"The central claim is that an LLM can be aligned to cite parameter knowledge as transparently as it cites external evidence, and that the resulting internal citations carry confidence scores that track their factual accuracy. The paper defines the Context-Prior Augmented Citation Generation task, where each cited segment is either an extractive span from the provided documents or a model-generated reference paired with a probability, and a refusal is expected when neither source answers the question. INTRALIGN builds a training set by sampling documents and direct answers from the target model itself, checking them with an NLI model, and generating RAEL-style gold responses with GPT-4o; the alignment loss then up-weights reference, confidence, and citation-marker tokens. Across Llama-3.1-8B, Llama-3.1-70B, and GPT-4o, the method reports higher accuracy, higher internal citation recall, and lower ECE than the baselines, with ablations attributing the gains to the RAEL paradigm and the weighted loss.","pith_inferences":["A stricter operationalization of “internal reference” would require the paraphrase distance between internal references and provided documents to exceed a threshold, or would penalize verbatim overlap; the paper's own plagiarism analysis shows this distinction is currently unenforced.","The NLI-checked self-consistency confidence formula could generalize beyond citation: the same method might calibrate chain-of-thought claims or tool-use outputs in other retrieval-augmented settings.","The ECE metric as defined checks whether confidence tracks the FACTSCORE of the reference, but not whether the confidence explains why the user should trust the span; a testable extension is to compare two systems with equal ECE but different confidence distributions.","The open-source models benefited most from INTRALIGN, suggesting alignment may substitute for scale in citation trustworthiness; a concrete test is to run the same pipeline on a model with different pretraining data and see whether internal-citation calibration shifts."],"forward_implications":["If INTRALIGN works as claimed, LLM answers can distinguish “stated in the retrieved documents” from “stated from the model's own knowledge” in the citation markup itself.","Confidence scores on internal citations become a checkable quantity: users can spot low-confidence parametric claims instead of treating all claims equally.","The task's refusal behavior gives a principled way to abstain when both retrieval and parametric knowledge fail, reducing hallucination in low-retrieval scenarios.","The paper's finding that internal citation recall is generally higher than external recall suggests models are more faithful when reciting their own knowledge than when quoting retrieved text, which bears on how RAG systems should mix sources.","The higher plagiarism rate of larger models indicates that alignment for convincingness may push models to rewrite external evidence as internal knowledge, a behavior the task currently does not penalize."],"supporting_citations":[{"why":"Defines the ALCE citation-generation paradigm and the NLI-based recall metric that the paper extends to internal references.","marker":"(Gao et al., 2023b)"},{"why":"Recitation-Augmented Generation, the baseline that samples passages from parameter knowledge and motivates the internal-citation design.","marker":"(Sun et al., 2023)"},{"why":"FACTSCORE, used to determine the factuality of internal references for the Expected Calibration Error computation.","marker":"(Min et al., 2023)"},{"why":"Self-consistency uncertainty measurement, the inspiration for the golden-confidence formula based on sampled documents.","marker":"(Wang et al., 2023)"},{"why":"GPT-4o, used for data generation in INTRALIGN and as one of the evaluated models and evaluators.","marker":"(OpenAI, 2024)"},{"why":"TRUE, the NLI model used for document annotation, answer-accuracy checks, and citation-recall checks.","marker":"(Honovich et al., 2022)"},{"why":"CRAG, one of the three datasets combined to construct the task's evaluation set.","marker":"(Yang et al., 2024)"},{"why":"FRAMES, another source dataset providing multi-document reasoning questions.","marker":"(Krishna et al., 2024)"},{"why":"SituatedFaithfulnessEval, the third source dataset contributing situated-faithfulness questions.","marker":"(Huang et al., 2024c)"}],"fun_headline_variants":["LLMs cite internal knowledge with calibrated confidence","New task and alignment make LLM citations trustworthy","Transparent internal citations via INTRALIGN alignment","Calibrated confidence for LLM internal knowledge citations","Aligning LLMs to cite both sources and own knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that when the model writes an “internal” reference it is genuinely drawing on parametric knowledge rather than quietly rewriting the documents it was just shown; the paper's own Section 5.4.3 shows this happens, and it does not penalize it.","fun_headline_variants_meta":{"raw":{"variants":["LLMs cite internal knowledge with calibrated confidence","New task and alignment make LLM citations trustworthy","Transparent internal citations via INTRALIGN alignment","Calibrated confidence for LLM internal knowledge citations","Aligning LLMs to cite both sources and own knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1493,"prompt_tokens":898,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":514,"tokens_out":595,"duration_ms":5740,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:38:48.521105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test question where the provided documents contain the full answer and the model nevertheless emits an internal citation. If that internal reference is a near-verbatim substring of a provided document (measured, say, by longest-common-subsequence overlap above 80 percent), the claim that internal citations transparentize parametric knowledge would be falsified for that model. A systematic version: measure internal-citation overlap with the provided documents across the GT, PK set and check whether confidence scores still calibrate after removing all such overlapping cases.","supporting_citations":[],"review_version":1}