{"id":"ef2c1b2e-66d2-4966-9729-2e27be452a3e","arxiv_id":"2412.14581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CORD improves RAG fine-tuning by adding consistency distillation between original and adaptively perturbed passage orders, balancing position robustness with retriever rank information.","lead":"CORD is a fine-tuning method that makes language models use all retrieved passages evenly while still respecting the retriever ranking. It improves question-answering accuracy on five retrieval-augmented benchmarks by balancing consistency training with rank-aware distillation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the distillation term: CORD's gains over IN2 may stem solely from adding the original-order NLL, not from adaptive rank distillation.","rationale":"The reader's weakest_assumption centers on the self-referential teacher-selection rule and score-gap heuristic. I agree those are risks, but the more immediate, concrete gap is the missing isolation of the distillation objective itself. The paper's headline contribution is the JSD-based distillation with an adaptive teacher; without a λ=0 baseline, the empirical support for that contribution is absent. This is a correctness-risk issue, not just a robustness concern: it determines whether the proposed method as described is actually what produces the gains. The paper does have strengths: it identifies a real tension between consistency and rank preservation, proposes an interpolation space for controlled perturbations, and provides a plausible post-hoc analysis in Figure 4. However, the main table's comparisons are confounded. The required ablation is simple and would settle the matter; until then, the conditional verdict is appropriate. I keep the reader's CONDITIONAL verdict unchanged, with the added condition that the λ=0 ablation be reported on all benchmarks.","tokens_in":7448,"tokens_out":6929,"duration_ms":48762,"concrete_test":"Train CORD with λ=0 (Eq. 4), i.e., NLL on {c, c1} with no JSD, using the same hyperparameters, α=0.5, Phi-3 3B, and all five benchmarks (MS MARCO, HotpotQA, NQ, MN, MN-IDK). Compare against CORD and Lnll on C1 in Table 2. If the λ=0 results match CORD within noise, the distillation component is not load-bearing and the paper must be reframed. Additionally, run CORD with a fixed c1_α teacher (no p-based selection) to test whether the adaptive teacher-selection rule contributes anything beyond fixed interpolation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CORD's controlled distillation (JSD plus adaptive teacher selection) balances consistency and rank distillation, outperforming IN2-style training. But Table 2 compares CORD only against Lnll on C1 (random perturbation only) and Lnll on C (original order only). The method actually differs from Lnll on C1 in two stacked ways: (1) it adds NLL on the original order c, and (2) it adds the JSD distillation term (Eq. 3). The paper never reports the λ=0 ablation, i.e., training NLL on both c and c1 with no distillation. Table 3 shows CORD outperforms Lnll on C on MS MARCO, but that could come entirely from the added c1 term or from the added c term. Table 4 only ablates adaptive α on MN/MN-IDK, not the JSD term itself. On MS MARCO, CORD's improvement over Lnll on C1 is +0.22 R-L and 0.00 GPT-4, consistent with the distillation term being nearly inert there; if the missing baseline matches CORD everywhere, the novelty reduces to data augmentation with the original order, not a new balancing mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CORD, a fine-tuning objective for RAG generators that combines (i) NLL training on both the retriever-given passage order c and a perturbed order c1, (ii) a Jensen-Shannon divergence consistency term between the two orders, and (iii) an adaptive perturbation mechanism that interpolates between c and c1 and selects the distillation teacher with higher model-assigned probability of the gold answer. The authors evaluate on MS MARCO, HotpotQA, NQ, and synthetic multi-needle (MN/MN-IDK) scenarios using Phi-3 3B with LoRA, reporting improvements over no finetuning and over an IN2-style baseline trained only on perturbed contexts, plus an additional gain from score-aware selection of the perturbation degree.","tokens_in":7695,"tokens_out":7907,"duration_ms":53256,"significance":"If the empirical claims are fully supported, CORD is a useful and inexpensive recipe for making RAG generators less sensitive to passage order without discarding retriever rank information. The method is clearly specified (Eq. 4), the interpolation space in Sec. 2.2 is a simple and natural extension of prior position-perturbation work, and the evaluation spans both rank-sensitive and rank-insensitive settings. The paper also reports a beneficial side effect: score-aware alpha selection improves MN/MN-IDK. The main reservation is that the current experiments do not isolate the distillation term from the added original-order NLL, so the named mechanism ('rank distillation') is not yet supported by the reported ablations.","major_comments":[{"comment":"The reported experiments do not isolate the effect of the distillation term. CORD optimizes L = Ln + lambda*Ld with Ln applied to both c and c1, while the baseline 'Lnll on C1' applies Ln to c1 only. Therefore CORD differs from that baseline in two ways: it adds original-order NLL on c, and it adds JSD distillation. The paper never reports the lambda=0 baseline (NLL on both c and c1 without JSD). On MS MARCO, CORD improves over Lnll on C1 by only +0.22 R-L and 0.00 GPT-4, which is consistent with the distillation term being nearly inert there. The central claim that adaptive rank distillation is responsible for the gains requires this missing ablation.","section":"Section 3.2, Tables 2 and 3, Eq. (4)"},{"comment":"The motivating numbers in Table 1 are inconsistent with the main results. For scenario A (MS MARCO R-L), '+ consistency' is reported as 36.87, but Table 2 reports the corresponding IN2-style 'Lnll on C1' value as 44.52. For scenario B (MN F1), '+ consistency' is 57.87, while Table 2 reports 56.25 for Lnll on C1. The authors should reconcile these values or clarify what '+ consistency' denotes; as written, the motivational claim that consistency regularization hurts in scenario A is contradicted by Table 2.","section":"Section 1, Table 1 vs Section 3.2, Table 2"},{"comment":"The adaptive teacher-selection rule chooses between c1_alpha and c1 based on the probability p(y_hat | x, .) assigned by the model being trained. This makes the distillation target a function of the model's current beliefs, but the paper gives no evidence that a higher self-assigned p(y_hat) identifies a better teacher; it could instead reinforce the model's existing ordering bias. At minimum, the authors should compare this selection rule against a random selection between c1_alpha and c1, or against always using c1_alpha, to show that the selection criterion itself matters.","section":"Section 2.2, Figure 3"},{"comment":"Score-aware alpha sampling is evaluated only on MN and MN-IDK. No results are shown for MS MARCO, HotpotQA, or NQ, where retriever scores and rank priors differ, and no comparison against fixed alpha=0.5 is provided for those benchmarks. The paper's broader claim that score-aware sampling improves diverse RAG scenarios is therefore not supported by the data.","section":"Section 2.3, Table 4"},{"comment":"All results appear to come from a single run with no variance estimates or significance tests, while several reported differences are small (e.g., +0.22 R-L on MS MARCO). The phrase 'consistently outperforms' should be tempered or supported by seed/CI information, especially given the discrepancy in Table 1.","section":"Section 3.2, Tables 2-4"}],"minor_comments":[{"comment":"There is a typo in 'connsistency' in the contribution summary; it should be 'consistency'.","section":"Section 1, contribution list"},{"comment":"Sampling alpha=0.5 is described as following the principle of maximum entropy, but a uniform prior over alpha in (0,1) is not the same as fixing alpha=0.5; the wording should be corrected.","section":"Section 2.3"},{"comment":"The sentence 'we assume a desirable single value of alpha for the given task is known a priori, and later discuss how to find it' is confusing because Appendix B reports fixed alpha=0.5 for all experiments; clarify which experiments use fixed alpha and which use score-aware alpha.","section":"Section 2.2 and Appendix B"},{"comment":"The axes of Figure 3 are not labeled; the text refers to the y-axis as probability of the ground-truth answer, but this should be shown explicitly, and c1_alpha should be defined in the caption.","section":"Figure 3"},{"comment":"The baseline 'Lnll on C1' should be described explicitly as a reimplementation of IN2-style training, and any differences from An et al. (2024) should be stated (e.g., number of training examples, position perturbation procedure).","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the proposed method is simple to implement. The main risk is that the contribution reduces to data augmentation with the original order if the missing lambda=0 ablation does not show a distillation benefit. I would condition acceptance on the authors providing that ablation and reconciling Table 1. The evaluation scope (one 3B model, one retriever for MN, no code release) may also be worth noting to the authors, though it is not a blocker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper proposes CORD, a RAG fine-tuning objective that augments the retriever-given order with a controlled perturbation, adds a JSD distillation term, and adaptively picks which perturbation to distill from. It's a clean, plausible recipe, and the empirical gains on five benchmarks are believable. But the paper never runs the ablation that would isolate the distillation term. That omission is load-bearing.\n\nWhat's genuinely new: the interpolation space over perturbation levels (α), the score-gap heuristic for choosing α per instance, and the practice of training on both the original order and a perturbed order rather than only the perturbed one, as IN2 does. The paper also gives a clear demonstration, in Table 1, that pure consistency training can hurt when the retriever's order carries real information. That framing is useful. The MN-IDK split is a nice addition.\n\nSoft spots. The stress-test note is correct. CORD differs from Lnll on C1 in two stacked ways: it adds NLL on the original order c, and it adds the JSD term. The paper compares against Lnll on C1 and Lnll on C, but never against the λ=0 baseline, i.e., training on both c and c1 with no distillation. Without that baseline, the central claim that balancing consistency and rank distillation drives the gains is not supported. On MS MARCO the gains over Lnll on C1 are about 0.2 R-L and 0 GPT-4, so the distillation term may be nearly inert there. Table 4 ablates adaptive α only on MN, not the JSD term itself.\n\nThe evaluation is also under-powered: one base model, one run per benchmark, no seeds, no significance tests, no code or data release. The teacher-selection rule depends on the model's own probability estimate, which is self-referential; as a heuristic it could work, but the paper needs to show robustness to miscalibration.\n\nWhat holds up: the method is clearly described, the objective in Eq. 4 is well defined, and the qualitative story behind Figure 4 is coherent. The paper is honest about limitations.\n\nFor whom: people working on RAG training and long-context LLM fine-tuning. It deserves a serious referee, but the referee should ask for the missing ablation, multiple seeds, and ideally code. I'd recommend major revision, not acceptance as-is.","headline":"CORD is a clean, plausible RAG fine-tuning recipe, but the missing λ=0 ablation leaves its central balancing claim unsupported.","tokens_in":8215,"tokens_out":2667,"would_cite":false,"duration_ms":19096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CORD makes retrieval-augmented generation resistant to context order by distilling from adaptively perturbed rankings rather than fixed random ones.","keywords":["retrieval-augmented generation","position bias","consistency regularization","knowledge distillation","rank distillation","context perturbation","LLM fine-tuning","retriever rank prior"],"falsifier":"Re-run the MS MARCO experiment with a deliberately miscalibrated base model, for instance one whose confidence stays high on random orderings, and compare CORD to fixed full perturbation; if adaptive teacher selection then performs no better than, or worse than, fixed randomization, the teacher-selection premise is falsified. Alternatively, an oracle experiment that replaces the model's confidence with the true probability of correct generation for each perturbation would show whether the adaptive mechanism is necessary, since oracle selection yielding no gain over fixed alpha=0.5 would indicate the heuristic is not carrying the result.","tokens_in":7255,"feed_emoji":"⚖️","tokens_out":6872,"duration_ms":46688,"temperature":0.7,"pith_summary":"The paper proposes a fine-tuning method, CORD, for retrieval-augmented generation (RAG) models. Its central claim is that a generator can be made less dependent on the order of retrieved passages—so that it does not over-rely on the first and last positions—while still respecting the retriever's ranking when that ranking carries useful information. The method pairs each training example with a position-perturbed version, regularizes the model to give consistent outputs on the pair, and distills between the two; but instead of always using a fully random perturbation, it samples an intermediate perturbation whose noise level adapts to the scenario. Across five RAG benchmarks, including a strong-rank-prior setting (MS MARCO) and a position-bias-sensitive multi-needle setting, CORD outperforms the fixed-perturbation baseline. If correct, this is a general recipe for making RAG systems order-stable and rank-aware from a small fine-tuning budget.","feed_headline":"CORD fine-tuning beats fixed perturbation across five RAG tasks","feed_subtitle":"Adaptively perturbing context order during training keeps retriever signals while addressing position bias.","key_machinery":"The central object is the interpolated perturbation space c1_alpha, a one-parameter family of contexts in which the bottom alpha fraction of retrieved passages is shuffled while the top fraction keeps its rank. Coupled with it is a teacher-selection rule: among the intermediate perturbation and the fully randomized one, the model distills from whichever yields the higher probability p(y_hat|x,c) for the ground-truth answer. A score-aware variant sets alpha per example from the largest gap in retriever scores, preserving passages above that gap. This machinery converts the binary choice of whether to perturb into a continuous, scenario-dependent choice, which is what lets the method pursue consistency (robustness to order) and rank distillation (respect for retriever prior) at once.","core_discovery":"The paper's core discovery is that consistency regularization and rank distillation are not necessarily in conflict, provided the perturbation used for distillation is chosen adaptively. CORD defines an interpolation space between the retriever-given order c and a fully randomized order c1, where an intermediate perturbation c1_alpha randomizes only the lower-ranked alpha fraction of contexts. The training objective combines a standard negative log-likelihood on both the given and perturbed contexts with a Jensen-Shannon divergence distillation loss between their output distributions. The teacher is selected by the model's own probability of the ground-truth answer: when the given order matters (MS MARCO), the model prefers a lightly perturbed c1_alpha, preserving the rank prior; when order is irrelevant (multi-needle), it freely pairs with the more random c1. The paper shows this adaptive pairing is what lets CORD outperform both plain fine-tuning and the IN2-style fixed-perturbation baseline on all five benchmarks.","pith_inferences":["The paper's logic implies a failure mode it does not test: if the model's confidence is miscalibrated on long contexts, adaptive selection could systematically prefer the more random teacher and reinforce, rather than correct, position bias; a calibration check on p(y_hat|x,c) across perturbation levels would settle this.","An extension the paper leaves open is combining the interpolation knob with other augmentations, such as query perturbation or targeted document edits, which could push the consistency-regularization benefit beyond order shuffling.","A practical implication not tested here is that per-example alpha adaptation may matter more than per-task tuning when deployment sees heterogeneous query streams, since the score-gap heuristic responds to each example's retriever output."],"forward_implications":["On MS MARCO, where the retriever's order is informative, CORD improves ROUGE-L from 41.34 with no fine-tuning and 44.52 with the fixed-perturbation baseline to 44.74, showing the rank prior is retained rather than unlearned.","On the multi-needle dataset, CORD raises F1 to 58.71 and MN-IDK refusal accuracy to 98.83, showing gains in both position-bias mitigation and avoidance of hallucination under irrelevant contexts.","Adaptive teacher selection is necessary: consistency-only training degrades MS MARCO performance in one representative scenario (36.87) while CORD improves it (44.74), because fully random perturbations unlearn the retriever's prior.","Score-aware alpha adjustment gives a further gain (F1 59.16 versus 58.71 on the multi-needle task), indicating that retriever score gaps contain usable signal for tuning perturbation strength per example."],"supporting_citations":[{"why":"Establishes the position-bias phenomenon in long-context language models that motivates consistency training.","marker":"Liu et al., 2024"},{"why":"Supplies the IN2 position-perturbation training recipe that CORD extends to both given and perturbed orderings.","marker":"An et al., 2024"},{"why":"Provides MS MARCO, the representative strong-rank-prior RAG scenario where full perturbation hurts.","marker":"Bajaj et al., 2018"},{"why":"Provides the GPT-4-based evaluation protocol used for answer correctness on the benchmarks.","marker":"Yang et al., 2024"},{"why":"Motivates the distillation loss as transferring dark knowledge between predictions on different orderings.","marker":"Hinton et al., 2015"},{"why":"Provides the NaturalQuestions benchmark reorganized by Liu et al. for RAG evaluation.","marker":"Kwiatkowski et al., 2019"},{"why":"Provides the HotpotQA multi-hop QA benchmark used in evaluation.","marker":"Yang et al., 2018"}],"fun_headline_variants":["CORD adapts to beat fixed perturbation across five RAG tasks","CORD balances consistency and rank, beating fixed perturbation","CORD: adaptive noise beats fixed ordering for RAG training","Adaptive context perturbation wins over fixed in RAG fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the model's own confidence that it can produce the correct answer from a reordered context reliably tells which reordering is the better training signal; if that confidence is unreliable, or if the gaps in retriever scores do not mark where finding the answer becomes harder, the adaptive choice can reinforce the very order-bias the method is meant to remove.","fun_headline_variants_meta":{"raw":{"variants":["CORD adapts to beat fixed perturbation across five RAG tasks","CORD balances consistency and rank, beating fixed perturbation","CORD: adaptive noise beats fixed ordering for RAG training","Adaptive context perturbation wins over fixed in RAG fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3639,"prompt_tokens":910,"completion_tokens":2729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2659}},"tokens_in":526,"tokens_out":2729,"duration_ms":15567,"temperature":1.0,"reasoning_tokens":2659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:05:39.413515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MS MARCO experiment with a deliberately miscalibrated base model, for instance one whose confidence stays high on random orderings, and compare CORD to fixed full perturbation; if adaptive teacher selection then performs no better than, or worse than, fixed randomization, the teacher-selection premise is falsified. Alternatively, an oracle experiment that replaces the model's confidence with the true probability of correct generation for each perturbation would show whether the adaptive mechanism is necessary, since oracle selection yielding no gain over fixed alpha=0.5 would indicate the heuristic is not carrying the result.","supporting_citations":[],"review_version":1}