{"id":"af928ff6-ac5c-4f1c-9eb0-f974a9c51c5e","arxiv_id":"2412.20756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Shapley-value-based counterfactual regularization improves dense retrievers' robustness to adversarial attacks and enables key passage extraction without passage-level relevance annotations.","lead":"This paper proposes a counterfactual contrastive learning method for dense retrieval models that uses Shapley values to identify key passages in documents. The regularized models show improved robustness to adversarial attacks and can extract key passages without passage-level labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adversarial counterfactual defined by Eq. (8) is an argmax, but the implementation appears to use random 5% word replacement; if so, Eq. (9)'s ordering and the L_adv robustness gains may be artifacts of generic noise, not the claimed adversarial counterfactual.","rationale":"The reader's weakest_assumption names both the undefined Shapley characteristic function and the adversarial inequality. I focus on the latter because it is the more direct load-bearing support for the paper's headline robustness claim and because Table 8's ablation attributes the largest robustness gain to L_adv. The gap is not a stylistic issue: Eq. (8) and Eq. (9) describe a specific optimization and an ordering that the text and implementation details do not instantiate. A random 5% replacement is not an argmax, and there is no evidence that the resulting d_adv lies between d+ and d-. The proposed concrete test settles the question by measuring the ordering directly and by comparing against a true argmax. If the concern lands, the authors can still fix the paper by revising the method description or re-running with the actual adversarial search; therefore the verdict remains conditional rather than reject. This partially agrees with the reader, whose weakest_assumption also flagged Eq. (9), and it does not change the overall CONDITIONAL verdict.","tokens_in":18394,"tokens_out":8329,"duration_ms":83036,"concrete_test":"Instrument the released/described training pipeline to log each training tuple and, using the frozen base retriever, compute the fraction of d_adv examples that satisfy both f(q,d+) > f(q,d_adv) and f(q,d_adv) > f(q,d-). Then retrain with d_adv produced by a genuine argmax over synonym/word substitutions (e.g., a PRADA-style search) and compare the Table 7 percentage decreases. If the random-5% version violates Eq. (9) often, or if the true-argmax version yields materially different robustness numbers, the paper's stated adversarial-counterfactual mechanism is not what was evaluated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central robustness claim hinges on the construction of adversarial counterfactual documents. Eq. (8) defines d_adv as the argmax of f(q,d')-f(q,d) over B(d,epsilon), and Eq. (9) asserts f(q,d+) > f(q,d_adv) > f(q,d-). However, the paper never gives a deterministic search procedure for this argmax; the only implementation detail is 'epsilon chosen for creating adversarial examples is 5%' (Sec. 4.3), and Section 3.3 describes B(d,epsilon) as a set where words are 'randomly replaced.' Random replacement does not maximize the score difference and can easily violate Eq. (9), e.g., by producing a document that scores below the negatives. Since Table 8 shows that adding L_adv is what most improves robustness, the headline 'surpassing the state-of-the-art anti-attack methods' is not supported if L_adv is actually trained on random noise. The paper needs to specify and, preferably, release the exact d_adv generation procedure and verify the Eq. (9) ordering on training data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a counterfactual contrastive learning method for dense retrieval. The authors use Shapley values over document passages to identify key passages without passage-level relevance annotations, then construct three types of counterfactual documents (partial, full, and adversarial) and add contrastive losses that push positive documents above counterfactuals above negatives. Experiments on MS MARCO document and passage datasets report improved key-passage extraction MRR@10p for DPR and ANCE, no degradation in document retrieval, and smaller relative performance drops under four adversarial attack methods, leading the authors to claim state-of-the-art robustness.","tokens_in":18607,"tokens_out":4722,"duration_ms":46765,"significance":"If the mechanism were fully specified and reproducible, the paper would make a useful empirical contribution: it addresses both explainability (key passage extraction without passage labels) and robustness of dense retrievers in one framework, and it evaluates across six retrieval models and four attack methods. The ablations in Tables 5 and 8 are informative, and the comparison against AT, CertDR, and PIAT is a reasonable first step. However, the central mechanisms are under-specified, and one of the headline claims in the abstract is stronger than Table 7 supports.","major_comments":[{"comment":"The Shapley-value key passage extraction is not reproducible as written. The manuscript does not specify how the relevance function v(P) is evaluated for an arbitrary subset of passages P when the retriever scores whole documents, how the exponential number of coalitions is sampled or approximated, or how the 'Non-overlap' and 'Merge' procedures described in Section 4.1 relate to Eq. (4). Since Table 2's central claim that Shapley values outperform δrank and δrel depends on this computation, please provide a precise algorithm, including the subset-to-document mapping, the approximation strategy, and the number of samples used, and ideally release the code.","section":"Section 3.2, Eq. (4)"},{"comment":"The adversarial counterfactual document d_adv is defined as the argmax of f(q,d') - f(q,d) over B(d,epsilon), but the only implementation detail given is that epsilon is 5% and B(d,epsilon) is described as a set where words are 'randomly replaced.' Random replacement is not an argmax search and can easily produce documents that violate the ordering f(q,d+) > f(q,d_adv) > f(q,d-) asserted in Eq. (9). Because Table 8 attributes the largest robustness gains to the L_adv term, the reported robustness improvements may be driven by exposure to generic noisy documents rather than by the claimed adversarial counterfactuals. Please specify the exact d_adv construction procedure, verify the Eq. (9) ordering on training data, and report the empirical fraction of examples for which it holds.","section":"Section 3.3, Eqs. (8)-(9) and Section 4.3"},{"comment":"The abstract and conclusion claim that the regularized dense retrieval models 'surpass the state-of-the-art anti-attack methods,' but Table 7 shows that DPR_counter outperforms PIAT only on TS and PRADA, while PIAT is significantly better under PAT and MCARA. The claim should be weakened to a statement such as 'comparable to or better than PIAT on some attacks without requiring attack-specific training data,' or additional evidence should be supplied to justify the stronger wording.","section":"Abstract and Table 7"},{"comment":"The main robustness comparison against AT, CertDR, and PIAT is reported only for DPR_counter; ANCE_counter appears in Table 6 but is not included in Table 7. Since the abstract refers to 'regularized dense retrieval models' in the plural, including ANCE_counter in the Table 7 comparison would substantially strengthen the generality of the robustness claim.","section":"Table 7"}],"minor_comments":[{"comment":"The title contains a typo, 'conterfactual,' and the abstract contains 'fine-graned'; both should be corrected.","section":"Title and abstract"},{"comment":"The sentence introducing L_neg says the loss maximizes the similarity between (q,d) and (q,d'), but the formula uses (q,d+) as the anchor; the text should be aligned with the equation.","section":"Section 3.3, Eq. (10)"},{"comment":"The text says the final loss is a weighted sum of 'all three loss functions,' but the expression contains L_cla, L_neg, L_pos, and L_adv; please clarify the grouping or change the wording.","section":"Section 3.3, Eq. (13)"},{"comment":"The 'couple learning' strategy is attributed to reference [33], whose title is 'Perturbation-Invariant Adversarial Training for Neural Ranking Models'; this citation does not appear to describe a couple-learning method, so please verify and, if needed, cite the correct source.","section":"Section 3.3, loss weight strategies"},{"comment":"The term 'unsupervised' should be qualified: the method does not use passage-level relevance labels, but it does use query-document relevance labels and hard negatives, so it is more accurately described as 'without passage-level relevance annotations.'","section":"Title and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical structure and the direction is valuable, but the missing implementation details for the Shapley computation and the adversarial counterfactual construction are load-bearing. The abstract's robustness claim also overstates Table 7. These issues are fixable in a revision, so I would not reject; I would ask for the exact algorithms, a verification of Eq. (9), and a toned-down claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen and colleagues do something genuinely useful: they use Shapley values to locate the passage in a document that a dense retriever leans on, then train with counterfactual contrastive losses to make the model more sensitive to that passage. The key passage extraction results are the most credible part of the paper. Across DPR, ANCE, ColBERT, and ME-BERT, their Shapley-based method beats delta-rank and delta-rel on MRR@10p, and the gap to the supervised upper bound is presented honestly. That contribution stands even if the robustness story weakens.\n\nThe soft spots are real, and the second-pass stress-test lands. Eq. (8) defines d_adv as the argmax of the score difference over B(d, epsilon), but the paper never specifies how that argmax is computed. The only implementation detail is 'epsilon chosen for creating adversarial examples is 5%'. Meanwhile B(d, epsilon) is described as 'randomly replaced' words. If the implementation generates a single random replacement (or even a few random samples without a proper search), the ordering in Eq. (9) is not guaranteed, and L_adv is closer to noise-robustness training than to adversarial training. Table 8 shows L_adv is the main driver of robustness, so this matters. The authors need to state exactly how d_adv is constructed, and ideally verify the Eq. (9) inequality holds on training data. If it doesn't, the 'surpassing state-of-the-art anti-attack methods' claim in the abstract is not supported, and in any case PIAT beats them on PAT and MCARA, so the abstract overclaims.\n\nThe Shapley approximation is also underspecified. The paper says they use non-overlap and merge strategies, but doesn't say how subsets are sampled or how many evaluations of v(P) are done. For a method whose whole point is model-agnostic attribution, this is a reproducibility hole. I'd also like to see ANCE_counter in Table 7, since the main experiments present both DPR and ANCE but the robustness table only uses DPR_counter.\n\nNone of this is fatal. The key passage extraction idea is sound, the evaluation design uses external attacks and human passage labels, and the modest gains in Table 6 are consistent. But the central robustness claim needs a correction and a clarification. I would send this to peer review, with a request for major revision and code release. I wouldn't cite it in its current form, but I'd look at a revised version.","headline":"Useful unsupervised key-passage extraction via Shapley values, but the adversarial robustness headline rests on an underspecified construction that may just be random noise.","tokens_in":19209,"tokens_out":2965,"would_cite":false,"duration_ms":28694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counterfactual training shields dense retrievers from ranking attacks.","keywords":["dense retrieval","counterfactual learning","Shapley value","adversarial robustness","key passage extraction","contrastive learning","MS MARCO"],"falsifier":"Two concrete tests would settle the claim. First, recompute the Shapley passage ranking after shuffling passage order and check whether the top-ranked passage still matches the labeled relevant passage. Second, regenerate adversarial counterfactuals with the argmax in Eq. (8) rather than random 5% replacement and rerun the four attacks: if the robustness gap over adversarial training shrinks or vanishes, the defense rested on easy counterfactuals.","tokens_in":18157,"feed_emoji":"🛡️","tokens_out":13719,"duration_ms":122332,"temperature":0.7,"pith_summary":"Dense retrieval models rank documents by embedding similarity, but they are easy to fool: small synonym substitutions can push an irrelevant document to the top. This paper argues that the root problem is insensitivity to fine-grained relevance, and proposes to fix it by forcing the model to react strongly when the passage that actually answers a query is altered, while staying calm when irrelevant passages change. To find that key passage without any passage-level labels, the authors treat a document's passages as players in a coalitional game and rank them by Shapley value computed from the retriever's own relevance scores. The same counterfactual passages then become training pivots in a contrastive loss, so the retriever learns to distinguish a true positive from a slightly damaged version and from a negative. The result, on MS MARCO document retrieval, is a model that locates key passages better than simpler deletion-based baselines and loses less ranking accuracy under four adversarial attacks than standard adversarial training and a certified defense.","feed_headline":"Counterfactual training shields dense retrievers from ranking attacks","feed_subtitle":"A Shapley-value regularizer teaches a retriever to point at the passage that justifies its ranking, with no extra labels.","key_machinery":"The load-bearing object is the Shapley value of a passage in a coalitional game whose players are the document's passages and whose value function $v(P)$ is the retriever's relevance score on the document made of the passage subset $P$. Equation (4) computes each passage's contribution as the weighted average of marginal score changes over all subsets, and the paper approximates the exponential sum by computing over non-overlapping windows and merging the results with a moving average. The companion mechanism is a stack of counterfactual contrastive losses: partial counterfactuals (one sentence removed from the key passage), full counterfactuals (key passage removed), and adversarial counterfactuals (5% of words replaced) serve as graded pivots between the positive and negative documents, forcing the embedding space to respect $f(q,d^+) > f(q,d') > f(q,d^*) > f(q,d^-)$ for counterfactuals and $f(q,d^+) > f(q,d_{adv}) > f(q,d^-)$ for attacks. Shapley values also set the loss weights, and a coupling-learning strategy that adapts the weights during training gives the best key-passage extraction and robustness.","core_discovery":"The paper's central claim is that the vulnerability of dense retrievers and their opacity can be addressed by making the relevance score obey an explicit sensitivity profile: high variance when the key passage of a relevant document is removed or corrupted, low variance when irrelevant passages are touched. The authors show that Shapley values computed over passages with the retriever's score as the value function identify the key passage more accurately than delta-rank or delta-relevance baselines, and that this signal can drive training even when passage labels are simulated by an LLM. They then use partial, full, and adversarially constructed counterfactual documents as graded pivots in three contrastive losses, so the regularized model places the positive above partial counterfactuals, full counterfactuals, adversarial counterfactuals, and negatives in that order. On term spamming, PRADA, PAT, and MCARA, the regularized DPR loses only 10.1 to 16.1 percent of MRR@10d, a smaller drop than adversarial training (13.5 to 19.2 percent) and the certified defense CertDR (12.4 to 17.3 percent), and it beats the attack-specific PIAT on two of the four attacks. The same regularization does not degrade retrieval effectiveness, which is the practical payoff: robustness and interpretability come from the same training signal.","pith_inferences":["Beyond the paper: the Shapley attribution is only as sound as the score function it is computed from, so a natural next check is whether the same passage ranking survives different document segmentations and window sizes; the moving-average merge used here partly masks that dependency.","Beyond the paper: because the defense does not require attack-specific training examples, the same counterfactual regularization could be inserted into late-interaction or generative retrieval models as a label-free robustness layer.","Beyond the paper: the coupling-learning weighting suggests that alpha and beta can be learned rather than tuned, pointing toward a curriculum that gradually moves from partial to full to adversarial counterfactuals.","Beyond the paper: since the method produces a causal, counterfactual rationale for each ranking, comparing those rationales against multi-passage human annotations on a dataset richer than a single relevant passage per document would be a sharper test of the explainability claim."],"forward_implications":["Key passage extraction becomes a byproduct of retrieval: any dense retriever can identify the passage that justifies a query without passage-level relevance labels.","The robustness gain does not require training on attack-specific adversarial examples, so the defense generalizes to attack methods the model never saw, whereas PIAT needs attack-generated training data.","The regularization leaves standard retrieval performance intact and even slightly improves MRR@10d on MS MARCO-doc and NDCG@10 on TREC DL2019, so robustness is not bought at the cost of effectiveness.","Shapley-based loss weighting beats relevance-score weighting, so the quality of the passage attribution directly controls the size of the robustness improvement.","Ablations show that the combination of pseudo-positive and hard-negative counterfactual losses drives key-passage extraction, while the adversarial counterfactual loss drives robustness, and only the full loss stack achieves the best of both."],"supporting_citations":[{"why":"Supplies the Shapley value formula used to rank a document's passages by their contribution to the retrieval score.","marker":"[59]"},{"why":"Defines the DPR dense retriever that serves as the primary backbone for the counterfactual contrastive learning experiments.","marker":"[17]"},{"why":"Defines ANCE, the second retriever backbone on which the counterfactual regularization is applied and evaluated.","marker":"[66]"},{"why":"Provides the MS MARCO-doc and MS MARCO-passage triples used to train and evaluate the retrieval and key-passage-extraction results.","marker":"[40]"},{"why":"PRADA, a decision-based black-box attack used to measure how much ranking accuracy the regularized model loses.","marker":"[63]"},{"why":"PAT, an anchor-based trigger attack used as one of the four robustness tests.","marker":"[27]"},{"why":"MCARA, the strongest tested dense-retrieval attack, used to show the retrained model still extracts key passages under attack.","marker":"[31]"},{"why":"CertDR, the certified defense baseline that the regularized model beats on relative MRR@10d decrease.","marker":"[61]"},{"why":"PIAT, the attack-specific adversarial training baseline, and the source of the coupling-learning strategy for setting the loss weights.","marker":"[33]"},{"why":"Supplies the vanilla adversarial training baseline whose relative MRR@10d decrease the counterfactual model is compared against.","marker":"[10]"}],"fun_headline_variants":["Shapley regularizer hardens dense retrievers against attacks","Counterfactual training makes dense retrievers robust and explainable","Shapley values teach retrievers to pinpoint key passages","Unsupervised counterfactual loss strengthens retrieval robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a retriever's relevance score on arbitrary subsets of a document's passages is a faithful measure of what those passages contribute, so deleting a passage lowers the score exactly when that passage matters.","fun_headline_variants_meta":{"raw":{"variants":["Shapley regularizer hardens dense retrievers against attacks","Counterfactual training makes dense retrievers robust and explainable","Shapley values teach retrievers to pinpoint key passages","Unsupervised counterfactual loss strengthens retrieval robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2802,"prompt_tokens":1061,"completion_tokens":1741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1673}},"tokens_in":677,"tokens_out":1741,"duration_ms":13099,"temperature":1.0,"reasoning_tokens":1673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:12:13.554828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two concrete tests would settle the claim. First, recompute the Shapley passage ranking after shuffling passage order and check whether the top-ranked passage still matches the labeled relevant passage. Second, regenerate adversarial counterfactuals with the argmax in Eq. (8) rather than random 5% replacement and rerun the four attacks: if the robustness gap over adversarial training shrinks or vanishes, the defense rested on easy counterfactuals.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Shapley value formula used to rank a document's passages by their contribution to the retrieval score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ANCE, the second retriever backbone on which the counterfactual regularization is applied and evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PAT, an anchor-based trigger attack used as one of the four robustness tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MCARA, the strongest tested dense-retrieval attack, used to show the retrained model still extracts key passages under attack."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CertDR, the certified defense baseline that the regularized model beats on relative MRR@10d decrease."},{"cited_title":"Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off","cited_arxiv_id":"2312.10329","evidence_quote":"PIAT, the attack-specific adversarial training baseline, and the source of the coupling-learning strategy for setting the loss weights."}],"review_version":1}