{"id":"05e98f65-4a14-49cc-b60a-0b0bc2a928d7","arxiv_id":"2412.11707","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A DPO-tuned summarizer compresses QA contexts to 12-20% of their length, raising EM-per-token 6.8x but lowering absolute EM.","lead":"Question answering systems often feed whole retrieved paragraphs to a reader, wasting tokens on irrelevant sentences. This paper trains a summarizer with DPO-style reward modeling to compress contexts, reporting a 6.8x gain in EM per token, at the cost of lower raw exact match and on a filtered subset of data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reader-training distribution shift, not just self-generated preferences, may drive the apparent efficiency gain; the Origin baseline likely was never seen by the reader.","rationale":"The reader's weakest assumption concerns the reliability of self-generated DPO preference pairs; that is a real weakness and is explicitly conceded in the Limitations section. However, the more directly load-bearing and more cheaply testable problem is the reader evaluation protocol: the baseline Origin condition appears to use a reader trained only on summaries, making the comparison unfair and potentially explaining the entire reported efficiency gain. This concern is independent of whether the DPO preference signal is well-calibrated, and it undermines the headline 6.8x EPT claim even if the preference pairs are valid. I partially agree with the reader because both issues point to unsupported central claims, but the reader-training distribution shift is the sharper, more mechanically decisive flaw. The proposed concrete test would settle whether the baseline is simply handicapped; if it is, the paper's rejection is clearly warranted; if it is not, the remaining concern about DPO preference validity would still need addressing before acceptance.","tokens_in":9126,"tokens_out":8771,"duration_ms":85746,"concrete_test":"Train an additional reader with identical hyperparameters on the original (unsummarized) train contexts, and also train one on SFT-summarized contexts; evaluate both readers on original contexts and on each filtered condition (SFT, DPO O1,O2, DPO O1,O3) using the same test sets. If the original-trained reader restores Origin EM to roughly the SFT/DPO level, or changes the EPT ranking, then the paper's efficiency/accuracy trade-off claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing weakness is the evaluation protocol. Section 3.4 says the reader is trained on filtered contexts generated by the Type 1 (SFT) summarizer, yet Table 1's Origin row evaluates on original, long contexts. If no reader was trained on original contexts, then the Origin baseline is out-of-distribution: the reader has never seen long, noisy inputs during training. The central trade-off claim in Section 4 ('efficiency gained from filtering outweighs the slight loss in accuracy') assumes Origin EM (59.59/77.38/68.32) is the performance of a fairly trained reader; under distribution shift it is not. This alone could account for the apparent advantage of filtering, because the SFT/DPO summaries are in-distribution for the reader. A related but secondary issue is that Table 1's EPT appears computed as aggregate EM divided by average token length, not as the mean of per-example EM/|c| from Eq. (2), so the headline 6.8x number may not be the defined quantity. The reader-training control is the more fundamental and directly testable concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a context-filtering approach for extractive question answering in which a FLAN-T5 summarizer is fine-tuned, first with SFT and then with DPO, to compress retrieved contexts before they are passed to a reader. DPO preference pairs are generated automatically from prompt variants that include or omit the question, answer, or context, without human annotation. The authors introduce an Exact Match per Token (EPT) metric and report that filtered contexts substantially improve EPT, including a claimed 6.8-fold improvement, while retaining most of the original EM accuracy. The paper concludes that the token-efficiency gains from filtering outweigh a slight accuracy loss.","tokens_in":9252,"tokens_out":5095,"duration_ms":49796,"significance":"If the findings were sound, the paper would offer a practical pipeline for lowering the inference cost of open-domain QA through learned context compression, and the EPT metric would be a useful complement to EM/F1 for comparing context efficiency. The release of code and data is also a positive factor. However, the evaluation protocol as presented does not support the headline efficiency claim: the baseline reader appears to be out-of-distribution, the EPT numbers in Table 1 do not obviously match Eq. (2), and the paper does not compare with existing context-compression baselines. The central contribution is therefore promising but not yet established.","major_comments":[{"comment":"The reader is trained only on filtered contexts ('we train the reader to generate answers using the filtered context'), yet the Origin row in Table 1 evaluates the same reader on original, long contexts that it never saw in training. The Origin baseline is therefore out-of-distribution for the reader, so the lower EM and EPT of Origin may reflect distribution shift rather than filtering quality. This directly undermines the Section 4 claim that efficiency gains outweigh accuracy loss. The authors should train a matched reader on original contexts and compare all conditions under the same training distribution, or otherwise control for this mismatch.","section":"§3.4 and Table 1"},{"comment":"Eq. (2) defines EPT as EM(y*, yhat)/|c| per example, but the EPT values in Table 1 match dataset-level EM divided by average token length (e.g., 59.59/147.30 = 0.40 for NQr Origin). These are not the same quantity unless EM and token length are uncorrelated across examples. The 6.8-fold headline number is derived from this table-level computation, so the paper's primary quantitative claim is not backed by the stated metric. The authors should report the mean of per-example EPT and, ideally, confidence intervals.","section":"§3.5, Eq. (2), and Table 1"},{"comment":"The DPO chosen/rejected pairs are generated by the model itself using Type 1 vs. Type 2/Type 3 prompts, with no human validation that the Type 1 summaries are actually preferred. Because the Type 1 prompt includes the answer, the chosen summaries contain the answer by construction, so DPO directly optimizes a proxy—answer inclusion—that is correlated with the downstream IRA and EM on filtered contexts. The authors themselves concede in the Limitations section that this data generation 'may introduce ... some unintended biases, leading to biased results and inconsistent outcomes across the datasets.' This concession should be taken as a load-bearing caveat: the reported DPO results cannot be interpreted as a validated filtering preference without additional control experiments or preference validation.","section":"§3.3 and Limitations"},{"comment":"The paper compares only against the original context and against SFT/DPO variants; it does not compare with existing context-filtering or compression baselines such as Wang et al. (2023) or Xu et al. (2023), both of which are cited in the text. Without such comparisons, the claim of 'significantly outperform[ing] the baseline' is not established relative to the state of the art. At minimum, the authors should include an extractive filtering baseline and a length-controlled baseline, and report variance or significance over multiple runs.","section":"§4 and Table 1"},{"comment":"The abstract's claim of a 6.8-fold increase in EPT is presented as 'significantly outperform[ing] the baseline,' but the corresponding rows in Table 1 show substantial EM decreases (e.g., TQAr Origin 77.38 vs. DPO_O1,O3 62.29; SQuADr Origin 68.32 vs. DPO_O1,O3 55.40). Calling this a 'slight loss in accuracy' in Section 4 understates the trade-off. The authors should either separate efficiency claims from accuracy claims, or define a combined metric that explicitly makes the trade-off and justify why the efficiency gain dominates.","section":"Abstract and §4"}],"minor_comments":[{"comment":"The word 'Hense' should be 'Hence'.","section":"§3.3"},{"comment":"The word 'accuray' should be 'accuracy'.","section":"§4"},{"comment":"The phrase 'for a given given task prompt' contains a duplicated 'given'.","section":"§2.2"},{"comment":"The notation in Eq. (2) is inconsistent with the surrounding text: the metric is called EPT but the equation reads 'EP T', and the variables y* and yhat are not defined until the next sentence.","section":"§3.5 and Eq. (2)"},{"comment":"The parenthetical numbers in the NQr and TQAr rows are not explained in the table caption; the text should clarify that these are the counts after the top-1 context filtering described in §3.1.","section":"Table 3 and Appendix C"},{"comment":"The example text in Figure 1 is rendered without spaces (e.g., 'ACountryBoyCanSurvive'), which makes the figure hard to read; please reformat it.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a candid limitations section, and the core idea—using DPO for context summarization—is worth exploring. My recommendation is driven by the evaluation protocol: the Origin baseline appears to be out-of-distribution for the reader, and the EPT calculation does not match the stated definition. These are fixable with additional experiments, but they are load-bearing for the central claim, so I would not accept the paper in its current form. I would also ask the authors to add a comparison with existing context-compression methods before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result, a 6.8-fold increase in EM-per-token, does not hold up under scrutiny, and the reason is the evaluation protocol rather than the core idea. Section 3.4 says the reader is trained on filtered contexts generated by the Type 1 summarizer. Table 1's Origin row evaluates on original long contexts, which the reader likely never saw during training. If no reader was trained on original contexts, the Origin baseline is out-of-distribution, and the EM drop (e.g., 59.59 to 55.21 on NQ) may reflect reader unfamiliarity with long inputs rather than filtering quality. The Section 4 claim that \"efficiency gained from filtering outweighs the slight loss in accuracy\" depends on Origin being a fairly trained reader; that assumption is not established. This is directly testable and could change the conclusion.\n\nWhat is genuinely new: the use of automatically generated preference pairs based on the presence or absence of question, answer, and context, and the EPT metric. That is a legitimate and potentially useful idea, and the authors deserve credit for releasing code, evaluating on three datasets, and introducing the IRA metric to isolate summarization quality. The Limitations section is also refreshingly honest about the risks of self-generated training data.\n\nThe soft spots beyond the reader-control issue: the evaluation restricts to examples where the answer is already in the top-1 retrieved context, which is a favorable subset. There is no comparison with existing context-filtering baselines like RECOMP or Wang et al.'s filter-then-read, even though those are cited. The DPO preference pairs are model-generated without human validation, and the authors concede this may introduce biased results. Also, the EPT values in Table 1 appear to be aggregate EM divided by average token length, not the mean of per-example EPT from Eq. (2); for skewed length distributions these can differ.\n\nThese are load-bearing flaws for the paper's central claim, not cosmetic issues. But the underlying approach is coherent and the flaws are fixable. The paper deserves a serious referee, with the expectation of major revision: add an Origin-trained reader control, compare against existing filtering baselines, and clarify the EPT computation. Without the control, the headline trade-off is unsupported. I would not cite this in its current form, but I would bring it to a reading group as a useful case study in baseline construction and evaluation pitfalls.","headline":"The efficiency claim is undercut by a likely out-of-distribution baseline for the reader, but the automatic preference-pair idea is worth a second look.","tokens_in":9814,"tokens_out":1843,"would_cite":false,"duration_ms":18908,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward-model-based context filtering can cut retrieved passages to a small fraction of their length while keeping most question-answering accuracy, raising accuracy-per-token by up to 6.8 times.","keywords":["context filtering","question answering","reward modeling","direct preference optimization","token efficiency","em per token","summarize-then-read","detrimental retrieval"],"falsifier":"Retrain the DPO summarizer after randomly flipping half of the chosen/rejected labels in the generated preference pairs; if EPT still improves by a comparable factor, the reward signal is not what drives the efficiency gain.","tokens_in":8866,"feed_emoji":"⚡","tokens_out":8372,"duration_ms":73461,"temperature":0.7,"pith_summary":"Context filtering in question answering usually means deciding which parts of a retrieved passage are worth keeping. The paper claims that this filtering can be learned with reward modeling instead of human labels: a summarization model is trained with Direct Preference Optimization on pairs where the 'chosen' summary was generated with the question, answer, and context all present, and the 'rejected' summaries were generated with one element missing. On SQuAD, Natural Questions, and TriviaQA, the resulting pipeline keeps most of the original accuracy while shortening the context to 12-20 percent of its original length. The central evidence is the new EM Per Token metric: dividing Exact Match by context length shows up to a 6.8-fold efficiency gain over using the full retrieved context. If true, this gives low-resource or token-budget settings a way to get more correct answers per token without expensive preference annotation.","feed_headline":"Reward-model context filtering lifts QA token efficiency 6.8-fold","feed_subtitle":"Cutting passages to 12-20% of length keeps 80-92% of answering accuracy.","key_machinery":"The central mechanism is the DPO preference-construction scheme built on the information tuple I=(Q,A,C). The authors define three prompt types: Type 1 feeds all three elements and produces the 'chosen' summary; Type 2 omits the answer; Type 3 omits the question, and summaries from these two types serve as 'rejected' outputs. DPO training then maximizes the margin between chosen and rejected summaries in reward space, and the resulting summarizer is used in a summarize-then-read pipeline. The other key object is the proposed EPT metric, $EPT(c, y^*, \\hat{y}) = EM(y^*, \\hat{y}) / |c|$, which is what the 6.8-fold claim is measured on.","core_discovery":"On the paper's own terms, the discovery is that answer-relevant filtering can be induced from the data itself: a summarizer trained to imitate complete-information summaries while being pushed away from incomplete-information summaries learns to keep the span the reader needs. The chosen and rejected outputs are generated from the same base model using three prompt types that combine or omit the question, answer, and context, and the DPO loss widens the reward gap between them. When the filtered contexts are fed to a trained reader, the model attains exact-match scores in the high 40s to high 60s on the three datasets, compared with high 50s to high 70s for the original contexts, but with roughly one-fifth to one-eighth the tokens, and the EPT values for the best DPO variant reach 2.58-3.42 versus 0.38-0.51 for the original contexts. The authors conclude that the efficiency gained from filtering outweighs the slight loss in accuracy.","pith_inferences":["Our inference: because the preference pairs are generated from the presence of the answer in the prompt, the method is likely to favor contexts where the answer appears lexically and may underperform in reasoning-heavy answering regimes; a reader-aware reward would be a natural correction.","Our inference: the EPT gain should be re-baselined against simply truncating contexts to the same lengths; if random-length truncation achieves comparable EPT, the value of the DPO filtering is in content selection rather than shortening alone.","Our inference: the same auto-generated preference scheme could be applied to other input-compression tasks such as retrieval-augmented generation with long documents, where the 'chosen' output is the full query-gold answer-document tuple and 'rejected' outputs are missing one component."],"forward_implications":["A QA system with a fixed token budget can answer more questions correctly by filtering contexts with this method than by feeding full retrieved passages, making it attractive for API-based and on-device deployment.","The self-generated preference pairs remove the need for human preference annotation, so the training recipe can be rerun on any QA dataset where gold answers are available.","The summarize-then-read pipeline can be extended to multi-hop and long-context QA, settings the paper explicitly notes have not yet been tested.","The EPT metric gives a common scale for comparing context efficiency across models, so future filtering and compression work can be evaluated on accuracy per token rather than raw exact match."],"supporting_citations":[{"why":"Supplies the Direct Preference Optimization objective that turns the preference pairs into a reward-based training signal.","marker":"Rafailov et al., 2023"},{"why":"Provides the summarize-then-read pipeline structure that the paper adapts and extends.","marker":"Inoue et al., 2021"},{"why":"Provides the SQuAD dataset and the exact-match and F1 evaluation metrics used throughout.","marker":"Rajpurkar et al., 2016"},{"why":"Supplies the DPR retriever that generates the contexts for Natural Questions and TriviaQA.","marker":"Karpukhin et al., 2020"},{"why":"Provides the Natural Questions dataset used in the experiments.","marker":"Kwiatkowski et al., 2019"},{"why":"Provides the TriviaQA dataset used in the experiments.","marker":"Joshi et al., 2017"},{"why":"Provides the FLAN-T5 base model used for both the summarization and reader components.","marker":"Chung et al., 2022"},{"why":"Motivates the EPT metric by showing that additional context produces diminishing returns in open-domain QA.","marker":"Izacard and Grave, 2021"}],"fun_headline_variants":["QA context filter via reward model boosts token efficiency 6.8x","Filtering contexts with reward modeling: 6.8x token efficiency gain","Token-efficient QA: reward-trained filtering cuts context by 80%+","Reward-model filtering shows 6.8x token efficiency in QA","QA token efficiency up 6.8x via reward-model context filtering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a summary generated from the full question-answer-context tuple is always the 'chosen' one and summaries missing either the question or the answer are always 'rejected', an assumption the authors concede may introduce unintended biases.","fun_headline_variants_meta":{"raw":{"variants":["QA context filter via reward model boosts token efficiency 6.8x","Filtering contexts with reward modeling: 6.8x token efficiency gain","Token-efficient QA: reward-trained filtering cuts context by 80%+","Reward-model filtering shows 6.8x token efficiency in QA","QA token efficiency up 6.8x via reward-model context filtering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3554,"prompt_tokens":886,"completion_tokens":2668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2571}},"tokens_in":502,"tokens_out":2668,"duration_ms":17497,"temperature":1.0,"reasoning_tokens":2571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:04.964620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the DPO summarizer after randomly flipping half of the chosen/rejected labels in the generated preference pairs; if EPT still improves by a comparable factor, the reward signal is not what drives the efficiency gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the summarize-then-read pipeline structure that the paper adapts and extends."}],"review_version":1}