{"id":"0860d0c8-601d-48b1-bedb-43b62ff08584","arxiv_id":"2412.18232","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoLoR, a preference-optimized passage compressor, cuts LCLM retrieval context by 1.91x while improving average retrieval accuracy by 6% over nine benchmarks.","lead":"This paper trains a passage compressor specifically for long-context language model retrieval, using retrieval success as the reward. On nine benchmarks it cuts the input size roughly in half while slightly improving average retrieval accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3) relies on a length ordering that the §3.2.1 labeling procedure does not guarantee: a failed compression labeled rejected can be shorter than the chosen one, which would make (|yl|-|yw|) negative and flip the preference gradient for that pair.","rationale":"The reader's weakest-assumption analysis identifies the same issue: the length-ordering assertion behind Eq. (3) is not guaranteed by the stated labeling criteria. I agree that this is the most load-bearing concern because it targets the formal training objective itself rather than the auxiliary claims. The other flagged issues, such as the overstatement of 'consistently outperforms' in §5.1 and the Table 11 result produced by GPT-4o-mini rather than CoLoR, are real but secondary: the average 6% gain and 1.91× compression do not logically require every per-dataset comparison to win, and Table 11 is presented as a supplementary long-context scenario. The length-regularization issue, by contrast, determines whether the method described in the paper is the method actually evaluated. The concern is directly testable because the code and preference data are released; a negative finding would not automatically reject the empirical result, but it would require the authors to clarify or correct the objective. Since the reader already set a CONDITIONAL verdict, I recommend no change to that verdict: the paper should be accepted only conditional on verifying the length condition or correcting the loss formulation.","tokens_in":23992,"tokens_out":8873,"duration_ms":86967,"concrete_test":"Run the released CoLoR training script with a hook that logs |yl|−|yw| for every preference pair and also inspect the TRL-based implementation of Eq. (3) to determine whether it uses the raw signed difference, an absolute value, or a filter that discards pairs where yl is shorter. Then, if any negative pairs exist, retrain CoLoR using only pairs with |yl|>|yw| (or with an absolute-value correction) and compare the resulting Table 6 and Table 1 numbers. If the main results are unchanged, the concern is benign; if they shift materially, the reported improvement is not attributable to the loss as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The formal objective in Eq. (3) is LCoLoR = LSFT + λ·LOR·(|yl|−|yw|), with the paper asserting that yl is always longer than yw based on the data-collection criteria. But the criteria in §3.2.1 label the shortest successfully retrieved compression as chosen, label other successful compressions as rejected, and also label every failed compression as rejected. Nothing in the stated procedure requires a failed compression to be longer than the chosen one; a failed compression can easily be shorter. In that case |yl|−|yw| is negative, and since the ORPO odds-ratio loss LOR is positive for a correctly oriented pair, a negative multiplier reverses the optimization direction for that pair, pushing the model to prefer the rejected compression. Table 9 reports only average token counts (rejected 135.61 vs. chosen 91.36), which does not rule out a substantial number of per-pair negative differences. No filtering step, absolute value, or pair-construction rule that enforces the length ordering is described in the paper or appendix. This is load-bearing because Eq. (3) is the precise statement of CoLoR's training objective; if the implemented loss differs from the stated one, the reported 6% gain and 1.91× compression cannot be attributed to the described length-regularized preference optimization without additional evidence. The concern is not about fraud: the code is released, so the implementation can be checked directly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoLoR, a passage-compression model for long-context language model (LCLM) retrieval. Synthetic compressed passages are generated with several LLMs, labeled chosen/rejected based on whether the LCLM retrieves the correct passage and on relative length, and used to train a Phi-3-mini compressor with ORPO plus a length-difference regularization. Evaluation on nine retrieval datasets reports an average relative improvement of 5.9% in retrieval performance and a 1.91x average compression rate over raw passages, with ablations and out-of-domain tests. Code is released.","tokens_in":24245,"tokens_out":7047,"duration_ms":56897,"significance":"If the reported average gains are robust, CoLoR is a practically useful and inexpensive way to reduce the inference cost of LCLM retrieval while preserving or improving accuracy. The paper provides a complete pipeline and a released implementation, which is a strength. However, the evidence has several gaps: the length-regularization objective is not guaranteed to have the sign implied by Eq. (3), the 'consistently outperforms' claim is contradicted by Table 1, and the long-context benchmark result in Table 11 is not produced by CoLoR.","major_comments":[{"comment":"The length-regularization term in Eq. (3) multiplies the ORPO odds-ratio loss by (|yl|−|yw|), with the claim that yl is always longer than yw based on the data collection criteria. However, the labeling procedure in §3.2.1 labels every failed compression as rejected, without any requirement that failed compressions be longer than the chosen one. A failed compression can be shorter than the chosen compression, making |yl|−|yw| negative and reversing the preference gradient for that pair. Table 9 reports only average token counts (rejected 135.61 vs. chosen 91.36), which does not rule out per-pair negative differences; no filtering or verification is described. Because Eq. (3) is the stated training objective, the paper needs to either verify the length ordering in the actual data or code, or modify the objective (e.g., use an absolute value or only pairs satisfying the ordering).","section":"§3.2.1, Eq. (3)"},{"comment":"The claim that CoLoR 'consistently outperforms all baseline methods' is not supported by Table 1. CoLoR underperforms Raw Passage on FEVER (0.94 vs. 0.95), HotpotQA (0.86 vs. 0.87), and QAMPARI (0.55 vs. 0.56); it also underperforms Comp. w/ GPT on FEVER (0.94 vs. 0.96) and NQ (0.98 vs. 0.99), and COMPACT on FEVER (0.94 vs. 0.96). While the average and compression-rate gains are real, the 'consistently' phrasing should be replaced with a statement about average improvements and per-dataset outcomes.","section":"§5.1, Table 1"},{"comment":"The long-context retrieval result in Table 11 is not produced by CoLoR: the appendix states that the compressed passages are generated using GPT-4o-mini prompted for summaries under 200 words, and that training CoLoR on those datasets is infeasible. This table therefore does not support the claim that CoLoR is effective in long-context retrieval scenarios; it should be relabeled as a compression-prompting baseline or removed from the CoLoR evaluation.","section":"Appendix B, Table 11"},{"comment":"No error bars, variance estimates, or significance tests are reported for any of the main results. Given the small absolute differences on several datasets (e.g., 0.98 vs. 0.99 on NQ, 0.55 vs. 0.56 on QAMPARI) and the regressions noted above, the robustness of the headline 6% improvement is unclear. The paper should report results over multiple seeds (or at least standard deviations) and state whether the average gain is statistically reliable.","section":"Table 1 and Table 2"},{"comment":"The preference data are labeled using GPT-4o-mini retrieval, and the final evaluation also uses GPT-4o-mini as the LCLM. This shared-LCLM design means CoLoR could be overfitting to the idiosyncrasies of GPT-4o-mini's retrieval behavior. The paper's generalizability experiments vary the compression base model and datasets but never vary the retrieval LCLM. A simple validation would be to evaluate the same compressed passages with a different LCLM (e.g., Llama-3.1-70B or Qwen2.5) to confirm that the gains transfer.","section":"§4.2 and §5"}],"minor_comments":[{"comment":"The word 'Langauge' should be 'Language'.","section":"Conclusion"},{"comment":"Several reference entries contain formatting artifacts, e.g., 'F ound. Trends Inf. Retr.' and 'V olume' should be 'Found.' and 'Volume'.","section":"References"},{"comment":"The scatter plot does not clearly label which point corresponds to which method for all methods; consider adding point labels or a table with the same data.","section":"Figure 3"},{"comment":"The asterisk on CoLoR* is not defined in the table caption; it is explained in the text, but a footnote would help.","section":"Table 2"},{"comment":"The text says 'reject' in a few places where 'rejected' is intended; please standardize the terminology.","section":"§3.2.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: a genuinely useful paper on a real problem—making LCLM retrieval cheaper by compressing passages. The new bit is training a passage compressor with preference optimization on synthetic labels derived from retrieval success, plus a length penalty. That is a real combination, and on average it works: 6% better retrieval with 1.91x compression across 9 datasets. Code is released, ablations and out-of-domain tests are real evidence.\n\nThe soft spots are real but mostly fixable. The biggest is Eq. (3). The length-regularization term multiplies the ORPO loss by (|yl| − |yw|), and the text asserts yl is always longer than yw because of the data collection criteria. That assertion does not follow from the procedure in §3.2.1. A failed compression is labeled rejected regardless of length, so a failed output can be shorter than the chosen one. In that case the multiplier is negative and the gradient flips for that pair. The appendix only reports average token counts, so we do not know how many pairs are affected. The code is out, so the authors can settle this, but as written the objective is under-specified. This needs either a filter on pair construction, an absolute value, or a different justification.\n\nSecond, the claim in Section 5.1 that CoLoR 'consistently outperforms all baseline methods' is not supported by Table 1. CoLoR is below Raw Passage on FEVER and HotPotQA, and below the GPT-4o-mini compression baseline on FEVER. The average is positive, but 'consistently' is the wrong word. A more careful statement would improve the paper.\n\nThird, Table 11 in the appendix is mislabeled. The compressed passages there are generated by GPT-4o-mini, not by CoLoR, yet the surrounding text attributes the 216x compression to 'the compression model.' That is a presentation error, but it should be corrected.\n\nMinor points: no error bars or significance tests, which matters given the size of the gains. And the same LCLM (GPT-4o-mini) is used for labeling and for evaluation; that is a mild self-referential setup, but it is not fatal because the stated goal is to speed up LCLM retrieval with that same model.\n\nOverall, the core idea is sound and likely useful. I would send this to a serious referee, with the expectation of major revision on the Eq. (3) issue and the overclaims. If the authors clarify or fix the length pairing and correct Table 11, this becomes a solid paper. I would cite it after seeing the code confirm the training objective.","headline":"Valuable compression method for LCLM retrieval, but the length-regularized loss has a real technical gap and the consistency claim is overstated.","tokens_in":24828,"tokens_out":5396,"would_cite":false,"duration_ms":44388,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small trained compressor can shrink LCLM retrieval inputs by 1.91x and still improve retrieval accuracy by 6%.","keywords":["long-context language models","information retrieval","passage compression","preference optimization","ORPO","in-context retrieval","synthetic data","retrieval efficiency"],"falsifier":"Inspect the 3,418 training pairs and count how often the rejected compression is shorter than the chosen one; if a non-negligible share has negative $(|y_l|-|y_w|)$, the stated length-regularization objective is not the one being optimized, and the reported 1.91x compression would need re-explanation. Re-running the same training with a sign-corrected factor would show whether the brevity gains come from the claimed mechanism.","tokens_in":23731,"feed_emoji":"🗜️","tokens_out":7957,"duration_ms":62990,"temperature":0.7,"pith_summary":"Long-context language models can retrieve by reading a whole corpus in one context window, but every extra token costs compute and time. This paper claims that a small passage-compression model trained specifically for retrieval—CoLoR—cuts that cost: across nine retrieval benchmarks it reduces the in-context token count by a factor of 1.91 while improving average retrieval performance by 6% over raw passages. The training recipe uses synthetic preference data: several language models summarize each passage, the LCLM's retrieval outcome labels each summary as chosen or rejected, and the shortest successful summary is preferred. If the claim holds, compression is not a quality trade-off for LCLM retrieval but a way to make it cheaper and slightly more accurate at the same time.","feed_headline":"CoLoR compresses LCLM contexts 1.91x, lifts retrieval 6%","feed_subtitle":"A retrieval-trained passage compressor beats raw passages on nine benchmarks while cutting in-context token count nearly in half.","key_machinery":"The load-bearing object is the training objective: Odds Ratio Preference Optimization (ORPO), a preference-learning method that raises the odds of a chosen output over a rejected one without a separate reference model, augmented by a length-regularization term. CoLoR multiplies the ORPO odds-ratio loss by $(|y_l|-|y_w|)$, the token-length difference between rejected and chosen compressions, so pairs where the chosen compression is much shorter produce larger updates. The synthetic preference data is equally central: multiple LLMs summarize each passage, and the LCLM's retrieval outcome labels each summary chosen or rejected, with the shortest successful summary chosen. That data-generation loop is what lets the compressor optimize retrieval success and brevity simultaneously without human-written summaries.","core_discovery":"CoLoR is an abstractive passage compressor that rewrites each corpus document into a shorter passage and lets the LCLM retrieve over the compressed corpus. The paper's central claim is that this compressor, trained with Odds Ratio Preference Optimization on automatically labeled pairs, produces passages that are both shorter and more retrieval-effective than the originals. The data-generation loop is the key: multiple LLMs summarize each passage, retrieval success with a fixed LCLM labels each summary chosen or rejected, and among successful summaries the shortest is chosen. The training objective adds a length-regularization term to the ORPO odds-ratio loss, scaling updates by $|y_l| - |y_w|$, the token-length difference between rejected and chosen compressions. On nine single- and multi-document datasets, CoLoR reports a 1.91x average compression with a 6% average retrieval gain over raw passages, and the paper further reports that it outperforms extractive, prompt-based, and RAG-oriented compression baselines, generalizes to out-of-domain datasets, and mitigates the lost-in-the-middle degradation.","pith_inferences":["If the preference-labeling loop is what carries the gains, the same recipe could be applied to other in-context tasks where output quality is easily evaluated—for example compressing documents for text-to-SQL or multi-document reasoning—rather than only retrieval.","The 1.91x compression is an average over datasets; the paper's own tables show larger gains on some datasets and near-parity on others, so a practical deployment would likely compress selectively, only where the trade-off is favorable.","A direct test of the brevity mechanism would be to compute $|y_l|-|y_w|$ on all 3,418 training pairs; if rejected compressions are sometimes shorter than chosen ones, the sign of the regularization flips and the stated objective does not match the implemented one.","Because compression is trained on retrieval outcomes from a specific LCLM, the compressor implicitly inherits that model's failure modes; using a different LCLM at inference time may not preserve the same gains without retraining or adaptation."],"forward_implications":["If CoLoR's results hold, LCLM retrieval can run at roughly half the context length without losing accuracy, lowering compute and memory per query and allowing larger corpora inside a fixed context window.","Compression becomes a one-time indexing step: passages are compressed and cached once, so the added cost is amortized across all future queries, similar to building an index.","The benefit is not confined to LCLMs: the paper reports that dense retrieval also improves on compressed passages, so the same compressor could serve both retrieval paradigms.","CoLoR mitigates the lost-in-the-middle effect: performance degrades less when relevant passages sit in the middle of the context, suggesting compression removes distracting content.","The approach transfers to unseen datasets and even to very long passages, where compressed inputs reached a 216x size reduction with a 42% accuracy gain over truncated raw passages in the paper's long-context benchmark."],"supporting_citations":[{"why":"Supplies the LCLM-retrieval setup, prompt format, and the nine benchmark datasets that define the paper's evaluation.","marker":"Lee et al. (2024)"},{"why":"Provides ORPO, the preference-optimization objective that CoLoR adapts for compression.","marker":"Hong et al. (2024)"},{"why":"Phi-3-mini is the base model CoLoR is trained on and one of the LLMs used to generate candidate compressions.","marker":"Abdin et al. (2024)"},{"why":"GPT-4o-mini is the LCLM used to evaluate retrieval success for both data labeling and final benchmarking.","marker":"OpenAI (2024)"},{"why":"LLMLingua is an extractive compression baseline that CoLoR must outperform.","marker":"Jiang et al. (2023b)"},{"why":"Selective Context is the other extractive compression baseline, evaluated at rates 0.3 and 0.6.","marker":"Li et al. (2023)"},{"why":"RECOMP is the RAG-oriented abstractive compression baseline that motivates task-specific training.","marker":"Xu et al. (2024a)"},{"why":"COMPACT is the iterative abstractive compression baseline compared on retrieval accuracy and compression rate.","marker":"Yoon et al. (2024)"},{"why":"Documents the lost-in-the-middle phenomenon that CoLoR's position analysis is designed to address.","marker":"Liu et al. (2024b)"}],"fun_headline_variants":["Shorter passages, better LCLM retrieval: CoLoR","1.91x smaller context, 6% better retrieval with CoLoR","Retrieval-trained compression beats raw passages in LCLM","Cut LCLM context 1.91x, gain 6% retrieval with CoLoR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training procedure relies on rejected compressions always being longer than chosen ones; if a failed compression is shorter, the brevity bonus reverses and pushes the model toward worse preferences.","fun_headline_variants_meta":{"raw":{"variants":["Shorter passages, better LCLM retrieval: CoLoR","1.91x smaller context, 6% better retrieval with CoLoR","Retrieval-trained compression beats raw passages in LCLM","Cut LCLM context 1.91x, gain 6% retrieval with CoLoR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2206,"prompt_tokens":1006,"completion_tokens":1200,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1116}},"tokens_in":622,"tokens_out":1200,"duration_ms":9748,"temperature":1.0,"reasoning_tokens":1116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:54:13.595704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the 3,418 training pairs and count how often the rejected compression is shorter than the chosen one; if a non-negligible share has negative $(|y_l|-|y_w|)$, the stated length-regularization objective is not the one being optimized, and the reported 1.91x compression would need re-explanation. Re-running the same training with a sign-corrected factor would show whether the brevity gains come from the claimed mechanism.","supporting_citations":[],"review_version":1}