{"id":"7787cd22-9234-4ddc-9fbf-26ddb40bab95","arxiv_id":"2505.13757","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoRank reranks scientific documents by first scoring 200 candidates from compact LLM-extracted features and then refining the top 20 with full text, improving average nDCG@10 from 50.6 to 55.5.","lead":"This paper proposes CoRank, a three-stage reranking method that first uses LLM-extracted document features such as category, section, and keywords to rank a wide pool of candidate papers, then reranks the top candidates using their full text. The method is training-free and reports consistent nDCG gains over vanilla listwise reranking across five scientific retrieval datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +4.9 nDCG@10 headline likely confounds the compact-feature coarse stage with the larger candidate pool, so the claimed advantage of semantic-feature representations over mere pool expansion is not established.","rationale":"The reader's verdict is CONDITIONAL and identifies the same confound: the paper does not isolate the effect of compact features from the effect of the larger candidate pool, and the paper's own Section 3.3.3 is the admission that features underperform full text at fixed document count. I agree with that reading. The most load-bearing consequence is that the headline +4.9 nDCG@10 gain may be attributable to expanding the pool to 200 documents rather than to the semantic-feature representation, which would undercut the claimed novelty. The proposed control is concrete and would settle the question. The paper has genuine independent support: human evaluation of the IE quality (Table 5), ablations isolating the feature components, and experiments across five datasets and multiple backbones. Those support the engineering claim that some form of broad-then-fine reranking helps, but they do not isolate the mechanism claimed. Therefore, CONDITIONAL is the right verdict: plausible and well-supported as a system, but the central conceptual claim needs the controlled test before it can be accepted as stated. No objection to the practical contribution, only to the attribution of the gains.","tokens_in":22357,"tokens_out":1356,"duration_ms":11296,"concrete_test":"Run a controlled comparison on LitSearch with GPT-4.1-mini: (i) CoRank as reported (feature-based coarse ranking over 200; fine over top 20 full text); (ii) full-text coarse ranking over 200 using the same two-stage protocol (if needed, use pooling/sliding to fit 200 full texts into context); (iii) random sampling of 20 full-text documents from the top 200. If (ii) matches or beats (i), the compact-feature representation is not the source of the gains and the central claim should be reframed; if (i) clearly beats (ii), the feature representation carries genuine signal beyond pool expansion.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that CoRank improves LLM listwise reranking by using compact semantic features to broaden the candidate pool. However, the main comparison (Table 1) contrasts CoRank (200 compact docs then 20 full text) against vanilla reranking (20 full text) and sliding windows (100 full text with windows). These baselines vary both the representation (compact features vs full text) and the pool size simultaneously. The paper's own Section 3.3.3 admits that at a fixed document count, feature-based representations underperform full text. Thus the observed gain may come almost entirely from considering 200 candidates instead of 20, rather than from the semantic-feature representation per se. A pooling oracle or a 'full-text-expanded-pool' control is missing. Because the framework's novelty hinges on the semantic features being information-preserving enough to help select the top 20, the design cannot distinguish 'semantic features matter' from 'more candidates matter'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoRank, a training-free, model-agnostic three-stage pipeline for LLM listwise reranking in scientific retrieval. In stage one, document-level semantic features (category, sections, keywords) are extracted offline by an LLM. In stage two, a coarse listwise reranking is performed over up to 200 compact feature-based document representations, producing a shortlist. In stage three, the top 20 shortlisted documents are reranked again using their full text. The authors motivate the design by the observation that first-stage retrieval in the scientific domain is weak and that full-text listwise reranking is limited to roughly 20 candidates per prompt. They evaluate on five datasets (LitSearch, CSFCube, NFCorpus, SciFact, Trec-Covid) with three LLM backbones, reporting consistent improvements over vanilla listwise reranking and over a sliding-window baseline, with a headline average gain of +4.9 nDCG@10. The paper also includes ablations, hyperparameter studies, retriever robustness tests, small-backbone experiments, and a human evaluation of the extracted features.","tokens_in":22546,"tokens_out":5758,"duration_ms":55330,"significance":"If the claimed mechanism is correct, the paper offers a practical and broadly applicable recipe for improving zero-shot listwise reranking under tight context budgets: use compact, structured document representations to widen the candidate pool, then refine with full text. The empirical scope is substantial (five datasets, three reranking backbones, four first-stage retrievers, two small backbones), and the ablation and human-evaluation studies are thoughtful. The idea is simple and orthogonal to existing reranking techniques, which makes it potentially useful to practitioners. However, the central attribution of the gains to the semantic-feature representation is not established, because the main comparisons confound representation type with candidate pool size, and the paper itself concedes that at a fixed document count compact features underperform full text.","major_comments":[{"comment":"The headline comparison varies candidate pool size and representation type simultaneously. CoRank reranks 200 compact documents before refining over 20 full-text documents, whereas the vanilla baseline reranks 20 full-text documents and the sliding-window baseline reranks 100 full-text documents. The paper itself states in §3.3.3 that when the number of documents is held constant, feature-based representations underperform full text. Therefore the consistent gains in Table 1 and the advertised +4.9 nDCG@10 are equally compatible with the hypothesis that the improvement comes from considering 200 candidates instead of 20, and they do not establish that LLM-extracted semantic features are the reason. To separate the two factors, please add controls such as (i) CoRank with 20 compact documents versus 20 full-text documents; (ii) a token-equivalent non-semantic compact representation (e.g., the first N tokens or a truncated abstract) reranked over 200 candidates; and (iii) a full-text-based expanded-pool baseline of the same candidate count (e.g., via multiple windows or a long-context model). Without these, the claimed advantage of the semantic-feature representation over mere pool expansion is not demonstrated.","section":"§5.1, Table 1; §3.3.3"},{"comment":"The efficiency comparison is subject to the same confound. The claim that CoRank needs only 40% of the sliding-window token budget attributes the savings to the semantic features, but any token-efficient representation, including simple truncation, would exhibit a similarly low token profile. Please report a cost-performance comparison at equal performance or equal candidate coverage, including a non-IE compact baseline, and clarify whether the reported token usage includes the offline feature-extraction cost. This would make the efficiency claim interpretable.","section":"§5.3.2, Table 4"},{"comment":"The default configuration uses 5 keywords, yet Table 2 shows that 15 keywords give the best coarse-stage nDCG@10 (42.6 vs. 41.1) at roughly 1.6x the token cost. Please either justify the choice of 5 as the default or present the main results with the better-performing configuration and discuss the sensitivity of the headline gain to this choice.","section":"§5.3.1, Table 2"}],"minor_comments":[{"comment":"The text says 'As exhibited in Table 5.3.5' but the table is numbered Table 5; please correct the reference.","section":"§5.3.5"},{"comment":"The abstract reports an average gain 'from 50.6 to 55.5' while §5.2.1 reports 'from 47.2 to 54.9' without sliding windows. The two numbers use different aggregations; please state the aggregation rule explicitly so readers can reconcile them.","section":"Abstract; §5.2.1"},{"comment":"The label 'CoRank Full' is ambiguous because CoRank is a two-stage method (compact features followed by full text). A label such as 'CoRank (compact→full)' would be clearer.","section":"Table 1"},{"comment":"The table caption should state explicitly that the score is an average over LitSearch and CSFCube and over the three tested models; the current text leaves this implicit.","section":"§5.3.2, Table 4"},{"comment":"The average full-text token length of roughly 200 tokens with a maximum of 5,774 is surprising for scientific papers; please clarify whether the 'full text' input is the complete paper or a truncated/abstract-only version, since this affects the interpretation of the token-efficiency analysis.","section":"§3.3.2, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The confounded comparison is the main obstacle. If the requested controls show that pool expansion alone reproduces the gains, the paper should be reframed as a candidate-pool-expansion method rather than as evidence for the value of LLM-extracted semantic features. The contribution is potentially publishable either way, but the current text overclaims the role of the IE representation. The remaining issues are presentation-level and should not block a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest empirical paper, but the reader's conditional verdict and the stress-test are both right. CoRank's two-stage idea—rank a large pool with cheap LLM-extracted features (category, section, keywords), then rerank the top 20 with full text—is a genuinely useful engineering recipe for scientific retrieval. The evaluation is more thorough than most reranking papers: five datasets, three backbones, a small-model check, four first-stage retrievers, and a human evaluation of the feature extraction itself. The ablations show each feature contributes, and the hyperparameter studies suggest the method is not brittle. I would trust the headline numbers.\n\nThe soft spot is the confound. Table 1 compares CoRank (200 compact + 20 full text) against vanilla (20 full text) and sliding windows (100 full text). Nothing here controls for candidate count. The paper's own Section 3.3.3 admits that at fixed document count, feature-based representations underperform full text. So the +4.9 nDCG@10 could be mostly 'we looked at 200 documents instead of 20,' and the compact representation is just the way to fit 200 into context. The claim that the semantic features preserve enough signal to guide the coarse stage is plausible but not isolated by the main experiment. A control with 200 full-text candidates (long-context model, or a windowed baseline over 200) or a pooling-oracle analysis would settle this. The lack of released code and prompt templates is also a real barrier for replication, because the feature extraction is central to the method.\n\nTo be clear, the confound is in attribution, not in the usefulness of the result. Even if all the gain comes from a larger pool, CoRank is a practical and token-efficient way to get that coverage, and the paper is honest about the fixed-count tradeoff. The related work is fairly cited, and the limitations are not buried. This deserves a serious referee; I would send it to review and ask for the control and artifacts.","headline":"CoRank is a practical, well-evaluated two-stage reranking method, but the reported gains confound semantic features with simply seeing more candidates.","tokens_in":23053,"tokens_out":2975,"would_cite":true,"duration_ms":26863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoRank claims that LLM reranking should first score 200 papers from compact semantic summaries, then re-score the top 20 on full text, and reports consistent top-10 ranking gains on five scientific retrieval benchmarks.","keywords":["Information Retrieval","Scientific Document Search","Large Language Models","LLM-Based Reranking","listwise reranking","zero-shot reranking","compact document representations","candidate coverage"],"falsifier":"Run CoRank on a benchmark where the first-stage retriever already has perfect recall within its top 20: if CoRank still beats vanilla full-text reranking over the same 20 documents, the gain is not coming from candidate coverage; equivalently, compare the coarse stage's recall at 20 against the retriever's recall at 20 and check whether the extra pool is actually adding relevant documents.","tokens_in":22183,"feed_emoji":"🔬","tokens_out":9612,"duration_ms":84448,"temperature":0.7,"pith_summary":"The paper argues that standard LLM listwise reranking is self-limiting for scientific search: because full text eats the context window, only about 20 candidates get reranked, and when the first-stage retriever is weak, the truly relevant papers are often outside that window. CoRank instead reranks in two passes. In the first pass, each paper is reduced offline to a compact semantic representation—hierarchical category, one query-relevant section, and a few top keywords—so 200 candidates fit in the same prompt; the top 20 from that coarse ranking are then reranked on full text. The authors report that this training-free, model-agnostic recipe lifts average nDCG@10 from 50.6 to 55.5 across five scientific retrieval datasets and several LLM backbones, at a fraction of the token cost of sliding-window expansion.","feed_headline":"Rerank 200 papers by summary, then refine the top 20","feed_subtitle":"Two-stage CoRank lifts average top-10 ranking quality from 50.6 to 55.5 across five scientific benchmarks","key_machinery":"The load-bearing object is the compact semantic document representation, specifically the paper's Form 4: a hierarchical category string, one query-selected section, and five query-selected keywords, produced offline by an LLM and cached. This cuts per-document tokens from roughly 200 for full text to tens, which is what lets the same context window hold 200 candidates instead of 20. The pipeline pairs this with a coarse-to-fine listwise reranking design: the LLM ranks the 200 summaries, the top 20 move to a second prompt with full text, and that second ranking is the final output. Adaptive selection uses embedding cosine similarity to keep only the section and keywords most relevant to the query.","core_discovery":"The central claim is that a coarse reranking pass over compact LLM-extracted features can expand the candidate pool by an order of magnitude, and a second full-text pass then restores the precision lost to compression. Concretely, CoRank takes 200 first-stage candidates, replaces each document with category, section, and keywords, uses an LLM to listwise-rank those summaries, keeps the top 20, and then listwise-ranks those 20 on full text. The paper reports that this beats vanilla full-text reranking over 20 documents on every tested dataset and backbone, and that both the adaptive selection of query-relevant sections and keywords and the fine-grained full-text stage contribute. It frames the result as evidence that information extraction and retrieval are synergistic: structured features improve coverage without sacrificing final ranking accuracy.","pith_inferences":["A testable transfer: the same breadth-then-depth recipe should help in any long-document search domain with imperfect first-stage retrieval, such as patents, legal opinions, or clinical notes, since the bottleneck is context length rather than scientific text specifically.","If coverage is the active mechanism, CoRank's edge should shrink as first-stage retrieval improves; an adaptive variant could size the coarse pool from the retriever's estimated recall.","The paper's human evaluation suggests extraction quality is high on a small sample, which leaves open whether stronger extraction would further improve reranking or whether the coarse stage already saturates the available signal."],"forward_implications":["The same LLM can consider ten times more candidates without extra model capacity or training, because the context budget is spent on coverage instead of per-document detail.","CoRank's gains are not tied to one retriever or one model family: the paper reports improvements across four different first-stage retrievers and with both open and proprietary LLM backbones.","Sliding-window expansion and CoRank stack, so the two breadth-increasing strategies can be combined rather than treated as alternatives.","Because feature extraction happens offline and is cached, the extra coverage comes with limited per-query latency and roughly 40% of the token cost of sliding windows."],"supporting_citations":[{"why":"Defines the vanilla zero-shot listwise reranking setup and the small 20-document context budget that CoRank targets.","marker":"[68]"},{"why":"Supplies the listwise prompting formulation and the sliding-window strategy used as a baseline and as a complementary method.","marker":"[38]"},{"why":"Provides the evaluation protocol, including the choice of first-stage retriever and similarity encoder for the pipeline.","marker":"[10]"},{"why":"Supplies the dense retriever used to generate first-stage candidates and to score feature relevance for adaptive selection.","marker":"[18]"},{"why":"LitSearch is the scientific retrieval benchmark used for the token-cost analysis, representation comparison, and most ablations.","marker":"[1]"},{"why":"Supplies the open-source LLM backbone and the small model used for offline document feature extraction.","marker":"[53]"}],"fun_headline_variants":["Rerank 200 via summaries, then refine top 20","Two-stage rerank: summaries first, full text second","Expand rerank pool 10x with compact features","CoRank lifts nDCG from 50.6 to 55.5","Summaries boost rerank coverage, full text restores precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-extracted category, section, and keywords keep enough query-relevant signal that ranking 200 summaries finds relevant papers a 20-document full-text ranking would miss; if the extraction discards the decisive cues, the wider pool cannot compensate, and the paper's own Section 3.3.3 reports that summaries underperform full text when the document count is held fixed.","fun_headline_variants_meta":{"raw":{"variants":["Rerank 200 via summaries, then refine top 20","Two-stage rerank: summaries first, full text second","Expand rerank pool 10x with compact features","CoRank lifts nDCG from 50.6 to 55.5","Summaries boost rerank coverage, full text restores precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1386,"prompt_tokens":998,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":614,"tokens_out":388,"duration_ms":3890,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:31.114220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoRank on a benchmark where the first-stage retriever already has perfect recall within its top 20: if CoRank still beats vanilla full-text reranking over the same 20 documents, the gain is not coming from candidate coverage; equivalently, compare the coarse stage's recall at 20 against the retriever's recall at 20 and check whether the extra pool is actually adding relevant documents.","supporting_citations":[],"review_version":1}