{"id":"3e75cfbd-88ab-49f2-b43e-ea9885aa7a7f","arxiv_id":"2412.07713","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A proposed benchmark with diagnostic datasets slices citation recommendation performance by field, year, citation count, context length, position, intent, and part of speech, with preliminary BM25 and transformer results.","lead":"This paper proposes a benchmark with diagnostic datasets for comparing citation recommendation models, built from the S2ORC and S2AG corpora, and reports recall and ranking scores for five models. The resource is intended to give researchers a common way to test how well citation recommenders handle different citation contexts, but the paper is written largely as a proposal and lacks key evaluation details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded: Section 4 admits BM25 searches the entire S2AG index while the neural models' candidate pools are unspecified, so BM25's 'best performance' in Table 1 is not a standardized evaluation of model quality.","rationale":"Reader's rejection is justified. I focus on a different load-bearing weakness than the reader's formal weakest_assumption. The reader's named assumption about S2ORC annotation quality is relevant, but the paper's own limitation section ('manual check of the accuracy') acknowledges quality control, and the diagnostic utility could survive some annotation noise. The stronger defect is the evaluation protocol: the central empirical claim, BM25 being best, is admitted to rest on BM25 searching 'the entire S2AG dataset,' whereas the neural models' candidate sets are unspecified. Without a shared candidate space, R@10 and MRR are not comparable quantities. This is not merely incomplete reporting; it directly undermines the benchmark's purpose of comparing models. The abstract and contributions also use future/proposal language ('propose developing,' 'aim is to curate'), and Section 5 lists missing standard framework and metrics, corroborating that the benchmark is not yet established. Therefore the verdict remains REJECT and no adjustment is needed.","tokens_in":16234,"tokens_out":3239,"duration_ms":32790,"concrete_test":"Use the released datasets and reproduce Table 1 with one controlled candidate pool per test sentence: define C as the union of the true cited paper and a fixed set of distractors sampled from S2AG under the same distribution for all systems, then evaluate BM25, NCN, LCR, and Galactica over that identical C and recompute R@10/MRR. If BM25's advantage largely disappears or reverses under equal candidate spaces, the comparison is an artifact of search-space asymmetry; if it persists, the concern is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's empirical support for the central claim is the statement that 'BM25 shows the best performance in terms of Recall and MRR for most datasets,' and the paper itself explains this by saying BM25 'has the advantage of querying the entire S2AG dataset, which comprises millions of records.' For a benchmark whose stated purpose is standardized comparison, Recall@10 and MRR@10 are only comparable if every model ranks over the same defined candidate set. No candidate-pool size, index, or negative-sample construction is reported for NCN, LCR, or Galactica; LCR and NCN are described merely as trained on Arxiv from different years, and Galactica is used as a language model without an explicit retrieval protocol. Thus every BM25-versus-neural score in Table 1 confounds retrieval breadth with model quality. The diagnostic datasets may still be useful, but the headline empirical conclusion and the claim that a standardized benchmark is established do not follow from the reported evidence. This is an internal-consistency problem, not a disagreement with field consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a benchmark for local citation recommendation. Using S2ORC and S2AG, the authors construct eight diagnostic datasets designed to isolate different features of the citation context (field, publication year, citation count, context length, citation location, citation intent, POS of surrounding words, and low-resource fields) and evaluate four model families (NCN, LCR, Galactica at three sizes, and a BM25 baseline) with Recall@10 and MRR@10 across all classes. Section 4 reports that BM25 achieves the best Recall and MRR for most datasets, and the Conclusion asserts that a comprehensive benchmark for evaluating and comparing citation recommendation models is established.","tokens_in":16487,"tokens_out":10259,"duration_ms":84304,"significance":"Standardized, diagnostic benchmarks are genuinely needed in citation recommendation, and the paper identifies a sensible, literature-motivated decomposition of citation-context features; the eight dataset dimensions, the inclusion of multiple Galactica sizes, and the release of code and data (GitHub and Google Drive links in Section 1) are useful steps. If the datasets, protocol, and comparison were fully specified, this could become a reusable evaluation resource. As it stands, however, the central claims cannot be assessed: dataset sizes are not reported anywhere, the evaluation protocol is not reproducible from the text, the headline model comparison is admitted in Section 4 to be confounded by unequal retrieval pools, and no error analysis is provided despite the paper's stated analytic goal. The paper is therefore better read as a description of a benchmark pipeline than as an established benchmark with verified results.","major_comments":[{"comment":"The headline comparison is confounded by unequal candidate pools. The paper's own explanation — \"BM25 demonstrates superior overall performance compared to other models, primarily because it has the advantage of querying the entire S2AG dataset, which comprises millions of records\" — concedes that BM25 ranks over the full S2AG index while no candidate pool, negative-sample set, or index is reported for NCN, LCR, or the Galactica models. Because Recall@10 and MRR@10 are computed over whatever candidate set each system returns, the Table 1 finding that \"BM25 shows the best performance in terms of Recall and MRR for most datasets\" does not support any conclusion about model quality and contradicts the benchmark's stated purpose of standardized comparison. The evaluation must be re-run with identical candidate pools for all models (or with BM25 restricted to each model's pool).","section":"§4 and Table 1"},{"comment":"No dataset sizes are reported. The sampling procedure states an upper limit of 10 papers per class group (step 4), but the final number of cited papers, citing sentences, and sentences per class per diagnostic dataset are never given, and the Limitations section concedes that \"The size of the diagnostic dataset could be more significant.\" Without per-class sample counts, the Table 1 metrics are uninterpretable (e.g., the Fields/Biology gap of 0.195 vs 0.1094 between BM25 and the best neural model could be sampling noise on a small test set), and the released resource cannot be evaluated by other researchers. The paper must report total and per-class sizes, the number of samples removed at each filtering stage, and ideally confidence intervals for the reported metrics.","section":"§3.2.1 and §5"},{"comment":"The evaluation protocol is underspecified to the point of irreproducibility. The text does not say how the extracted sentence with <REF> is encoded for each model, how candidate papers are generated for NCN, LCR, and Galactica, what negative samples or context-window sizes are used (prior work in §2 uses 100, 200, 600, or 2048 tokens), or what inference hyperparameters are applied. In addition, Table 1 contains many zero entries — Galactica 125m is 0.0 for nearly every class — but the paper neither discusses these nor provides the error analysis that its diagnostic-dataset framing (Section 3) promises. The authors should document the full protocol and add an error analysis that accounts for the zeros.","section":"§4 and Table 1"},{"comment":"The validity of all downstream scores rests on S2ORC's in-text citation annotations linking each citation mention to the cited paper's corpus ID, but no quality check is reported. The paper itself documents malformed citations such as \"(Lee et al., 1997a(Lee et al., 1997b)\", and the only manual validation mentioned is for the POS dataset (§3.1.5) plus a general reference to manual checking in Section 5, with no agreement or error statistics. The authors should report the fraction of candidate sentences rejected by the citation parser, the rate at which S2ORC annotations were found to be incorrect, and any manual validation results, since a noisy mapping would propagate into every dataset and every score in Table 1.","section":"§3.2.2"}],"minor_comments":[{"comment":"Throughout Section 3 the text mixes future and past tense, e.g., \"The position will then be normalized\" in Section 3.1.2 and \"a dataset will be created\" in Section 3.1.5, which is confusing given that Section 4 reports completed experiments; the manuscript should be rewritten in a consistent tense that reflects what was actually done.","section":"§3"},{"comment":"In the Citation Count 8 row for Galactica 6.7b, the entry \"0.0363 00170\" appears to have a missing decimal point and should presumably read \"0.0363 0.0170\".","section":"Table 1"},{"comment":"Reference [3] is misattributed; the author name \"Bridger Waleed Ammar\" should be \"Waleed Ammar\".","section":"References"},{"comment":"The class boundaries for context length (short/medium/long) and citation location (first/middle/last) are never stated; the paper should report the actual token ranges and normalized-position cutoffs derived from the mean and standard deviation.","section":"§3.1.1 and §3.1.2"},{"comment":"The sentence \"Additionally, BM25 achieves higher MRR across all classes and for BM25, except for the preceding Preposition and Others categories\" is unclear about which model's POS classes are meant.","section":"§4"},{"comment":"The explanation that Galactica's weaker performance on citations at the start or middle of a sentence arises from \"the Transformer model's decoder, which retains information about the preceding context before the masked token\" is not an accurate description of a decoder-only causal language model and should be revised.","section":"§4"},{"comment":"The Conclusion's claim that a comprehensive benchmark is \"established\" is undercut by the Limitations section's own admissions of small dataset sizes, a limited metric set, incomplete model coverage, and the absence of a standard evaluation framework; the conclusion should be qualified accordingly.","section":"§6"}],"recommendation":"reject","confidential_remarks":"To the editor: the most serious problem is internal rather than a matter of field consensus: Section 4 explicitly attributes BM25's superior Table 1 performance to its access to the full S2AG index, so the main empirical claim is acknowledged in the manuscript itself to be confounded. Combined with the complete absence of dataset-size reporting and the underspecified protocol, the paper's central claim of an established, standardized benchmark is not supported by the reported evidence. The diagnostic design idea is reasonable and the resource may be salvageable, but addressing the concerns would require re-running all models over matched candidate pools, reporting full dataset statistics and filtering counts, documenting the protocol, and adding error analysis — substantial additional experimentation rather than a localized revision. On the current evidence I would not invite a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a proposal for a benchmark, not a benchmark that has been built and validated. The one piece worth keeping is the eight-dimensional taxonomy of citation context (field, year, citation count, context length, location, intent, POS, low-resource); that is a reasonable decomposition, and the sampling design is described in more detail than most papers. The rest—the 'results' table and the conclusion that a comprehensive benchmark is established—does not hold up.\n\nThe paper is honest about its own incompleteness: the abstract says 'propose developing,' Section 3 repeatedly uses future tense, and the Limitations section admits the dataset is small, the model coverage is thin, and the metric set is narrow. Good. But the Conclusion then claims the benchmark is established, and that mismatch is not a small thing.\n\nThe load-bearing problem is the evaluation. As Section 4 itself states, BM25 queries the entire S2AG index while the candidate pools for NCN, LCR, and Galactica are never defined. So BM25's higher Recall/MRR in Table 1 likely reflects retrieval breadth, not model quality. That confound invalidates the central empirical claim. The stress-test note on this is exactly right.\n\nOther soft spots: no dataset sizes anywhere, so we can't tell whether the 'datasets' are a few hundred sentences or millions. The citation annotation quality of S2ORC is taken as ground truth, with no manual check or agreement measure; the paper says a manual check was needed for POS but doesn't report it. There is no error analysis, so the diagnostic value of the datasets is asserted, not demonstrated. The links to code and data are there, but I can't verify them from the text.\n\nWhat's genuinely good: the handling of citation parsing (the broken parentheses example), the stratified sampling within fields/years/citation counts, and the decision to fix context length and location as controlled conditions per diagnostic dataset. Those are careful choices, and someone building a real benchmark could use them.\n\nBottom line: this is a useful preliminary design document, but it is not a finished benchmark and it should not be cited as establishing one. I would not send it to peer review as-is. There is a clear path: complete the dataset construction, report sizes and quality checks, standardize candidate pools for all models, and release the code/data. If the author does that, the taxonomy alone might merit a workshop paper.","headline":"A careful proposal for a citation-recommendation benchmark with a useful taxonomy, but the evaluation is confounded and the paper is not a finished benchmark.","tokens_in":16962,"tokens_out":2690,"would_cite":false,"duration_ms":26196,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes a standardized benchmark for local citation recommendation and reports that BM25, the simplest method, achieves the best Recall@10 and MRR@10 on most of its diagnostic datasets.","keywords":["citation recommendation","benchmark","diagnostic datasets","S2ORC","citation context","BM25 baseline","Recall@10","MRR@10"],"falsifier":"Re-run the dataset construction after manually verifying a random sample of S2ORC citation links, or after replacing the citation parser with a stricter one, and compare the results to Table 1; if BM25's lead over LCR and Galactica shrinks or reverses, the headline result is an artifact of noisy citation mapping. More directly, ask annotators whether the masked sentence alone identifies the cited paper; if it often does not, the diagnostic task is not measuring what the benchmark claims.","tokens_in":16044,"feed_emoji":"📚","tokens_out":6012,"duration_ms":50166,"temperature":0.7,"pith_summary":"Citation recommendation systems are built on many different datasets, context definitions, and metrics, so results across papers are hard to compare. This paper argues that the field needs a standardized benchmark and delivers one: a set of diagnostic datasets, derived from the S2ORC and S2AG corpora, each isolating a single feature of the citing context such as sentence length, citation position, citation intent, part-of-speech surroundings, field, year, popularity, or low-resource status. On these datasets the paper evaluates four models—two neural local recommendation systems, the Galactica language models, and a BM25 baseline—and reports Recall@10 and MRR@10 in a uniform way. The main empirical result is that BM25, despite being the simplest method, achieves the best recall and MRR on most datasets, with neural models winning on specific classes such as sentence length and intent. If the benchmark is used by the community, model comparisons become meaningful and the conditions under which complex models actually help become visible.","feed_headline":"Simple search beats citation recommenders on benchmark tests","feed_subtitle":"Diagnostic datasets isolate context length, location, intent, and more, so strengths and weaknesses become visible.","key_machinery":"The central object is the set of eight diagnostic datasets built by stratified sampling from the S2ORC full-text corpus and S2AG metadata. For each dataset, a single feature of the citation context is isolated—field, publication year group, citation-count decile, context length class, normalized citation position, citation intent, part-of-speech of the preceding or following word, or low-resource field—while other features are held fixed; for example, the length dataset keeps the citation at the end of the sentence. The extraction pipeline samples papers hierarchically by field, year, and citation count, parses sentences with a scientific text-processing pipeline, keeps only sentences containing exactly one citation, verifies the citation with a regular-expression citation parser, replaces the citation with the <REF> placeholder, and assigns the sentence to a class. These datasets are what make the evaluation diagnostic: a model's score on a given dataset shows how well it handles that specific contextual feature.","core_discovery":"On its own terms, the paper claims to establish the first comprehensive benchmark for evaluating citation recommendation models, focused on local (context-based) recommendation. The load-bearing empirical claim is that BM25 ranking, a classical lexical retrieval baseline, outperforms the neural and large-language-model alternatives on most diagnostic datasets in both Recall@10 and MRR@10, and that the remaining wins are concentrated in specific classes: LCR beats BM25 on recall for the context-length and citation-intent datasets, while BM25 dominates on fields, years, citation-count groups, and the low-resource set. The paper also reports feature-level patterns, such as neural models doing better on short contexts and BM25 doing better when the citation appears at the start of the sentence. The contribution is therefore twofold: a reusable evaluation resource with controlled context features, and an initial comparative result showing where simple retrieval still wins.","pith_inferences":["The author's explanation that BM25 queries the full S2AG metadata suggests a testable separation: ablating BM25's access to metadata would reveal how much of its lead comes from the citation context itself versus from matching titles and other bibliographic fields.","The same diagnostic construction could be extended to global citation recommendation, where the input is the whole paper rather than a single sentence; the paper notes such extensions as future work.","The benchmark's usefulness depends on the accuracy of S2ORC's citation annotations; an independent re-annotation study of a random sentence sample would show whether the reported rankings are stable under annotation noise.","Because the paper reports only Recall@10 and MRR@10, adding nDCG and F1 would test whether the ranking conclusions survive a change of metric—something the author lists among the limitations."],"forward_implications":["Researchers can evaluate any new citation recommendation model on the same diagnostic datasets and the same Recall@10 and MRR@10 metrics, making published results directly comparable.","BM25 becomes a default baseline that more complex models must beat; its strong performance means simple lexical retrieval is the bar, not the floor.","Models can be profiled by feature, e.g., LCR's recall advantage on length and intent datasets and BM25's advantage on low-resource fields point to where architectural investment pays off.","Because each dataset controls for other context features, differences in scores can be attributed to the feature under study rather than to accidental dataset variation."],"supporting_citations":[{"why":"Supplies the full text and in-text citation annotations used to build all diagnostic datasets.","marker":"[25]"},{"why":"Supplies paper metadata and citation-intent labels used to define dataset classes.","marker":"[21]"},{"why":"NCN, one of the four evaluated models in the benchmark comparison.","marker":"[12]"},{"why":"LCR, the hierarchical-attention and SciBERT-reranking model evaluated in the comparison.","marker":"[16]"},{"why":"Galactica, the large-language-model family evaluated in the comparison.","marker":"[35]"},{"why":"GLUE, the template for standardized multi-task evaluation that motivates the benchmark design.","marker":"[40]"},{"why":"Cited to motivate the need for cohesive benchmarks and to warn against benchmark bias.","marker":"[11]"}],"fun_headline_variants":["BM25 beats neural citation recommenders in new benchmark","Simple baseline tops citation recommenders in benchmark tests","New benchmark shows BM25 wins on most citation tasks","Benchmark reveals neural citation models lag simple search","Citation benchmark: BM25 outperforms neural models on most tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's scores inherit whatever errors are in S2ORC's in-text citation links: if a citation mention is mapped to the wrong paper, or the one-citation sentence is too ambiguous to identify the intended reference, every diagnostic dataset and score is affected.","fun_headline_variants_meta":{"raw":{"variants":["BM25 beats neural citation recommenders in new benchmark","Simple baseline tops citation recommenders in benchmark tests","New benchmark shows BM25 wins on most citation tasks","Benchmark reveals neural citation models lag simple search","Citation benchmark: BM25 outperforms neural models on most tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1193,"prompt_tokens":931,"completion_tokens":262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":547,"tokens_out":262,"duration_ms":2837,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:33:57.921733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the dataset construction after manually verifying a random sample of S2ORC citation links, or after replacing the citation parser with a stricter one, and compare the results to Table 1; if BM25's lead over LCR and Galactica shrinks or reverses, the headline result is an artifact of noisy citation mapping. More directly, ask annotators whether the masked sentence alone identifies the cited paper; if it often does not, the diagnostic task is not measuring what the benchmark claims.","supporting_citations":[{"cited_title":"Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic","cited_arxiv_id":null,"evidence_quote":"Galactica, the large-language-model family evaluated in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the full text and in-text citation annotations used to build all diagnostic datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCN, one of the four evaluated models in the benchmark comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LCR, the hierarchical-attention and SciBERT-reranking model evaluated in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GLUE, the template for standardized multi-task evaluation that motivates the benchmark design."}],"review_version":1}