DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval

Aleksandar Pramov; Anthony Miyaguchi; Imran Afrulbasha

REVIEW 3 major objections 7 minor 16 references

DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On a French web corpus, reranking drives NDCG@10 gains while LLM query expansion backfires, and performance drops sharply for snapshots older than a year.

desk verdict Solid reranking result on LongEval French data, but the temporal 'regime change' is not supported by the evidence. read the letter →

arxiv 2507.08360 v1 pith:B2JDARCV submitted 2025-07-11 cs.IR

Anthony Miyaguchi , Imran Afrulbasha , Aleksandar Pramov This is my paper

classification cs.IR

keywords TemporalDriftTwo-StageRetrievalBM25QueryExpansionCross-EncoderRerankingNDCGTopicModelingLongEval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports what happens to a two-stage web search pipeline when it is run against the LongEval French web corpus, which is collected in monthly snapshots from June 2022 to August 2023. The authors compare four systems: BM25 alone, BM25 with LLM query expansion, BM25 with a cross-encoder reranker, and the full combination, then ablate the stages. The central finding is that reranking is the main driver of performance, since removing it costs 0.11 in mean NDCG@10, while removing query expansion actually improves the score by 0.01. The paper also documents a temporal regime change in which all systems score far lower on snapshots older than roughly a year, interpreting this as evidence of temporal drift in the underlying data. A sympathetic reader would care because the result suggests simple, reranker-based pipelines beat LLM-heavy query expansion on evolving corpora and that recency is baked into the evaluation.

What carries the argument

The mechanism that carries the argument is the two-stage retrieval pipeline: BM25, implemented through Pyserini, retrieves the top 100 candidate documents, and a French cross-encoder (antoinelouis/crossencoder-camembert-base-mmarcoFR) reranks them; a Gemini LLM generates optional query expansions inserted before the BM25 stage. The comparison across the four submitted systems is an ablation that isolates the contribution of each stage, with mean NDCG@10 as the yardstick. NMF and LDA topic models over the corpus act as a secondary exploratory lens, used to connect visible topic shifts, such as fuel prices and elections, to the NDCG curve over time.

What would settle it

Re-run the four systems on the 2022-06 through 2022-09 snapshots after re-collecting or filtering non-French documents and fixing the intermittent failure that produced the NULL for 2022-08; if NDCG@10 for those months rises to the 0.3+ range seen from October 2022 on, the regime-change claim is falsified. Computing monthly NDCG with a fixed query set would further show whether the cliff is corpus-side or query-side.

Watch

Extended reading notes

Core claim

On the LongEval-Retrieval task, the team's best configuration, BM25 candidate retrieval followed by a French cross-encoder reranker without query expansion, achieves a mean NDCG@10 of 0.296 across the full training and test period and a best monthly score of 0.395 in 2023-05. The four-way ablation shows reranking provides most of the gain: removing the reranker lowers average NDCG@10 by 0.11, while removing the expanded queries raises it by 0.01. The paper concludes that pretrained rerankers supply relevancy signals that keyword result sets lack, that this particular LLM query expansion loses frequency-adjusted keywords, and that the reranker's semantic space can partially compensate for those losses. It further reports a regime change around September 2022, one year before the test window ends, with all four systems degrading on older partitions, and offers non-French documents and event-driven queries as unverified explanations. In the exploratory analysis, NMF topic groupings separate more clearly over time than LDA groupings and align with the October 2022 NDCG spike.

Load-bearing premise

The temporal-regime-change conclusion rests on the assumption that the low early-month scores reflect genuine drift in the web corpus rather than artifacts of the pipeline or dataset collection; the paper itself flags a processing failure and unverified alternative explanations such as non-French documents and event-driven queries.

Editorial extensions

If this is right

A pretrained French cross-encoder improves BM25 results on every monthly snapshot without any fine-tuning, so reranking is a portable and cheap upgrade for keyword retrieval.
LLM query expansion as configured here is counterproductive, losing frequency-adjusted keywords; only the reranker's semantic space partially rescues the expanded result set.
All four systems share the same sharp drop before October 2022, so performance on snapshots older than about a year cannot be extrapolated from recent scores; recency bias is structural to this kind of static evaluation.
The reported cost data, about 21 GPU-hours to embed a single month's documents and roughly $2,500 to read the full training set as LLM context, makes dense retrieval or corpus-wide LLM processing a resource-planning decision rather than a default choice.
NMF produces more separable temporal topic clusters than LDA on this French corpus, indicating that model choice and preprocessing determine whether topic drift is visible at all.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not separate query drift from document drift; one could hold a fixed query set across all monthly snapshots and recompute NDCG. If the September 2022 cliff persists, the corpus changed; if it flattens, the anomaly is query-side.
Because the best system omits query expansion, a direct practical extension is to drop the LLM stage entirely, which would also sidestep the content-filter failures the paper hit when an external API refused explicit queries.
The regime-change claim could be stress-tested by cleaning the early partitions, re-running the failed 2022-08 batch, and filtering non-French documents; the paper itself records a NULL for that month and lists non-French content as a possible cause.
The NMF topic clusters that align with the October 2022 spike, notably fuel prices and elections, suggest a testable hypothesis: score each topic's queries separately to see whether temporal NDCG spikes are driven by event-salient topics rather than global system behavior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

Quick take: this is a competent CLEF working note with one solid empirical finding and one overreached conclusion. The solid part: on the 2025 LongEval French Qwant data, a standard BM25 + cross-encoder rerank pipeline lifts mean NDCG@10 from 0.242 to 0.296, and the gain holds across months. The Gemini query expansion does not help; in the full pipeline it makes things very slightly worse, and alone it is clearly worse than plain BM25 (0.194 vs 0.242). That's a useful data point for anyone building on this collection.

What's new is narrow: specific ablation numbers on the 2025 French collection, plus some descriptive topic-modeling comparison (NMF more interpretable than LDA here). The topic modeling is clearly labeled exploratory and not load-bearing. The paper is honest about pipeline failures (a NULL month, content-filtering issues forcing a switch to DeepSeek). The code is on GitHub, though without a commit hash and with external API dependencies.

Soft spots, in order of size. First, the temporal 'regime change' claim in Sections 7 and 9 is not established. The drop before 2022-09 could be a change in corpus composition, event-driven queries, or the same kind of processing failure that produced the NULL in 2022-08. The authors themselves say it's 'challenging to explain' and offer two unverified hypotheses, yet the Conclusions state it as a finding. That's overreach. The reranking result does not depend on it. Second, the ablation effect of removing query expansion is reported as '0.01' but Table 6 shows 0.001 (0.296 vs 0.295). That's a factor-of-ten slip and should be corrected. Third, there are no variances or significance tests on the key differences; the table's 'Std Dev' numbers look like across-query spread and don't speak to the ablation contrast. Minor: one month is missing for one system.

Bottom line: the central NDCG comparison is credible and the write-up is honest. The temporal conclusion needs to be downgraded to conjecture. This deserves peer review as a working note; a serious referee should ask for the 0.01 fix and a softened 'regime change' claim.

Referee Report

3 major / 7 minor

Summary. This working-notes paper reports DS@GT's participation in the LongEval-Retrieval task at CLEF 2025. The authors build a two-stage retrieval pipeline: BM25 over a French Qwant snapshot collection, optional query expansion using Gemini (with DeepSeek R1 for a final batch), and optional reranking with a French Camembert-based cross-encoder. Four submitted systems are compared by NDCG@10, both aggregated and per month, and the paper also presents LDA/NMF topic-modeling visualizations over time. The main reported findings are that reranking yields the largest improvement, query expansion slightly hurts, and retrieval performance drops sharply for partitions before October 2022, which the authors call a temporal anomaly.

Significance. The paper has a genuine strength: it reports a clean ablation on an externally judged, temporally distributed benchmark, and it ships source code, making the reranking benefit (BM25 reranked 0.296 vs 0.242 mean NDCG@10) reproducible. The cost and throughput estimates for embedding the corpus are practically useful for future dense-retrieval experiments. The significance is limited, however, by three issues: the central ablation numbers are misreported in the text; no variance or significance testing is given for monthly or aggregated effects; and the temporal-regime conclusion is asserted despite being acknowledged as unexplained and being confounded by a pipeline failure and possible corpus-composition changes.

major comments (3)

[Section 6.1, Table 6] The text states that removing the reranking stage reduces performance by 0.11 and that removing query-expanded results increases the score by 0.01. These values are not supported by Table 6: the reranking ablation for the full system is 0.295 - 0.194 = 0.101, and the query-expansion effect in the reranked condition is 0.296 - 0.295 = 0.001. Please correct the arithmetic, and state whether the 2022-08 NULL is excluded from the aggregate means in Table 6.
[Table 6, Table 7] All conclusions about the relative performance of the four systems are based on means without any measure of uncertainty. Because every system is evaluated on the same queries and topics, paired query-level tests (e.g., bootstrap or permutation over queries, or per-query confidence intervals) are needed to support claims such as 'the performance pattern holds generally over time' and to assess whether the 0.001 difference between bm25-reranked and bm25-expanded-reranked is distinguishable from noise. At a minimum, the paper should report the number of queries per month and per-query standard errors.
[Sections 7 and 9, Table 7] The conclusion that 'query performance degrades in partitions older than a year' is presented as a main finding, but the paper's own text says the shift is 'challenging to explain' and lists two unverified alternative explanations (non-French documents and temporally salient events); there is also an intermittent processing failure for 2022-08. A drop in scores before 2022-09 is equally compatible with an indexing/analyzer artifact or a corpus-composition change. To retain this claim, the authors need a diagnostic (e.g., per-month language distribution, per-topic degradation, or a reprocessed 2022-08 run), or the claim must be reclassified as an open hypothesis rather than a conclusion.

minor comments (7)

[Section 3.2, Table 2] The text says there are about two million documents per time-step, but Table 2 reports 19,000,580 documents for the whole train split; please report document counts per date or correct the wording.
[Equation (4)] The relative NDCG drop RND uses lag6 and lag8, but neither of these variables is defined or used elsewhere; please define the lags or remove the equation.
[Table 6] The column heading 'nDCG@10 Std Dev' should state what the standard deviation is computed over (queries, months, or runs) so readers can interpret the dispersion.
[Sections 4.2.1 and 4.2.2] The LDA/NMF preprocessing and hyperparameters (sample size, number of iterations, random seeds, alpha/beta for LDA) are not given; the EDA claim that NMF gives more meaningful clusters is qualitative and would benefit from quantitative coherence metrics, as the authors themselves note.
[Section 5.1, Figure 4] No details are given for BM25 parameters (k1, b) or Lucene analyzer configuration, which are needed to reproduce the exact Pyserini runs.
[Section 7] The use of DeepSeek R1 for the last 1/100th of queries introduces a model-mix confound in the query-expansion condition; given its small size this is unlikely to change the main conclusion, but it should be disclosed in the method section and ideally in the table caption.
[Section 4.2.2, Figure 2] The alleged correspondence between an NMF topic cluster and the NDCG spike around October 2022 is anecdotal; if intended as evidence, it needs a quantitative measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all results come from external LongEval relevance judgments and ablations on the same external data, with no fitted input renamed as a prediction.

full rationale

The paper's central claims are empirical comparisons of four fixed retrieval pipelines against the official LongEval relevance judgments. The reranking gain, the query-expansion loss, and the temporal performance pattern are all read directly from NDCG@10 scores in Tables 6 and 7, which are produced by running the submitted systems on the externally provided benchmark. No parameter is fitted to a subset of that data and then reported as a prediction of a closely related quantity; the Cross Encoder reranker is pre-trained and explicitly not fine-tuned, and the LLM query-expansion hyperparameters are left at API defaults. The topic-modeling section is exploratory and does not feed back into the retrieval results in a way that would make the evaluation self-referential. References [1] and [10] describe the LongEval task and dataset and are authored by the lab organizers, not by the present paper's authors, so there is no load-bearing self-citation chain. The temporal-anomaly conclusion is admittedly 'challenging to explain' and the authors offer unverified hypotheses about non-French documents or event-driven queries; that is a weakness in evidentiary support or correctness risk, not circularity, because the conclusion is an empirical observation on external judgments rather than a derivation from premises that assume it. No equation is defined in terms of the result it is used to establish, and no known empirical pattern is merely renamed. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The retrieval scores rest on the official LongEval labels and on the choice of pre-trained reranker; the topic-modeling section adds hand-chosen hyperparameters (k=20). No free parameters are fitted to the target metric, so the central NDCG results are not circular, but the EDA conclusions are sensitive to the selected topic count and sample.

free parameters (3)

Topic count (k=20) = 20
Chosen by hand for both NMF and LDA in Section 4.2.1; affects the topic-modeling EDA, not the IR scores.
BM25 candidate pool size = 100
Top 100 documents retrieved per query (Section 5.1); sets the recall ceiling for the reranker.
Topic-modeling sample size
Random sample of documents used for NMF/LDA, size not specified in Section 4.2.1; affects topic quality and runtime.

assumptions (3)

domain assumption The LongEval relevance labels, derived from click-model alpha estimates, are a valid ground truth for nDCG@10 evaluation.
Section 3.1 describes the click-model based relevance; all scores depend on these labels.
domain assumption The pre-trained cross-encoder (antoinelouis/crossencoder-camembert-base-mmarcoFR) transfers to this French query-document distribution without fine-tuning.
Section 6.1 states that no fine-tuning is performed; the reranking improvement claim rests on this.
ad hoc to paper The early-month low scores reflect temporal drift rather than pipeline or data-collection artifacts.
Section 7 offers hypotheses but no confirmation, and Table 7 already shows one pipeline failure in 2022-08.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval." pith.science (2026). https://pith.science/paper/B2JDARCV

@misc{pith2026250708360,
  author       = {Pith},
  title        = {Pith review of: DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2JDARCV}},
  note         = {Machine review of arXiv:2507.08360}
}

read the original abstract

Information Retrieval (IR) models are often trained on static datasets, making them vulnerable to performance degradation as web content evolves. The DS@GT competition team participated in the Longitudinal Evaluation of Model Performance (LongEval) lab at CLEF 2025, which evaluates IR systems across temporally distributed web snapshots. Our analysis of the Qwant web dataset includes exploratory data analysis with topic modeling over time. The two-phase retrieval system employs sparse keyword searches, utilizing query expansion and document reranking. Our best system achieves an average NDCG@10 of 0.296 across the entire training and test dataset, with an overall best score of 0.395 on 2023-05. The accompanying source code for this paper is at https://github.com/dsgt-arc/longeval-2025

Figures

Figures reproduced from arXiv: 2507.08360 by the authors.

**Figure 2.** Comparison of topic model distributions from June 2022 to February 2023 in two-month intervals, i.e., 2022-06, 2022-08, 2022-10, 2022-12, and 2023-02. Each subfigure (a-d) shows the temporal evolution of topics for a given model (NMF/LDA) and dimensionality reduction technique (PCA/GRP). 4.2.2. Discussion Figures 1a and 1b show topic groupings for equally spread out subsequences in the overall collection spanning Ju… view at source ↗

**Figure 3.** The overall architecture for the two-stage evaluation system. Our information retrieval engine for document ranking is a preliminary attempt to handle large-scale data processing within high-performance computing environments. We use a standard two-stage retrieval pipeline to find relevant documents for each query. Queries are expanded to include a larger [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The prompt and JSON schema used for query expansion. (a) The prompt instructs a Large Language Model to generate French query expansions suitable for a hybrid search system using BM25 and vector search. (b) The required output schema enforces a structured response cont…

**Figure 5.** Figure 5: NDCG performance over time by experimentation. Note that there is a recency bias with a regime change around 2022-09. We also note that during one of the batches of query expansion prompting, we encountered contentbased restrictions due to the appearance of explicit c…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

[1]

Alkhalifa, H

R. Alkhalifa, H. Borkakoty, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, T. Fink, L. Goeuriot, D. Iommi, P. Galuščáková, G. Gonzalez-Saez, M. Liakata, H. T. Madabushi, P. Medina-Alias, P. Mulhem, F. Piroi, M. Popel, A. Zubiaga, Extended overview of the clef 2024 longeval lab on longitudinal evaluation of model performance, in: CLEF 2024: Conference and La...

work page 2024
[2]

Campos, G

R. Campos, G. Dias, A. Jorge, C. Nunes, Survey of temporal information retrieval and related applications, ACM Computing Surveys 47 (2016) 15:1–15:41

work page 2016
[3]

Keikha, K

M. Keikha, K. Radinsky, M. de Rijke, Time-sensitive query auto-completion, in: Proceedings of SIGIR, 2011, pp. 593–602

work page 2011
[4]

X. Wei, W. B. Croft, Lda-based document models for ad-hoc retrieval, in: Proceedings of the 29th Annual International ACM SIGIR Conference, 2006, pp. 178–185

work page 2006
[5]

Q. Ai, L. Yang, W. B. Croft, Contextual ranking features for web search, in: Proceedings of the 2016 ACM SIGIR International Conference on the Theory of Information Retrieval, 2016, pp. 43–52

work page 2016
[6]

K. A. Hambarde, H. Proenca, Information retrieval: recent advances and beyond, IEEE Access 11 (2023) 76581–76604

work page 2023
[7]

Clavié, rerankers: A lightweight python library to unify ranking methods, 2024

B. Clavié, rerankers: A lightweight python library to unify ranking methods, 2024. URL: https: //arxiv.org/abs/2408.17344. arXiv:2408.17344

arXiv 2024
[8]

Carpineto, G

C. Carpineto, G. Romano, A survey of automatic query expansion in information retrieval, ACM Computing Surveys 44 (2012) 1–50

work page 2012

Show all 16 references

[9]

URL: https://gemini.google.com, generative AI model

Google, Gemini Large Language Model, 2025. URL: https://gemini.google.com, generative AI model

2025
[10]

Galuščáková, R

P. Galuščáková, R. Deveaud, G. Gonzalez-Saez, P. Mulhem, L. Goeuriot, F. Piroi, M. Popel, Longeval- retrieval: French-english dynamic test collection for continuous web search evaluation, arXiv preprint arXiv:2303.03229 (2023). URL: https://arxiv.org/pdf/2303.03229

2023 arXiv
[11]

Kassab, E

L. Kassab, E. George, D. Needell, H. Geng, N. J. Nia, A. Li, Towards a fairer non-negative matrix factorization, arXiv preprint arXiv:2411.09847 (2024). URL: https://arxiv.org/abs/2411.09847

2024
[12]

learn Developers, Non-negative matrix factorization (nmf) in scikit-learn, 2025

S. learn Developers, Non-negative matrix factorization (nmf) in scikit-learn, 2025. URL: https: //scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html, accessed: 2025-05- 11

2025
[13]

Park, Non-negative matrix factorization: Techniques and applications, in: Advances in Data Analysis, Springer, 2025

H. Park, Non-negative matrix factorization: Techniques and applications, in: Advances in Data Analysis, Springer, 2025. URL: https://faculty.cc.gatech.edu/~hpark/papers/nmf_book_chapter.pdf, accessed: 2025-05-11

2025
[14]

Developers, Luigi: A python module for workflow management, 2025

L. Developers, Luigi: A python module for workflow management, 2025. URL: https://luigi. readthedocs.io/en/stable/, accessed: 2025-05-10

2025
[15]

URL: https://slurm.schedmd

SchedMD, Slurm: Simple linux utility for resource management, 2025. URL: https://slurm.schedmd. com/documentation.html, accessed: 2025-05-10

2025
[16]

Lin, et al., Pyserini: A python toolkit for reproducible information retrieval research, 2025

J. Lin, et al., Pyserini: A python toolkit for reproducible information retrieval research, 2025. URL: https://github.com/castorini/pyserini, accessed: 2025-05-10

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Alkhalifa, H

R. Alkhalifa, H. Borkakoty, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, T. Fink, L. Goeuriot, D. Iommi, P. Galuščáková, G. Gonzalez-Saez, M. Liakata, H. T. Madabushi, P. Medina-Alias, P. Mulhem, F. Piroi, M. Popel, A. Zubiaga, Extended overview of the clef 2024 longeval lab on longitudinal evaluation of model performance, in: CLEF 2024: Conference and La...

work page 2024

[2] [2]

Campos, G

R. Campos, G. Dias, A. Jorge, C. Nunes, Survey of temporal information retrieval and related applications, ACM Computing Surveys 47 (2016) 15:1–15:41

work page 2016

[3] [3]

Keikha, K

M. Keikha, K. Radinsky, M. de Rijke, Time-sensitive query auto-completion, in: Proceedings of SIGIR, 2011, pp. 593–602

work page 2011

[4] [4]

X. Wei, W. B. Croft, Lda-based document models for ad-hoc retrieval, in: Proceedings of the 29th Annual International ACM SIGIR Conference, 2006, pp. 178–185

work page 2006

[5] [5]

Q. Ai, L. Yang, W. B. Croft, Contextual ranking features for web search, in: Proceedings of the 2016 ACM SIGIR International Conference on the Theory of Information Retrieval, 2016, pp. 43–52

work page 2016

[6] [6]

K. A. Hambarde, H. Proenca, Information retrieval: recent advances and beyond, IEEE Access 11 (2023) 76581–76604

work page 2023

[7] [7]

Clavié, rerankers: A lightweight python library to unify ranking methods, 2024

B. Clavié, rerankers: A lightweight python library to unify ranking methods, 2024. URL: https: //arxiv.org/abs/2408.17344. arXiv:2408.17344

arXiv 2024

[8] [8]

Carpineto, G

C. Carpineto, G. Romano, A survey of automatic query expansion in information retrieval, ACM Computing Surveys 44 (2012) 1–50

work page 2012