Pith. sign in

REVIEW 3 major objections 5 minor 84 references

Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Semantic similarity is what users prefer for related-item recommendations in academic search, beating term matching and session-history prediction in a live interleaving test.

desk verdict A useful living-lab case study: the overall embedding-over-lexical ranking is credible, but the category-level claims and the SST assertion don't hold up. read the letter →

arxiv 2607.26380 v1 pith:2D6CLSMG submitted 2026-07-29 cs.IR

Mehmet Deniz Türkmen , Daniel Hienert This is my paper
classification cs.IR
keywords continuousevaluationinterleavingrecommendersystemsacademicsearchsemanticsimilaritytextembeddingssession-basedrecommendationsocialscience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish which recommendation strategy users actually prefer in a living academic search portal: embedding-based semantic similarity, lexical term matching, or session-history prediction. Using continuous online interleaving with real clicks over roughly 23 weeks, it finds a stable preference ordering, with semantic similarity best, click-path session prediction in the middle, and term matching last. The same data shows that the best strategy depends on information type: simpler term matching stays competitive for research data and variable search, where users often look for known items, while semantic similarity dominates for publications and general exploration. If the result holds, the practical consequence is that embedding-based 'related items' should be the default for scholarly search, with lightweight lexical matching as a sensible fallback for known-item categories. The paper also demonstrates that continuous online evaluation can produce a consistent ranking with far fewer comparisons than exhaustive pairwise testing.

What carries the argument

The load-bearing mechanism is team-draft interleaving: for each impression, two recommender systems each produce a ranked list, and the evaluation service merges them into a single displayed list by drafting items from the two lists in randomized order, so that every click can be attributed to the system that supplied the clicked item. Pairwise comparisons are then tested with a two-sided binomial sign test using Holm-Bonferroni correction. The recommenders themselves are standard content-based components: a term-similarity baseline using more-like-this lexical matching, three embedding-based variants that retrieve near neighbours by cosine similarity in a dense vector index, and a session r

What would settle it

Re-attribute the collected clicks while controlling for display position and item popularity; if the pairwise ordering reverses or the nomic advantage disappears after such control, the central claim fails. A dwell-time or follow-up-engagement measure that ranks the systems opposite to click counts would also undercut it.

Watch

Extended reading notes

Core claim

The paper's central claim is a preference ordering among five recommender systems observed through live interleaved comparisons: the nomic embedding model beats MiniLM and mpnet-base, which tie; all three beat a session-based click-path recommender; and that in turn beats a term-similarity baseline. The ordering held across ~11,765 clicks with adjusted significance. The category analysis qualifies it: term similarity improves by 15-20% for Research Data and Variables, where it even surpasses the click-path recommender, while semantic models dominate for Publications. The authors read this as evidence that academic users want items semantically close to the item they are viewing, and that dat

Load-bearing premise

The whole ranking rests on treating a click on a recommended item as a valid, independent, and comparable signal of user preference between two interleaved systems.

Editorial extensions

If this is right

  • If the reported ordering is right, embedding-based semantic similarity should be the default 'related items' strategy in academic search portals, replacing or supplementing lexical matching.
  • Term-based methods remain a cheap, competitive option for research-data and variable search, where users appear to be doing known-item lookup rather than exploration.
  • Session-history prediction is a weaker fit for scholarly search than for e-commerce browsing, because academic sessions stay topically focused.
  • Models of similar quality can differ greatly in cost: the MiniLM model matches mpnet-base performance with half the embedding dimensionality, so infrastructure savings are possible without losing clicks.
  • Because the observed preferences satisfy Strong Stochastic Transitivity, future portals can rank n recommenders with O(n log n) interleaved comparisons instead of all pairwise tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The category-dependent result suggests a recommender portfolio that switches strategy per category—term-based for data/variables, semantic for publications—could beat any single global strategy; the paper identifies the pattern but does not test such a hybrid.
  • Editorial inference: Because the evaluation rests only on click counts, the close tie between MiniLM and mpnet-base could dissolve under a satisfaction-oriented signal such as dwell time or later reuse of the recommended item; the authors concede this limitation.
  • Editorial inference: If the mechanism is that users prefer semantically coherent suggestions, the result should transfer to other multilingual academic portals with rich structured metadata, but it may not transfer to portals with thin or noisy metadata where embeddings are lower quality.
  • Editorial inference: A testable extension would be to expose recommendations from a term-based system alongside a semantic one for research-data queries only, and measure whether the category-specific advantage reproduces; the current evidence is post-hoc from pooled clicks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a 23-week online interleaving evaluation of five recommender systems deployed in GESIS Search, a social-science academic search engine. Using the STELLA framework, the authors compare term-based 'more-like-this' recommendations, three embedding-based semantic similarity recommenders (nomic, MiniLM, mpnet-base), and a session-based click-path recommender (CPR). The overall empirical finding is a clear preference ordering: nomic > MiniLM ≈ mpnet-base > CPR > term-similarity, based on ten pairwise comparisons with roughly 1,000 or more clicks each. The paper also reports a category-level breakdown, arguing that recommendation effectiveness varies by content type and that term-based methods remain competitive for Research Data and Variables.

Significance. If the central ranking holds, this is a valuable applied contribution: it is one of the few living-lab evaluations of lexical, semantic, and session-based recommenders for a mixed scholarly search portal, and it demonstrates the feasibility of continuous online evaluation in this setting. The study's strengths include a complete pairwise interleaving design, large cumulative click counts, use of Holm–Bonferroni correction for the overall comparisons, and an unusually candid discussion of limitations. These strengths make the headline ordering credible. However, the category-level conclusions and the claimed satisfaction of Strong Stochastic Transitivity are not supported by the evidence as presented, and both appear in the abstract, discussion, and conclusion as load-bearing takeaways.

major comments (3)
  1. [Section 5.3, Table 3; Section 6; Conclusion] The category-level claim that term-based methods remain competitive for Research Data and Variables is not statistically supported. Table 3 reports only raw win counts and percentages, with no standard errors, confidence intervals, p-values, or multiple-testing correction for the roughly 50 category×pair tests. Recomputing simple binomial tests from the table: in Variables, CPR vs term-sim is 42 vs 44 (term-sim 51.2%, n=86, two-sided p≈0.83), so the statement that term-sim 'outperformed' CPR in Variables is unsupported; in Research Data, term-sim vs MiniLM is 156 vs 169 (48.0%, n=325, p≈0.47), so term-sim is not shown superior to MiniLM there either. The paper explicitly dismisses Instruments & Tools due to small counts yet applies the same raw-rate interpretation to Variables and other categories. Before the conclusion 'for Research Data and Variables, simpler term-based methods remain
  2. [Section 6, SST claim vs Table 2] The paper states: 'Our evaluation results satisfy Strong Stochastic Transitivity (SST).' This is contradicted by the paper's own Table 2. Let P(A,B) be the observed probability that system A wins over B. From Table 2, P(nomic, MiniLM)=769/1359≈0.566 and P(MiniLM, mpnet-base)=613/1219≈0.503, both ≥0.5. SST then requires P(nomic, mpnet-base) ≥ max(0.566,0.503)=0.566. The table gives P(nomic, mpnet-base)=767/1382≈0.555, which is lower. Thus the SST claim fails on the presented data. Consequently, the statements that the setup 'produces consistent rankings' and that a ranking can be obtained with O(n log n) rather than all pairwise comparisons are unsupported. This passage should be removed or substantially revised, and any argument about evaluation cost should be grounded in the actual pairwise design.
  3. [Section 5.2 and Section 6] The inferential basis for the headline ranking is clicks on interleaved recommendation lists. The authors acknowledge that clicks 'may also reflect curiosity, accidental interactions, or brief inspections' and that only click-based metrics are used. Because the same users can contribute multiple queries over the 23-week period, the binomial tests may overstate precision if clicks are clustered by user or by item; position bias and item popularity are also not modeled. I am not asking the authors to abandon click-based evaluation—the design is reasonable and the limitations are acknowledged—but the paper should address these dependencies explicitly and, ideally, report user-level or item-level clustered standard errors or confidence intervals for the win probabilities. This would strengthen the central claim without changing the experimental design.
minor comments (5)
  1. [Abstract/Section 6] The text says 'current daily traffic of GESIS Search with 30k document views per week'; the units are inconsistent (daily vs per week). Please fix.
  2. [Table 2] The table reports raw p-values, but the text says Holm–Bonferroni correction was applied. Please report adjusted p-values or clearly state which p-values remain significant after correction.
  3. [Section 5.2] The term 'wins' should be defined precisely: is a win one impression where the clicked item came from a given system? What happens to impressions with no click? Clarify the unit of analysis.
  4. [Section 3.2] There is a typo in the category list: 'V ariables' should be 'Variables'.
  5. [Section 6, SST paragraph] The SST citation [42] is to a general psychology decision theory text; a more standard reference from IR or recommender-system evaluation would help readers verify the property.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the evaluation is an empirical live-click comparison with external feedback not used to fit the compared systems.

full rationale

The paper is an empirical online evaluation, not a derivation, and no central claim reduces to its own inputs by construction. The five recommenders are fixed algorithms — Elasticsearch More-Like-This lexical similarity, three pretrained embedding models, and a GRU session recommender — with stated hyperparameters (cosine threshold 0.8, num_candidates 100, GRU hidden size 1000). None of these parameters was fitted to the interleaving click outcomes that define the reported ranking. The click-path recommender is trained on eight years of historical search logs and then evaluated on new live interleaved clicks, which is a standard train/test separation; the click-based evaluation signal is not used to train or tune the model. The only self-citation involving an author ([30]) describes the GESIS Search platform itself and is descriptive, not load-bearing. The statement that the results satisfy Strong Stochastic Transitivity is a post-hoc observation of the pairwise win data, not an assumption used to force the ranking. The category-specific claim about Research Data and Variables is statistically fragile because Table 3 reports raw win counts without confidence intervals or multiple-testing correction, but that is a correctness/statistical-inference concern, not circularity: the counts are external click measurements rather than definitions or fitted outputs. Therefore no circular step can be identified, and the honest finding is no circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central claims rest on a handful of hand-chosen thresholds/weights and on assumptions about click behavior and evaluation infrastructure. The most consequential unmodeled parameter is the cosine threshold 0.8 and the MLT field weights; both affect system outputs directly.

free parameters (5)
  • cosine_similarity_threshold = 0.8
    In §4.2, candidate documents with cosine similarity above 0.8 are retained as recommendations. The threshold is chosen without justification and affects the number/quality of semantic recommendations.
  • mlt_field_weights = title^10, topic^7, abstract^3, etc.
    Listing 1 manually assigns weights to fields in the more-like-this query; these weights shape the lexical recommender's output.
  • knn_num_candidates = 100
    Listing 2 sets num_candidates=100 to balance efficiency and effectiveness; a hand-picked operational parameter.
  • gru_hidden_units = 1000
    §4.3: single GRU layer with 1000 hidden units; model size chosen by hand with no reported tuning.
  • fc_units = 512
    §4.3: fully connected layer with 512 units predicts the next-click embedding; architecture choice not tuned.
assumptions (6)
  • domain assumption Clicks on recommended items express user preference between the two systems in an interleaved list.
    §5.2–5.3 and §6: all win/loss counts are derived from clicks; authors acknowledge clicks may reflect curiosity or accidents.
  • domain assumption User queries (interleaving impressions) are independent for the binomial sign test.
    §5.2: statistical significance uses a two-sided binomial sign test; sequential 23-week experiment may allow repeated user contributions.
  • domain assumption The STELLA framework correctly performs team-draft interleaving and click attribution.
    §3.1: the evaluation infrastructure is taken as given; no validation of attribution logic is reported.
  • domain assumption Historic user interaction logs (8 years, 16M interactions) are a valid training source for session-based recommendations.
    §4.3: CPR is trained on these logs; the authors later note the model was trained on search logs rather than recommendation logs (§6).
  • domain assumption Recommendations should be restricted to the same information category as the source item.
    §4.2 and listing 2 filter by item type; this restricts the evaluation to within-category recommendations.
  • domain assumption Relevance and information need are stable across the 23-week evaluation period.
    §6: comparisons ran sequentially over ~23 weeks; no stationarity test is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search." pith.science (2026). https://pith.science/paper/2D6CLSMG

@misc{pith2026260726380,
  author       = {Pith},
  title        = {Pith review of: Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2D6CLSMG}},
  note         = {Machine review of arXiv:2607.26380}
}
read the original abstract

Delivering relevant recommendations in academic search engines is a complex task due to the diversity of subject areas, information types, and user preferences. In this case study, we address these challenges by integrating and evaluating a range of recommendation systems within GESIS Search - a domain-specific search engine for the social sciences that provides researchers with access to research data, publications, variables, and measurement instruments. To support continuous, real-time evaluation of multiple recommendation strategies with actual platform users, we utilize the STELLA evaluation framework. We implement and compare a diverse set of algorithms, including traditional lexical similarity, semantic document similarity by using transformer-based embeddings, and session-based recommendations based on click paths from historical user sessions. Our results show that users prefer recommendations based on semantic similarity, which outperformed term-similarity and session-based methods. However, the performance of recommenders varies across categories within GESIS Search, suggesting that information-seeking behavior differs by information type. Overall, our study provides insights into how continuous evaluation can be incorporated to develop recommendations that better align with the preferences in academic search portals.

Figures

Figures reproduced from arXiv: 2607.26380 by the authors.

Figure 1
Figure 1. Recommendations in the section Similar Research Data for a given re￾search dataset about social media usage. For all comparisons, we used team-draft interleaving (TDI) [63], which creates a single result list by selecting items from the two systems in a randomized draft order, minimizing bias. User interactions with the interleaved rankings were interpreted as implicit preferences for one of the two systems. Because… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 2 linked inside Pith

  1. [1]

    Journal of Informetrics11(2), 553–563 (2017)

    Alhoori, H., Furuta, R.: Recommendation of scholarly venues based on dy- namic user interests. Journal of Informetrics11(2), 553–563 (2017)

    2017

  2. [2]

    arXiv preprint arXiv:2001.06910 (2020),https://arxiv.org/pdf/2001.06910

    Balog, K., Flekova, L., Hagen, M., Jones, R., Potthast, M., Radlinski, F., Sanderson, M., Vakulenko, S., Zamani, H.: Common conversational community prototype: Scholarly conversational assistant. arXiv preprint arXiv:2001.06910 (2020),https://arxiv.org/pdf/2001.06910

    arXiv 2001

  3. [3]

    dlib’s living lab for scholarly recommendations

    Beel, J., Collins, A., Kopp, O., Dietz, L.W., Knoth, P.: Online evaluations for everyone: Mr. dlib’s living lab for scholarly recommendations. In: Euro- pean Conference on Information Retrieval. pp. 213–219. Springer (2019)

    2019

  4. [4]

    In: Proceedings of the international workshop on reproducibility and replication in recommender systems eval- uation

    Beel, J., Genzmehr, M., Langer, S., N¨ urnberger, A., Gipp, B.: A compar- ative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In: Proceedings of the international workshop on reproducibility and replication in recommender systems eval- uation. pp. 7–14 (2013)

    2013

  5. [5]

    International Journal on Digital Libraries17(4), 305–338 (2016)

    Beel, J., Gipp, B., Langer, S., Breitinger, C.: Paper recommender systems: a literature survey. International Journal on Digital Libraries17(4), 305–338 (2016)

    2016

  6. [6]

    In: International conference on theory and practice of digital libraries

    Beel, J., Langer, S.: A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: International conference on theory and practice of digital libraries. pp. 153–168. Springer (2015)

    2015

  7. [7]

    In: International Conference on Theory and Practice of Digital Libraries

    Beel, J., Langer, S., Genzmehr, M., N¨ urnberger, A.: Persistence in recom- mender systems: giving the same recommendations to the same users mul- tiple times. In: International Conference on Theory and Practice of Digital Libraries. pp. 386–390. Springer (2013)

    2013

  8. [8]

    In: International conference on theory and practice of digital libraries

    Beel, J., Langer, S., N¨ urnberger, A., Genzmehr, M.: The impact of de- mographics (age and gender) and other user-characteristics on evaluating recommender systems. In: International conference on theory and practice of digital libraries. pp. 396–400. Springer (2013)

    2013

Show all 84 references
  1. [9]

    In: International Conference on Theory and Practice of Digital Libraries

    Brack, A., Hoppe, A., Ewerth, R.: Citation recommendation for research papers via knowledge graphs. In: International Conference on Theory and Practice of Digital Libraries. pp. 165–174. Springer (2021)

    2021

  2. [10]

    OSIRRC@ SIGIR2409, 8–11 (2019)

    Breuer, T., Schaer, P., Tavakolpoursaleh, N., Schaible, J., Wolff, B., M¨ uller, B.: Stella: Towards a framework for the reproducibility of online search experiments. OSIRRC@ SIGIR2409, 8–11 (2019)

    2019

  3. [11]

    In: The world wide web conference

    Brickley, D., Burgess, M., Noy, N.: Google dataset search: Building a search engine for datasets in an open web ecosystem. In: The world wide web conference. pp. 1365–1375 (2019)

    2019

  4. [12]

    In: Proceedings of the 2nd international conference on web intelligence, mining and semantics

    Capelle, M., Frasincar, F., Moerland, M., Hogenboom, F.: Semantics-based news recommendation. In: Proceedings of the 2nd international conference on web intelligence, mining and semantics. pp. 1–9 (2012) Continuous Online Evaluation of Recommendations... 17

    2012

  5. [13]

    In: International Conference on Theory and Practice of Digital Libraries

    Carevic, Z., Roy, D., Mayr, P.: Characteristics of dataset retrieval sessions: experiences from a real-life digital library. In: International Conference on Theory and Practice of Digital Libraries. pp. 185–193. Springer (2020)

    2020

  6. [14]

    In: International Conference on Theory and Practice of Digital Libraries

    Charalampous, A., Knoth, P.: Classifying document types to enhance search and recommendations in digital libraries. In: International Conference on Theory and Practice of Digital Libraries. pp. 181–192. Springer (2017)

    2017

  7. [15]

    In: Aslib pro- ceedings

    Cleverdon, C.: The cranfield tests on index language devices. In: Aslib pro- ceedings. vol. 19, pp. 173–194. MCB UP Ltd (1967)

    1967

  8. [16]

    Knowledge and Information Systems61(2), 1147–1178 (2019)

    Dehghani Champiri, Z., Asemi, A., Siti Salwah Binti, S.: Meta-analysis of evaluation methods and metrics used in context-aware scholarly rec- ommender systems. Knowledge and Information Systems61(2), 1147–1178 (2019)

    2019

  9. [17]

    Publishing Research Quarterly 11(3), 104–108 (1995)

    Ekwurzel, D.: The history and scope of the american economic association’s econlit and the economic literature index. Publishing Research Quarterly 11(3), 104–108 (1995)

    1995

  10. [18]

    In: International Conference on Theory and Practice of Digital Libraries

    Entrup, E., Eppelin, A., Ewerth, R., Hartwig, J., Tullney, M., Wohlgemuth, M., Hoppe, A.: B! son: A tool for open access journal recommendation. In: International Conference on Theory and Practice of Digital Libraries. pp. 357–364. Springer (2022)

    2022

  11. [19]

    In: International Conference on Theory and Practice of Digital Libraries

    Entrup, E., Ewerth, R., Hoppe, A.: A comparison of automated journal recommender systems. In: International Conference on Theory and Practice of Digital Libraries. pp. 230–238. Springer (2023)

    2023

  12. [20]

    https://doi.org/10.25495/7GXK-RD71,https://www.zenodo.org/

    European Organization For Nuclear Research, OpenAIRE: Zenodo (2013). https://doi.org/10.25495/7GXK-RD71,https://www.zenodo.org/

    2013 doi

  13. [21]

    ACM Transactions on Information Systems (TOIS)23(2), 147–168 (2005)

    Fox, S., Karnawat, K., Mydland, M., Dumais, S., White, T.: Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS)23(2), 147–168 (2005)

    2005

  14. [22]

    In: Business Intelligence Applications and the Web: Models, Systems and Technologies, pp

    Frasincar, F., IJntema, W., Goossen, F., Hogenboom, F.: A semantic ap- proach for news recommendation. In: Business Intelligence Applications and the Web: Models, Systems and Technologies, pp. 102–121. IGI Global Sci- entific Publishing (2012)

    2012

  15. [23]

    Journal of Korean Medical Science31(5), 660–673 (2016)

    Gasparyan, A.Y., Yessirkepov, M., Voronov, A.A., Trukhachev, V.I., Kostyukova, E.I., Gerasimov, A.N., Kitas, G.D.: Specialist bibliographic databases. Journal of Korean Medical Science31(5), 660–673 (2016)

    2016

  16. [24]

    Nature Reviews Physics3(9), 602–603 (2021)

    Ginsparg, P.: Lessons from arxiv’s 30 years of information sharing. Nature Reviews Physics3(9), 602–603 (2021)

    2021

  17. [25]

    Library Philosophy and Practice pp

    Gupta, V., Pandey, S.R.: Recommender systems for digital libraries: a re- view of concepts and concerns. Library Philosophy and Practice pp. 1–9 (2019)

    2019

  18. [26]

    Hahnel, M.: Exclusive: figshare a new open data project that wants to change the future of scholarly publishing. Impact of Social Sciences Blog (2012),https://researchonline.lse.ac.uk/id/eprint/51893/ 1/blogs.lse.ac.uk-Exclusive_figshare_a_new_open_data_project_ that_wants_to_...

    2012

  19. [27]

    International Journal of Hybrid Information Technology8(3), 255– 272 (2015)

    Hanyurwimfura, D., Bo, L., Havyarimana, V., Njagi, D., Kagorora, F.: An effective academic research papers recommendation for non-profiled 18 Mehmet Deniz T¨ urkmen and Daniel Hienert users. International Journal of Hybrid Information Technology8(3), 255– 272 (2015)

    2015

  20. [28]

    In: Proceedings of the 16th annual international ACM SIGIR conference on Research and devel- opment in information retrieval

    Harman, D.: Overview of the first trec conference. In: Proceedings of the 16th annual international ACM SIGIR conference on Research and devel- opment in information retrieval. pp. 36–47 (1993)

    1993

  21. [29]

    Hassan, H.A.M., Sansonetti, G., Gasparetti, F., Micarelli, A., Beel, J.: Bert, elmo, use and infersent sentence encoders: The panacea for research-paper recommendation? In: RecSys (Late-Breaking Results). pp. 6–10 (2019)

    2019

  22. [30]

    In: 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

    Hienert, D., Kern, D., Boland, K., Zapilko, B., Mutschke, P.: A digital library for research data and related information in the social sciences. In: 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL). pp. 148–

    2019

  23. [31]

    In: Proceedings of the 1st workshop on scholarly web mining

    Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., Jack, K.: Building recommender systems for scholarly information. In: Proceedings of the 1st workshop on scholarly web mining. pp. 25–32 (2017)

    2017

  24. [32]

    In: International Conference on Theory and Practice of Digital Libraries

    Hurtado Mart ´ ın, G., Schockaert, S., Cornelis, C., Naessens, H.: Metadata impact on research paper similarity. In: International Conference on Theory and Practice of Digital Libraries. pp. 457–460. Springer (2010)

    2010

  25. [33]

    IEEE Open Journal of the Com- puter Society (2025)

    Ijebu, F.F., Liu, Y., Sun, C., Jere, N., Mienye, I.D., Usip, P.U.: Ensemble answer selection leveraging cross-lingual dealignment for improved question answering with mixture-of-experts setup. IEEE Open Journal of the Com- puter Society (2025)

    2025

  26. [34]

    Online information review 29(2), 208–214 (2005)

    Jacs´ o, P.: Google scholar: the pros and the cons. Online information review 29(2), 208–214 (2005)

    2005

  27. [35]

    ACM Transactions on Information Systems (TOIS)26(1), 1–es (2007)

    Jensen, E.C., Beitzel, S.M., Chowdhury, A., Frieder, O.: Repeatable eval- uation of search services in dynamic environments. ACM Transactions on Information Systems (TOIS)26(1), 1–es (2007)

    2007

  28. [36]

    In: Pro- ceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining

    Joachims, T.: Optimizing search engines using clickthrough data. In: Pro- ceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining. pp. 133–142 (2002)

    2002

  29. [37]

    In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

    Kaur, H., Downey, D., Singh, A., Cheng, E.Y.Y., Weld, D., Bragg, J.: Feedlens: polymorphic lenses for personalizing exploratory search over knowledge graphs. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. pp. 1–15 (2022)

    2022

  30. [38]

    In: Proceedings of the 2024 ACM SIGIR International Confer- ence on Theory of Information Retrieval

    Keller, J., Breuer, T., Schaer, P.: Evaluation of temporal change in ir test collections. In: Proceedings of the 2024 ACM SIGIR International Confer- ence on Theory of Information Retrieval. pp. 3–13 (2024)

    2024

  31. [39]

    In: International conference on electrical engineering and computer (2015)

    Keshavarz, S., Honarvar, A.R.: A parallel paper recommender system in big data scholarly. In: International conference on electrical engineering and computer (2015)

    2015

  32. [40]

    Data mining and knowl- edge discovery18(1), 140–181 (2009)

    Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled ex- periments on the web: survey and practical guide. Data mining and knowl- edge discovery18(1), 140–181 (2009)

    2009

  33. [41]

    IEEE Transactions on Emerging Topics in Computing 9(1), 226–237 (2018) Continuous Online Evaluation of Recommendations

    Kong, X., Mao, M., Wang, W., Liu, J., Xu, B.: Voprec: Vector represen- tation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing 9(1), 226–237 (2018) Continuous Online Evaluation of Recommendat...

    2018

  34. [42]

    Kozielecki, J.: Psychological decision theory, vol. 24. Springer Science & Business Media (1982)

    1982

  35. [43]

    International Journal on Digital Libraries22(2), 175–195 (2021)

    Kr¨ amer, T., Papenmeier, A., Carevic, Z., Kern, D., Mathiak, B.: Data- seeking behaviour in the social sciences. International Journal on Digital Libraries22(2), 175–195 (2021)

    2021

  36. [44]

    In: Proceedings of the 7th ACM Conference on Recommender Systems

    Krestel, R., Smyth, P.: Recommending patents based on latent topics. In: Proceedings of the 7th ACM Conference on Recommender Systems. pp. 395–398 (2013)

    2013

  37. [45]

    In: Pro- ceedings of the 2013 IEEE/ACM international conference on Advances in social networks analysis and mining

    K¨ u¸ c¨ uktun¸ c, O., Saule, E., Kaya, K., C ¸ ataly¨ urek,¨U.V.: Towards a personal- ized, scalable, and exploratory academic recommendation service. In: Pro- ceedings of the 2013 IEEE/ACM international conference on Advances in social networks analysis and mining. pp. 636–6...

    2013

  38. [46]

    In: International symposium on string processing and information retrieval

    Ley, M.: The dblp computer science bibliography: Evolution, research is- sues, perspectives. In: International symposium on string processing and information retrieval. pp. 1–10. Springer (2002)

    2002

  39. [47]

    In: Findings of the Association for Computational Linguistics: ACL 2025

    Li, C., Deng, Y., Zhang, J., Zong, C.: Group then scale: Dynamic mixture- of-experts multilingual language model. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 1730–1754 (2025)

    2025

  40. [48]

    ACM Transac- tions on Information Systems (TOIS)37(3), 1–23 (2019)

    Li, X., Chen, Y., Pettit, B., Rijke, M.D.: Personalised reranking of paper recommendations using paper content and user behavior. ACM Transac- tions on Information Systems (TOIS)37(3), 1–23 (2019)

    2019

  41. [49]

    Decision Support Systems146, 113546 (2021)

    Li, Y., Wang, R., Nan, G., Li, D., Li, M.: A personalized paper recom- mendation method considering diverse user preferences. Decision Support Systems146, 113546 (2021)

    2021

  42. [50]

    In: Proceedings of the 13th International Workshop on Au- tomating Test Case Design, Selection and Evaluation

    Li, Z., Rainer, A.: Academic search engines: constraints, bugs, and recom- mendations. In: Proceedings of the 13th International Workshop on Au- tomating Test Case Design, Selection and Evaluation. pp. 25–32 (2022)

    2022

  43. [51]

    ACM Computing Surveys57(2), 1–37 (2024)

    Li, Z., Yang, C., Chen, Y., Wang, X., Chen, H., Xu, G., Yao, L., Sheng, M.: Graph and sequential neural networks in session-based recommendation: A survey. ACM Computing Surveys57(2), 1–37 (2024)

    2024

  44. [52]

    In: Pro- ceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining

    Liu, Q., Zeng, Y., Mokhosi, R., Zhang, H.: Stamp: short-term atten- tion/memory priority model for session-based recommendation. In: Pro- ceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 1831–1839 (2018)

    2018

  45. [53]

    Library Hi Tech24(4), 614–619 (2006)

    Lossau, N., Rahmsdorf, S., Pieper, D., Summann, F.: Bielefeld academic search engine (base) an end-user oriented institutional repository search service. Library Hi Tech24(4), 614–619 (2006)

    2006

  46. [54]

    ludewig, d

    Ludewig, M., Jannach, D.: Evaluation of session-based recommendation al- gorithms: M. ludewig, d. jannach. User Modeling and User-Adapted Inter- action28(4), 331–390 (2018)

    2018

  47. [55]

    In: 2025 8th International Symposium on Big Data and Applied Statistics (ISBDAS)

    Ma, Z., Liu, B., Li, Y., Zhang, X.: Crs-mentor: A conversational recom- mender system for large language model-enhanced academic resource men- torship. In: 2025 8th International Symposium on Big Data and Applied Statistics (ISBDAS). pp. 42–49. IEEE (2025)

    2025

  48. [56]

    In: International Conference on Theory and Practice of Digital Libraries

    Medrek, J., Otto, C., Ewerth, R.: Recommending scientific videos based on metadata enrichment using linked open data. In: International Conference on Theory and Practice of Digital Libraries. pp. 286–292. Springer (2018) 20 Mehmet Deniz T¨ urkmen and Daniel Hienert

    2018

  49. [57]

    Advances in Neural Information Processing Systems34, 6466–6477 (2021)

    Nandy, P., Venugopalan, D., Lo, C., Chatterjee, S.: A/b testing for recom- mender systems in a two-sided marketplace. Advances in Neural Information Processing Systems34, 6466–6477 (2021)

    2021

  50. [58]

    Nussbaum, Z., Duderstadt, B.: Training sparse mixture of experts text em- bedding models (2025),https://arxiv.org/abs/2502.07972

    2025 arXiv

  51. [59]

    In: Companion Proceedings of the ACM Web Conference 2024

    Ong, D., Truong, Q.T., Lauw, H.W.: Cornac-ab: An open-source recom- mendation framework with native a/b testing integration. In: Companion Proceedings of the ACM Web Conference 2024. pp. 1027–1030 (2024)

    2024

  52. [60]

    In: Proceedings of the 20th ACM international conference on Information and knowledge management

    Pera, M.S., Ng, Y.K.: A personalized recommendation system on scholarly publications. In: Proceedings of the 20th ACM international conference on Information and knowledge management. pp. 2133–2136 (2011)

    2011

  53. [61]

    Peters, C.: Cross-Language Information Retrieval and Evaluation: Work- shop of Cross-Language Evaluation Forum, CLEF 2000, Lisbon, Portugal, September 21-22, 2000, Revised Papers, vol. 2069. Springer (2003)

    2000

  54. [62]

    In: Proceedings of the sixth ACM international conference on Web search and data mining

    Radlinski, F., Craswell, N.: Optimized interleaving for online retrieval eval- uation. In: Proceedings of the sixth ACM international conference on Web search and data mining. pp. 245–254 (2013)

    2013

  55. [63]

    Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality? In: Proceedings of the 17th ACM conference on Informa- tion and knowledge management. pp. 43–52 (2008)

    2008

  56. [64]

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 conference on empiri- cal methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). pp....

    2019

  57. [65]

    In: International Conference on Theory and Practice of Digital Libraries

    Risch, J., Krestel, R.: What should i cite? cross-collection reference recom- mendation of patents and papers. In: International Conference on Theory and Practice of Digital Libraries. pp. 40–46. Springer (2017)

    2017

  58. [66]

    Foundations and Trends®in Information Retrieval4(4), 247–375 (2010)

    Sanderson, M.: Test collection based evaluation of information retrieval sys- tems. Foundations and Trends®in Information Retrieval4(4), 247–375 (2010)

    2010

  59. [67]

    SBERT Pretrained Models.https://www.sbert.net/docs/sentence_ transformer/pretrained_models.html, accessed: 2026-04-30

    2026

  60. [68]

    In: European Conference on Information Retrieval

    Schaer, P., Schaible, J., Castro, L.J.: Living lab evaluation for life and social sciences search platforms-lilas at clef 2021. In: European Conference on Information Retrieval. pp. 657–664. Springer (2021)

    2021

  61. [69]

    In: European Conference on Information Retrieval

    Schaer, P., Schaible, J., M¨ uller, B.: Living labs for academic search at clef 2020. In: European Conference on Information Retrieval. pp. 580–586. Springer (2020)

    2020

  62. [70]

    Schonfeld, R.C.: Jstor: a history (2012)

    2012

  63. [71]

    In: Proceedings of the 28th annual inter- national ACM SIGIR conference on Research and development in informa- tion retrieval

    Sharma, H., Jansen, B.J.: Automated evaluation of search engine perfor- mance via implicit user feedback. In: Proceedings of the 28th annual inter- national ACM SIGIR conference on Research and development in informa- tion retrieval. pp. 649–650 (2005)

    2005

  64. [72]

    International Journal on digital libraries5(4), 299–308 (2005) Continuous Online Evaluation of Recommendations

    Smeaton, A.F., Callan, J.: Personalisation and recommender systems in dig- ital libraries. International Journal on digital libraries5(4), 299–308 (2005) Continuous Online Evaluation of Recommendations... 21

    2005

  65. [73]

    IEEE Access 10, 16–23 (2021)

    Sterling, J.A., Montemore, M.M.: Combining citation network information and text similarity for research article recommender systems. IEEE Access 10, 16–23 (2021)

    2021

  66. [74]

    Information, Communication & Society15(2), 151–170 (2012)

    Valtysson, B.: Europeana: The digital construction of europe’s collective memory. Information, Communication & Society15(2), 151–170 (2012)

    2012

  67. [75]

    Voorhees, E.M., Harman, D.K., et al.: TREC: Experiment and evaluation in information retrieval, vol. 63. MIT press Cambridge (2005)

    2005

  68. [76]

    In: Asia-Pacific web conference

    Wang, Q., Li, W., Zhang, X., Lu, S.: Academic paper recommendation based on community detection in citation-collaboration networks. In: Asia-Pacific web conference. pp. 124–136. Springer (2016)

    2016

  69. [77]

    Computers & Education: Artificial Intelligence10 (2025)

    Wang, X., Duong-Trung, N., Bhoyar, R.R., Jose, A.M.: Llm-based litera- ture recommender system in higher education: A case study of supervising students’ term papers. Computers & Education: Artificial Intelligence10 (2025)

    2025

  70. [78]

    Medical reference services quarterly39(4), 382–387 (2020)

    White, J.: Pubmed 2.0. Medical reference services quarterly39(4), 382–387 (2020)

    2020

  71. [79]

    Communications of the ACM 42(4), 30–31 (1999)

    White, J.: Acm digital library enhancements. Communications of the ACM 42(4), 30–31 (1999)

    1999

  72. [80]

    IEEE Transactions on Knowledge and Data Engineering36(8), 4275–4289 (2024)

    Wu, L., Li, Z., Zhao, H., Huang, Z., Han, Y., Jiang, J., Chen, E.: Sup- porting your idea reasonably: A knowledge-aware topic reasoning strategy for citation recommendation. IEEE Transactions on Knowledge and Data Engineering36(8), 4275–4289 (2024)

    2024

  73. [81]

    In: Proceedings of the AAAI conference on artificial intelligence

    Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., Tan, T.: Session-based rec- ommendation with graph neural networks. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 346–353 (2019)

    2019

  74. [82]

    In: 2012 11th international conference on machine learning and applications

    Yang, Z., Davison, B.D.: Venue recommendation: Submitting your paper with style. In: 2012 11th international conference on machine learning and applications. vol. 1, pp. 681–686. IEEE (2012)

    2012

  75. [83]

    ACM computing surveys55(8), 1–38 (2022)

    Zangerle, E., Bauer, C.: Evaluating recommender systems: survey and framework. ACM computing surveys55(8), 1–38 (2022)

    2022

  76. [84]

    Econometrics1(3), 249– 280 (2013)

    Zimmermann, C.: Academic rankings with repec. Econometrics1(3), 249– 280 (2013)

    2013

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.