{"id":"85df309c-1851-4ade-ac0a-8b25aaf2df41","arxiv_id":"2607.26380","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In a live test on the GESIS social-science search engine, semantic-embedding recommenders beat term-matching and click-session models, with dataset and variable search favoring simpler matching.","lead":"Researchers tested five ways of recommending related items in a social science search engine by showing real users mixed lists from pairs of systems and tracking which links they clicked. Embedding-based semantic similarity got the most clicks overall, while simple word matching remained competitive for research datasets and variables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Category-level claim that term-based methods remain competitive for Research Data and Variables rests on raw win rates with no confidence intervals or multiple-testing correction; Table 3 does not statistically support the Variables conclusion.","rationale":"The headline ranking (nomic > MiniLM≈mpnet-base > CPR > term-similarity) is supported by large pairwise samples and highly significant p-values; the acknowledged click-based preference limitation affects all interleaving studies and does not specifically undermine this comparison. I did not select the SST claim as the load-bearing concern: it is indeed contradicted by Table 2 (e.g., P(MiniLM,mpnet-base)=0.503 and P(mpnet-base,term-sim)=0.702 would require P(MiniLM,term-sim)≥0.702, but Table 2 gives 0.692), but that false claim only supports the O(n log n) efficiency argument, not the central ranking. The category-level claim, by contrast, is part of the abstract and conclusion, is used to give practical advice for dataset/variable search, and rests on Table 3 raw win rates without error bars or correction for multiple comparisons. The reader's weakest-assumption choice (click validity/independence) is reasonable and shared, but the category-level inference is the place where a concrete re-analysis of reported numbers could change the stated conclusions. Since my read supports the same CONDITIONAL verdict, no adjustment is needed.","tokens_in":14399,"tokens_out":13361,"duration_ms":147991,"concrete_test":"Re-analyze Table 3 counts directly: fit a logistic regression of winner on system pair, category, and system×category interaction (or run per-category two-sided binomial tests with Holm-Bonferroni correction across all 50 category-pair tests) and compute 95% confidence intervals for each category-level win rate. If the interaction is not significant, or if term-sim's advantage over CPR in Variables and any term-sim edge in Research Data disappears after correction, the category-specific conclusion should be restricted to Research Data or removed. This check uses only already-reported counts and settles whether the category dependence claim is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The secondary central claim—that the best recommender depends on information category and that term similarity remains competitive for Research Data and Variables (Section 5.3, Section 6)—is not established by the reported evidence. Table 3 reports only raw win counts/percentages per category, with no standard errors, confidence intervals, or p-values. Recomputing simple binomial tests from the counts: in Variables, CPR vs term-sim is 42 vs 44 (term-sim 51.2%, n=86; two-sided p≈0.83), so the statement that term-sim 'outperformed' CPR in Variables is unsupported. In Research Data, term-sim vs MiniLM is 156 vs 169 (48.0%, n=325), so term-sim is not shown superior to MiniLM there either. With 5 categories × 10 pairwise comparisons = 50 tests, raw rate differences can easily be chance fluctuations; no multiple-comparison correction and no system×category interaction test is reported. The paper explicitly concedes the Instruments & Tools counts are too small for conclusions, yet applies the same raw-rate interpretation to Variables and to other category contrasts. Because the practical recommendation 'for Research Data and Variables, simpler term-based methods remain competitive' depends on these numbers, the category-level analysis needs formal inference before that claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a 23-week online interleaving evaluation of five recommender systems deployed in GESIS Search, a social-science academic search engine. Using the STELLA framework, the authors compare term-based 'more-like-this' recommendations, three embedding-based semantic similarity recommenders (nomic, MiniLM, mpnet-base), and a session-based click-path recommender (CPR). The overall empirical finding is a clear preference ordering: nomic > MiniLM ≈ mpnet-base > CPR > term-similarity, based on ten pairwise comparisons with roughly 1,000 or more clicks each. The paper also reports a category-level breakdown, arguing that recommendation effectiveness varies by content type and that term-based methods remain competitive for Research Data and Variables.","tokens_in":14697,"tokens_out":6048,"duration_ms":70124,"significance":"If the central ranking holds, this is a valuable applied contribution: it is one of the few living-lab evaluations of lexical, semantic, and session-based recommenders for a mixed scholarly search portal, and it demonstrates the feasibility of continuous online evaluation in this setting. The study's strengths include a complete pairwise interleaving design, large cumulative click counts, use of Holm–Bonferroni correction for the overall comparisons, and an unusually candid discussion of limitations. These strengths make the headline ordering credible. However, the category-level conclusions and the claimed satisfaction of Strong Stochastic Transitivity are not supported by the evidence as presented, and both appear in the abstract, discussion, and conclusion as load-bearing takeaways.","major_comments":[{"comment":"The category-level claim that term-based methods remain competitive for Research Data and Variables is not statistically supported. Table 3 reports only raw win counts and percentages, with no standard errors, confidence intervals, p-values, or multiple-testing correction for the roughly 50 category×pair tests. Recomputing simple binomial tests from the table: in Variables, CPR vs term-sim is 42 vs 44 (term-sim 51.2%, n=86, two-sided p≈0.83), so the statement that term-sim 'outperformed' CPR in Variables is unsupported; in Research Data, term-sim vs MiniLM is 156 vs 169 (48.0%, n=325, p≈0.47), so term-sim is not shown superior to MiniLM there either. The paper explicitly dismisses Instruments & Tools due to small counts yet applies the same raw-rate interpretation to Variables and other categories. Before the conclusion 'for Research Data and Variables, simpler term-based methods remain","section":"Section 5.3, Table 3; Section 6; Conclusion"},{"comment":"The paper states: 'Our evaluation results satisfy Strong Stochastic Transitivity (SST).' This is contradicted by the paper's own Table 2. Let P(A,B) be the observed probability that system A wins over B. From Table 2, P(nomic, MiniLM)=769/1359≈0.566 and P(MiniLM, mpnet-base)=613/1219≈0.503, both ≥0.5. SST then requires P(nomic, mpnet-base) ≥ max(0.566,0.503)=0.566. The table gives P(nomic, mpnet-base)=767/1382≈0.555, which is lower. Thus the SST claim fails on the presented data. Consequently, the statements that the setup 'produces consistent rankings' and that a ranking can be obtained with O(n log n) rather than all pairwise comparisons are unsupported. This passage should be removed or substantially revised, and any argument about evaluation cost should be grounded in the actual pairwise design.","section":"Section 6, SST claim vs Table 2"},{"comment":"The inferential basis for the headline ranking is clicks on interleaved recommendation lists. The authors acknowledge that clicks 'may also reflect curiosity, accidental interactions, or brief inspections' and that only click-based metrics are used. Because the same users can contribute multiple queries over the 23-week period, the binomial tests may overstate precision if clicks are clustered by user or by item; position bias and item popularity are also not modeled. I am not asking the authors to abandon click-based evaluation—the design is reasonable and the limitations are acknowledged—but the paper should address these dependencies explicitly and, ideally, report user-level or item-level clustered standard errors or confidence intervals for the win probabilities. This would strengthen the central claim without changing the experimental design.","section":"Section 5.2 and Section 6"}],"minor_comments":[{"comment":"The text says 'current daily traffic of GESIS Search with 30k document views per week'; the units are inconsistent (daily vs per week). Please fix.","section":"Abstract/Section 6"},{"comment":"The table reports raw p-values, but the text says Holm–Bonferroni correction was applied. Please report adjusted p-values or clearly state which p-values remain significant after correction.","section":"Table 2"},{"comment":"The term 'wins' should be defined precisely: is a win one impression where the clicked item came from a given system? What happens to impressions with no click? Clarify the unit of analysis.","section":"Section 5.2"},{"comment":"There is a typo in the category list: 'V ariables' should be 'Variables'.","section":"Section 3.2"},{"comment":"The SST citation [42] is to a general psychology decision theory text; a more standard reference from IR or recommender-system evaluation would help readers verify the property.","section":"Section 6, SST paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical case study and the overall pairwise ranking appears credible. The main obstacles are the unsupported category-level inference in Table 3 and the false SST claim. Both are fixable within the manuscript's scope, but they affect central statements in the abstract and conclusion, so I recommend major revision rather than minor revision. If the authors reanalyze Table 3 with appropriate statistical tests and remove or correct the SST paragraph, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on the GESIS Search paper. It's a solid empirical case study, and the headline result—semantic embeddings beat lexical and session-based recommenders on live clicks—is credible. What's genuinely useful is the setting: a multi-category academic search engine (publications, datasets, variables, instruments) where online evaluation is rare; most previous work stops at publications. The experimental machinery is sound: STELLA interleaving, team-draft, binomial sign tests with Holm–Bonferroni correction, and 10 pairwise comparisons each with over 1,000 clicks. The authors also acknowledge the click-as-relevance limitation honestly. For practitioners, the practical takeaway—use embedding-based related items as a default, and keep lexical methods for known-item dataset/variable search—is plausible.\n\nBut there are two soft spots that matter. First, the category-level analysis (Table 3) is raw win counts with no error bars, no p-values, and no correction for the 50 tests one could run across categories and pairs. The conclusion that term-based methods 'remain competitive' for Research Data and Variables isn't established by those numbers. In Variables, CPR vs term-sim is 42–44, a coin flip; in Research Data, term-sim vs MiniLM is 156–169, not significant. That claim should be softened to 'warrants further study,' not stated as a finding. Second, the paper's assertion that the results satisfy Strong Stochastic Transitivity is contradicted by its own Table 2. For nomic, MiniLM, mpnet-base: P(nomic, miniLM)=0.566, P(miniLM, mpnet-base)=0.503, but P(nomic, mpnet-base)=0.555, which is less than the required max. So the O(n log n) efficiency argument doesn't hold. The overall ranking is unaffected, but the assertion needs to be fixed.\n\nWho is this for? People building recommenders for academic portals and anyone doing online evaluation of IR systems. It's not a landmark, but it's a well-executed case study in an underevaluated setting. It deserves a serious referee. I'd want the authors to correct the SST claim and add proper significance testing to the category-level analysis before acceptance. If they do, this is a citable result.","headline":"A useful living-lab case study: the overall embedding-over-lexical ranking is credible, but the category-level claims and the SST assertion don't hold up.","tokens_in":15181,"tokens_out":5983,"would_cite":true,"duration_ms":57780,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic similarity is what users prefer for related-item recommendations in academic search, beating term matching and session-history prediction in a live interleaving test.","keywords":["continuous evaluation","interleaving","recommender systems","academic search","semantic similarity","text embeddings","session-based recommendation","social science search"],"falsifier":"Re-attribute the collected clicks while controlling for display position and item popularity; if the pairwise ordering reverses or the nomic advantage disappears after such control, the central claim fails. A dwell-time or follow-up-engagement measure that ranks the systems opposite to click counts would also undercut it.","tokens_in":14263,"feed_emoji":"🔎","tokens_out":8766,"duration_ms":88623,"temperature":0.7,"pith_summary":"This paper tries to establish which recommendation strategy users actually prefer in a living academic search portal: embedding-based semantic similarity, lexical term matching, or session-history prediction. Using continuous online interleaving with real clicks over roughly 23 weeks, it finds a stable preference ordering, with semantic similarity best, click-path session prediction in the middle, and term matching last. The same data shows that the best strategy depends on information type: simpler term matching stays competitive for research data and variable search, where users often look for known items, while semantic similarity dominates for publications and general exploration. If the result holds, the practical consequence is that embedding-based 'related items' should be the default for scholarly search, with lightweight lexical matching as a sensible fallback for known-item categories. The paper also demonstrates that continuous online evaluation can produce a consistent ranking with far fewer comparisons than exhaustive pairwise testing.","feed_headline":"Semantic similarity wins live test of academic search recommendations","feed_subtitle":"Users clicked embedding-based recommendations most often; simpler term matching still won for data and variable search.","key_machinery":"The load-bearing mechanism is team-draft interleaving: for each impression, two recommender systems each produce a ranked list, and the evaluation service merges them into a single displayed list by drafting items from the two lists in randomized order, so that every click can be attributed to the system that supplied the clicked item. Pairwise comparisons are then tested with a two-sided binomial sign test using Holm-Bonferroni correction. The recommenders themselves are standard content-based components: a term-similarity baseline using more-like-this lexical matching, three embedding-based variants that retrieve near neighbours by cosine similarity in a dense vector index, and a session r","core_discovery":"The paper's central claim is a preference ordering among five recommender systems observed through live interleaved comparisons: the nomic embedding model beats MiniLM and mpnet-base, which tie; all three beat a session-based click-path recommender; and that in turn beats a term-similarity baseline. The ordering held across ~11,765 clicks with adjusted significance. The category analysis qualifies it: term similarity improves by 15-20% for Research Data and Variables, where it even surpasses the click-path recommender, while semantic models dominate for Publications. The authors read this as evidence that academic users want items semantically close to the item they are viewing, and that dat","pith_inferences":["Editorial inference: The category-dependent result suggests a recommender portfolio that switches strategy per category—term-based for data/variables, semantic for publications—could beat any single global strategy; the paper identifies the pattern but does not test such a hybrid.","Editorial inference: Because the evaluation rests only on click counts, the close tie between MiniLM and mpnet-base could dissolve under a satisfaction-oriented signal such as dwell time or later reuse of the recommended item; the authors concede this limitation.","Editorial inference: If the mechanism is that users prefer semantically coherent suggestions, the result should transfer to other multilingual academic portals with rich structured metadata, but it may not transfer to portals with thin or noisy metadata where embeddings are lower quality.","Editorial inference: A testable extension would be to expose recommendations from a term-based system alongside a semantic one for research-data queries only, and measure whether the category-specific advantage reproduces; the current evidence is post-hoc from pooled clicks."],"forward_implications":["If the reported ordering is right, embedding-based semantic similarity should be the default 'related items' strategy in academic search portals, replacing or supplementing lexical matching.","Term-based methods remain a cheap, competitive option for research-data and variable search, where users appear to be doing known-item lookup rather than exploration.","Session-history prediction is a weaker fit for scholarly search than for e-commerce browsing, because academic sessions stay topically focused.","Models of similar quality can differ greatly in cost: the MiniLM model matches mpnet-base performance with half the embedding dimensionality, so infrastructure savings are possible without losing clicks.","Because the observed preferences satisfy Strong Stochastic Transitivity, future portals can rank n recommenders with O(n log n) interleaved comparisons instead of all pairwise tests."],"fun_headline_variants":["Semantic embeddings top live test of academic search recommenders","In live A/B test, scholars pick semantic recommendations","Semantic similarity beats session history in real-world search test","Academic search users favor semantic similarity in live head-to-head","Live test: semantic recommenders win academic search clicks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole ranking rests on treating a click on a recommended item as a valid, independent, and comparable signal of user preference between two interleaved systems.","fun_headline_variants_meta":{"raw":{"variants":["Semantic embeddings top live test of academic search recommenders","In live A/B test, scholars pick semantic recommendations","Semantic similarity beats session history in real-world search test","Academic search users favor semantic similarity in live head-to-head","Live test: semantic recommenders win academic search clicks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2633,"prompt_tokens":703,"completion_tokens":1930,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1851}},"tokens_in":447,"tokens_out":1930,"duration_ms":14806,"temperature":1.0,"reasoning_tokens":1851,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:59:54.394817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-attribute the collected clicks while controlling for display position and item popularity; if the pairwise ordering reverses or the nomic advantage disappears after such control, the central claim fails. A dwell-time or follow-up-engagement measure that ranks the systems opposite to click counts would also undercut it.","supporting_citations":[],"review_version":1}