{"id":"ba75086f-ebfa-422d-8db6-6a21668bcaa5","arxiv_id":"2504.20679","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On a longitudinal survey dataset from 1946-2020, BGE-m3 retrieves concept-equivalent questions with an F1 score of 0.79, only four points above the BM25 keyword baseline.","lead":"This paper tests whether search and language models can automatically find questions with the same meaning across decades of social surveys. It finds that a neural search model works best, but simple keyword matching stays close behind.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BGE-m3 vs BM25 advantage may vanish under the paper's own stricter equivalence labels, so the central performance claim is not yet supported.","rationale":"The reader identified the coarse top-level topic-code metric as the weakest assumption; this pass agrees and sharpens the concern with a concrete computation from the paper's own Table 3. Under the stricter labels the authors themselves define, the BGE-m3 advantage over BM25 shrinks from four F1 points to about one percentage point on a 203-pair sample, which is within likely sampling error. This makes the central summary statement 'IR-specialised neural models achieve the highest overall performance' unsupported as presented, although the qualitative finding that all models struggle with sub-concept mismatches is well supported and valuable. The paper deserves credit for including expert annotations and transparent qualitative examples, which enable this re-analysis, but the headline quantitative comparison needs to be re-run on the strict labels before the conclusion is accepted. The verdict remains CONDITIONAL: the paper is a useful task formulation and negative-result qualitative analysis, but its central quantitative claim requires the proposed re-analysis. Since the reader already assigned a conditional verdict, I do not change the verdict, but the rationale for conditionality is now sharper and directly tied to the paper's own data.","tokens_in":7819,"tokens_out":3510,"duration_ms":36952,"concrete_test":"Re-score the 203 expert-annotated pairs in Table 3 with a binary gold label: correct if the annotator label is 1 or 1a, incorrect if 2 or 3, and compare BM25 against BGE-m3 (and all other models) using McNemar's test with exact binomial confidence intervals. If a larger expert-annotated sample is feasible, recompute Table 2's F1, precision, and recall on top-1 retrievals with this strict label to see whether BGE-m3's advantage over BM25 remains statistically significant. If the 0.79 vs 0.75 gap falls within the confidence interval or disappears, the headline claim should be revised to state that BM25 and BGE-m3 are statistically indistinguishable on true conceptual equivalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core quantitative claim—that IR-specialised BGE-m3 outperforms BM25 for survey harmonisation (F1 0.79 vs 0.75)—relies entirely on a proxy metric: a retrieved pair counts as correct if the query and retrieved question share a 16-way top-level topic code (Section 3, Evaluation). This is not equivalent to conceptual equivalence, which the task defines as matching concept and sub-concept. The paper's own expert annotation (Table 3) allows a direct check: treating labels 1 (exact) and 1a (equivalent) as correct and labels 2 (sub-concept mismatch) and 3 (total mismatch) as incorrect yields correct rates of 74.9% for BM25 and 75.9% for BGE-m3 on the 203 sampled pairs—roughly one percentage point apart, versus the four-point F1 gap in Table 2. The qualitative examples (Table 4, rows a/b/c) show that broad-topic matches with sub-concept mismatch are frequent and that high lexical overlap from response options can inflate scores. Consequently, the reported F1 advantage may be an artifact of the coarse label, and the conclusion that neural ranking gives only a modest gain is untested: the gain may be negligible once true equivalence is required. Since code and evaluation data are not released, readers cannot currently check this independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes treating the harmonisation of longitudinal survey questions as an information retrieval task. It constructs query–candidate pairs from code-list questions in a longitudinal survey collection spanning 1946–2020, concatenating question text and response options to form input sequences. It compares BM25, SBERT, linearly probed DeBERTa and Qwen embeddings, and the IR-specialised BGE-m3 and BGE-reranker-m3 models, all with frozen pretrained weights. Performance is measured by whether the single top-ranked retrieved question shares the same 16-way top-level topic code as the query. The paper reports that BGE-m3 achieves the highest overall performance (F1=0.79), that BM25 remains competitive (F1=0.75), and that reranking BM25 outputs with neural models yields only modest gains. A qualitative expert annotation of 203 randomly selected pairs identifies sub-concept mismatches as a common failure mode, and the authors conclude that a balance of syntactic and semantic information is critical for this task.","tokens_in":8083,"tokens_out":5963,"duration_ms":63676,"significance":"If the quantitative claims are valid, the paper would make a useful contribution by formulating survey harmonisation as an IR problem, providing an evaluation dataset and baseline comparisons, and showing that lexical matching remains important in this domain. The evaluation is clean in one important respect: all models are used with frozen pretrained weights, no parameters are fitted to the survey data, and the topic labels come from an external annotation ontology, so there is no circularity. The qualitative analysis by survey specialists is a genuine strength and directly informs the conclusion about sub-concept sensitivity. However, the central quantitative claims rest on a coarse proxy metric, and the paper's own finer-grained labels suggest that the headline BM25-versus-BGE-m3 gap may be much smaller than reported. The lack of uncertainty quantification and the absence of released code or data further limit the strength of the conclusions. The task and findings are of interest to the SIGIR community, but the evaluation needs to be substantially tightened before the central claims can be accepted.","major_comments":[{"comment":"The evaluation proxy is not equivalent to the task being studied. The task is defined in terms of concept and sub-concept equivalence, but a retrieved pair is counted as correct if the query and the retrieved question share one of 16 top-level topic codes. The paper's own expert annotation shows why this matters: recomputing the 203 sampled pairs with labels 1 and 1a treated as correct gives 74.9% for BM25 and 75.9% for BGE-m3, a gap of about one percentage point, whereas Table 2 reports a four-point F1 gap (0.79 vs 0.75). The examples in Table 4 (a) and (b) are precisely cases where top-level topics match but sub-concepts differ. The headline conclusion that IR-specialised neural models achieve the highest overall performance is therefore not established by the reported metric. The authors should report results under the stricter expert-labelled definition on the 203-pair sample, or otherwise validate that top-level topic agreement is a faithful proxy for harmonisation quality, and should qualify the conclusion in Section 5 accordingly.","section":"§3 (Evaluation), §4 (Tables 3 and 4), §5"},{"comment":"The comparison lacks any uncertainty quantification. All retrieval models are deterministic, but the evaluation itself is a finite sample, and the 203-pair expert subset introduces additional sampling variability. No confidence intervals, bootstrap estimates, or significance tests are reported for the Table 2 metrics. A 0.04 F1 gap between BGE-m3 and BM25 may well be within sampling noise, especially since the stricter labels in Table 3 reduce the gap to about one point. The authors should report paired significance tests (for example, McNemar's test on the 203-pair sample) and confidence intervals for the main F1 comparisons.","section":"§3 (Evaluation), §4 (Table 2)"},{"comment":"The size of the retrieval collection is stated inconsistently. The main text says there are 42,161 questions in total, while the footnote on the same page says '130,863 code list questions in total.' This is not a purely cosmetic discrepancy: the number of candidates in the retrieval pool affects all of the reported scores, especially the end-to-end ranking results, and it also affects the interpretation of the re-ranking setup. The authors should clarify which number is correct and report the exact number of query questions and candidate questions used in each experiment.","section":"§3 (Dataset) and footnote"},{"comment":"The evaluation only considers the top-1 retrieved question, even though the stated task is to identify all potentially equivalent questions. This discards all ranking information beyond the first result and gives no measure of how well the methods would support the intended harmonisation workflow. The authors should either use ranking-aware metrics (for example, recall at k, MRR, or nDCG) or justify why top-1 correctness is the appropriate operational measure for the task.","section":"§3 (Evaluation)"}],"minor_comments":[{"comment":"The phrase 'linear probing' is misleading because no linear layer is trained: the authors state that all models are training-free and have fixed weights. What is actually used is embedding extraction with pooling, so the terminology should be changed to avoid confusion with the standard linear-probing protocol.","section":"§3 (Implementation Details)"},{"comment":"The notation for the union of questionnaires appears malformed as 'U = —n i=1 Si'; it should be typeset as a union operator with proper subscripts and superscripts.","section":"§3 (Dataset)"},{"comment":"The expert annotation in Table 3 does not report the number of annotators or inter-annotator agreement. Since the qualitative conclusions about sub-concept mismatches are part of the paper's central argument, at least a brief statement on annotation reliability would be valuable.","section":"§4 (Qualitative Analysis)"},{"comment":"There is a typo in 'Traditional IR continuous to adopt'; this should be 'Traditional IR continues to adopt'.","section":"§2 (Related Work)"},{"comment":"The relationship between precision and recall is not fully specified: some rows have equal precision and recall while others do not, and the relevant set for each query is never formally defined. A concrete definition of the gold relevance set would remove ambiguity.","section":"§4 (Table 2)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile and timely task, and the qualitative expert analysis is a strong point. The main obstacle is that the quantitative claim of BGE-m3's superiority over BM25 is built on a coarse topic-code proxy, and the authors' own finer-grained labels suggest the gap nearly vanishes. I would not accept the paper in its current form, but the issues are fixable within the manuscript's scope if the authors reanalyse the expert-labelled sample, add significance testing, and clarify the dataset size. I would also encourage the authors to release the evaluation data, as this would substantially increase the value of the resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper sells a 0.04 F1 advantage for BGE-m3 over BM25, but its own Table 3 cuts that to about one point under stricter equivalence labels. Read it for the task formulation and the qualitative failure modes, not for the ranking.\n\nThe genuinely new thing is casting survey question harmonization as an IR task, with question + response options as a single input. The comparison of BM25, SBERT, DeBERTa, Qwen, BGE-m3 on 75 years of survey questions is straightforward and training-free, and the qualitative annotation of 203 pairs by survey specialists is real work. The observation that response options inflate lexical overlap and that models are insensitive to sub-concept mismatches is honest and useful. I believe the qualitative analysis is the most durable contribution.\n\nNow the soft spots. The evaluation metric is coarse: a retrieved pair is correct if the query and candidate share one of 16 top-level topic codes, for the top-1 candidate only. Precision equals recall by construction. No significance tests or error bars. The paper acknowledges the proxy's limitation but still headlines the F1=0.79 vs 0.75 in the abstract and conclusion. If you instead score the 203 expert-labeled pairs as correct for labels 1 and 1a, BM25 gets about 74.9% and BGE-m3 about 75.9% — a one-point gap. So the four-point gap in Table 2 is likely an artifact of the coarse label. The dataset and code are also not released, so the numbers cannot be independently checked. That is a real limitation for a short empirical paper.\n\nOne minor inconsistency: the abstract says re-ranking gives at most 0.07 F1 improvement, but the re-ranking rows in Table 2 show only 0.02 over BM25 at best; 0.07 does not cleanly map to the table.\n\nNone of this makes the paper a waste. The task definition and the qualitative typology are worth having. But the central quantitative claim needs to be either fixed with a proper equivalence judgment task or walked back to \"comparable performance with a slight edge that may vanish under stricter labels.\"\n\nFor peer review: yes, send it out. The topic is legitimate, the task formulation is new, and the qualitative analysis is carefully done. Referees should push for a better metric or a softer claim, and for public artifacts. I would probably cite the task formulation and the qualitative findings, not the F1 numbers.","headline":"A legitimate task formulation and an honest qualitative analysis, but the paper's headline F1 gap between BGE-m3 and BM25 shrinks to about one point under its own stricter labels.","tokens_in":8619,"tokens_out":3105,"would_cite":true,"duration_ms":29346,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating survey harmonisation as a search problem, the paper finds that an IR-specialised neural model, BGE-m3, reaches the top F1 of 0.79, while the lexical baseline BM25 trails by only 0.04.","keywords":["survey harmonisation","longitudinal surveys","information retrieval","semantic equivalence","BM25","neural retrieval","concept granularity","M3-Embedding"],"falsifier":"Take a random sample of top-1 retrievals from BM25 and BGE-m3 and have survey specialists score true equivalence; if BM25's expert-agreement rate equals or exceeds BGE-m3's, the claim that IR-specialised models are best would be overturned. A second check: re-run the evaluation using the 120 fine-grained sub-topic codes instead of the 16 top-level codes; if the ranking of models changes, the paper's headline comparison is an artefact of label granularity.","tokens_in":7617,"feed_emoji":"📋","tokens_out":4817,"duration_ms":41534,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf information-retrieval methods, originally built for search, can decide when two survey questions from different waves of longitudinal studies measure the same thing. It frames harmonisation as a retrieval task: for each question, with its response options attached, the model must find the most conceptually equivalent question in a corpus of 42,161 questions spanning 1946 to 2020. The central result is that an IR-specialised neural model, BGE-m3, reaches the highest $F_1$ of 0.79, while the lexical baseline BM25 reaches $F_1=0.75$. Re-ranking BM25 results with neural models adds at most 0.07 in $F_1$. Expert inspection of retrieved pairs shows that all models are least reliable exactly where lexical overlap is high but the sub-concept differs, so the practical bottleneck is concept granularity rather than vocabulary matching.","feed_headline":"Neural retrieval edges BM25 by 0.04 F1 on survey harmonisation","feed_subtitle":"Across 75 years of surveys, lexical matching stays competitive and models miss sub-concept mismatches.","key_machinery":"The central object is a retrieval task built by concatenating each survey question with its response options into a single input sequence, so that the construct being measured is represented jointly by question text and answer categories. For a query, each model ranks all other questions and the top-1 hit is judged correct if its top-level topic code, one of 16 broad topics such as Education or Health, matches the query's code; a separate expert labelling of 203 random pairs adds finer labels: exact match, equivalent, sub-concept mismatch, and total mismatch. The compared machinery includes BM25 term-frequency ranking; bi-encoder embeddings from SBERT, DeBERTa-v3, and Qwen-2.5 with mean-pooled or summary-token representations; and M3-Embedding (BGE-m3), whose weighted combination of dense, sparse, and multi-vector scores is what gives it the small edge over BM25.","core_discovery":"The paper claims that conceptual equivalence between longitudinal survey questions can be treated and evaluated as an information-retrieval problem, and that in this framing no method clearly dominates. IR-specialised embeddings (BGE-m3) achieve the best overall score ($F_1=0.79$), but the probabilistic lexical model BM25 is surprisingly close ($F_1=0.75$), and neural re-ranking of BM25's top-50 outputs improves $F_1$ by at most 0.07. The authors conclude that lexical matching remains essential, that a balance of syntactic and semantic signals matters more than the depth of semantic modelling, and that the main failure mode is insensitivity to sub-concept mismatches in questions that share surface wording.","pith_inferences":["If the evaluation were refined to require exact sub-topic or expert-level equivalence instead of matching one of 16 broad topics, the gap between BGE-m3 and BM25 could shrink or invert, since the expert sample shows both models commit similar rates of sub-concept mismatch.","The concatenation of response options may be inflating lexical-overlap signals; a controlled test using question text alone, response options alone, and both together would isolate where the retrieval signal actually lives.","The same retrieval framing could transfer to harmonising codebooks, variable labels, or open-ended survey instruments in other cross-cultural or cross-temporal datasets, where concept drift is the same problem.","A testable extension is to build a sub-concept-aware retriever that uses the 120 fine-grained sub-topics as training signal, directly targeting the failure mode identified here."],"forward_implications":["Adding a neural semantic layer to survey harmonisation buys only a small gain over simple lexical matching; teams with limited resources can rely on BM25 as a strong baseline.","Because re-ranking BM25's top-50 results with neural models improves $F_1$ by at most 0.07, the expensive step is the initial search over all questions, not the final ranking.","Mean-pooled representations consistently beat summary-token representations, suggesting that for this task shallower semantic encoding is preferable, which challenges the assumption that deeper semantics always helps text matching.","The main obstacle to automated harmonisation is sub-concept mismatches: models retrieve questions that look alike but measure a different construct, so future approaches should focus on concept granularity rather than vocabulary."],"supporting_citations":[{"why":"Supplies the probabilistic lexical baseline (BM25) whose competitiveness anchors the comparison.","marker":"[14]"},{"why":"Supplies the IR-specialised model (BGE-m3) that achieves the highest F1.","marker":"[1]"},{"why":"Provides the DeBERTa-v3 encoder used for linear-probing reader embeddings.","marker":"[6]"},{"why":"Provides the Qwen decoder used for linear-probing reader embeddings.","marker":"[19]"},{"why":"Provides the SBERT sentence-transformer baseline for bi-encoder retrieval.","marker":"[13]"},{"why":"Provides the efficient bm25s implementation used in the experiments.","marker":"[10]"},{"why":"Supplies the late-interaction framework used for efficient pre-computed representations.","marker":"[8]"},{"why":"Is the XLM-RoBERTa backbone of the BGE-m3 model.","marker":"[2]"}],"fun_headline_variants":["BM25 trails best neural model by just 0.04 F1","Survey harmonisation: lexical BM25 nearly matches neural IR","Sub-concept mismatches trip even best survey matchers","IR for survey questions: no method dominates, BM25 close","Neural reranking aids survey matching only marginally"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a retrieved question is correct when its coarse 16-topic code matches the query's; this proxy can mark a genuinely mismatched sub-concept as a hit, so the reported F1 numbers likely overstate true harmonisation quality.","fun_headline_variants_meta":{"raw":{"variants":["BM25 trails best neural model by just 0.04 F1","Survey harmonisation: lexical BM25 nearly matches neural IR","Sub-concept mismatches trip even best survey matchers","IR for survey questions: no method dominates, BM25 close","Neural reranking aids survey matching only marginally"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2866,"prompt_tokens":930,"completion_tokens":1936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":546,"tokens_out":1936,"duration_ms":14691,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:22:29.739129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of top-1 retrievals from BM25 and BGE-m3 and have survey specialists score true equivalence; if BM25's expert-agreement rate equals or exceeds BGE-m3's, the claim that IR-specialised models are best would be overturned. A second check: re-run the evaluation using the 120 fine-grained sub-topic codes instead of the 16 top-level codes; if the ranking of models changes, the paper's headline comparison is an artefact of label granularity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the late-interaction framework used for efficient pre-computed representations."}],"review_version":1}