{"id":"67619bf5-5686-4278-a4e9-e9493ddbf60b","arxiv_id":"2501.16111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning a sentence transformer so query-plus-options embeddings mimic oracle query-plus-answer embeddings improves evidence retrieval and multiple-choice accuracy on QuALITY.","lead":"This paper introduces OADR, a way to fine-tune dense retrieval models for multiple-choice questions by training them to imitate the embedding of the question paired with the correct answer. On the QuALITY long-context benchmark, the method improves answer accuracy over standard retrieval baselines, suggesting that answer-aware retrieval is a practical lever for long-document QA.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuned query embeddings are compared to frozen pre-trained context embeddings at inference (Sec. 3.2, Fig. 1); those spaces are uncalibrated, so OADR's central claim is unsupported as written until this contradiction is resolved.","rationale":"The reader's weakest assumption correctly identifies that answer-label-derived triplets may not transfer to sentence-level relevance. My concern is adjacent but more immediate: the manuscript describes an inference pipeline in which the query encoder is fine-tuned while the document encoder is not. Section 3.2 explicitly says context sentences are encoded by a pre-trained sentence transformer, and Figure 1 depicts separate pretrained and fine-tuned encoders. Because the triplet objective in Section 3.1 only optimizes distances among query-side strings, there is no training signal that aligns the fine-tuned query embedding space to the frozen document embedding space. If this is really the implementation, negative Euclidean distances at inference are not reliable relevance scores; if it is a typo, the paper needs correction and re-evaluation. The t-SNE visualization and oracle-overlap metric in Section 5.2 do not settle this, since they use the same oracle target that defines the training objective and t-SNE placements across different encoder weights are not evidence of meaningful inter-model distances. This is a load-bearing flaw because it determines whether the retrieval method as described can function at all, not merely whether it is optimal. I therefore recommend UNVERDICTED until the encoder setup is disambiguated and Table 1 is reproduced under both variants.","tokens_in":5832,"tokens_out":9405,"duration_ms":92282,"concrete_test":"Obtain or retrain the OADR checkpoint per Section 4.3 and run the QuALITY dev retrieval twice: (i) exactly as written, with the fine-tuned query encoder and the pre-trained context encoder; (ii) with the same fine-tuned encoder also used to encode context sentences. Report oracle-overlap (Table 2) and downstream accuracy (Table 1) for both variants. If variant (ii) does not reproduce or exceed the reported OADR numbers, and variant (i) does not clearly beat the pre-trained-query/pre-trained-context ST baseline, then the central claim depends on the inconsistent encoder pairing.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The most load-bearing concern is an internal architectural inconsistency in the retrieval step. Section 3.2 states that the options-aware query is encoded by a fine-tuned sentence transformer, while 'Context sentences are processed individually using a pre-trained sentence transformer'; Figure 1 also draws separate 'Pretrained S.T.' and 'Fine-Tuned S.T.' boxes. The triplet training in Section 3.1 only involves query-side strings (oracle query, query+all options, query+wrong options), so the fine-tuned weights are never trained against document sentences. If the context encoder remains the original pre-trained model, the L2 distances used to rank sentences are computed between two different embedding spaces and are not semantically calibrated. If the context encoder is meant to be fine-tuned too, then the text and figure are misleading and the reported experiments need to be re-run with that clarified design. The t-SNE and overlap analyses in Section 5.2 do not resolve this: t-SNE distances across differently parameterized models are not meaningful, and Table 2's oracle overlap uses the same oracle target that defines the training objective. Without code or an unambiguous statement that the same fine-tuned model encodes both sides, Table 1's gains cannot be attributed to the proposed method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Options-Aware Dense Retrieval (OADR) for long-context multiple-choice QA. OADR fine-tunes a sentence-transformer retrieval model with a contrastive triplet dataset derived from QuALITY/RACE questions: the anchor is the oracle query (Q + the correct option), the positive is Q + all options, and the negative is Q + the wrong options. At inference, an options-aware query is embedded and used to rank pre-encoded context sentences by negative Euclidean L2 distance; the top-K sentences are concatenated and fed to a standard MCQA model. Experiments on QuALITY report that OADR with DeBERTaV3-large achieves 59.3% full-set and 48.9% hard-set accuracy in the RACE -> QuALITY configuration, outperforming the DPR and pre-trained sentence-transformer baselines.","tokens_in":6132,"tokens_out":7676,"duration_ms":67744,"significance":"If the central result holds, OADR offers a simple, label-efficient way to adapt dense retrieval to long-context MCQA without sentence-level relevance labels, and the Table 1 gains are consistent across several MCQA heads (RoBERTa-base/large and DeBERTaV3-base/large). The final QA accuracy is a falsifiable, independent outcome that is not forced by the training objective, and the use of a standard benchmark (QuALITY) is a strength. However, the paper provides no code, no error bars or significance tests, and no independent retrieval-relevance labels, and the supporting analyses in Section 5.2 are self-referential; the central claim currently rests on a pipeline description that is internally inconsistent (Section 3.2).","major_comments":[{"comment":"The inference procedure as written computes negative Euclidean distances between two embedding spaces that are never jointly trained or calibrated: the options-aware query is encoded by the fine-tuned sentence transformer, whereas each context sentence is encoded by a pre-trained sentence transformer (the text explicitly says 'pre-trained' and Figure 1 draws separate boxes). The triplet training in §3.1 only formats query-side strings (oracle query, query+all options, query+wrong options) and never includes context or document sentences, so the fine-tuned weights are never exposed to the document encoder. If the document encoder is truly frozen, the L2 ranking is between uncalibrated spaces and the Table 1 gains cannot be attributed to the method as described; if the intended design is to use the same fine-tuned model for both sides, the paper must say so and the experiments must be rerun or explicitly confirmed under that design. The t-SNE and overlap analyses in §5.2 do not repair this, because t-SNE distances across different parameterizations are not meaningful and the overlap metric uses the same oracle target used for training.","section":"§3.2 and Figure 1"},{"comment":"The retrieval-overlap evaluation is circular. Table 2 measures the percentage of sentences retrieved by an options-aware query that also appear among the sentences retrieved by the oracle query (Q + correct option), but that oracle query is exactly the anchor used to construct the triplets in §3.1. Consequently, the high overlap of the fine-tuned options-aware embeddings with the oracle retrieval is to be expected from the training objective and is not independent evidence of improved retrieval quality. The abstract's 'retrieval quality' claim should be supported by external relevance judgments, such as human-annotated evidence spans or a retrieval-recall evaluation on a labeled set, rather than by this self-referential overlap.","section":"§5.2, Table 2"},{"comment":"The evaluation protocol is underspecified. Section 4.1 says the validation and test sets each contain about 2,000 samples, but Table 1 reports accuracy on 'full QuALITY' and 'QuALITY-HARD' without stating whether these numbers are on the validation set, the test set, or another partition, and without saying whether model selection was performed on the evaluated set. The meaning of the RACE -> QuALITY column is also not fully defined: the text says RACE is used to train both OADR and the MCQA model, but it is not clear whether OADR is first trained on RACE and then fine-tuned on QuALITY, or whether the datasets are combined in a single training run. These ambiguities are necessary to reproduce or compare the Table 1 numbers.","section":"§4.1 and Table 1"}],"minor_comments":[{"comment":"Training uses a query concatenated with all options, while inference is described as combining the query with each option; the paper should clarify which query string is actually embedded at inference.","section":"§3.1 vs §3.2"},{"comment":"The retrieval model is initialized from 'multi-qa-mpnet-base-dot-v1', which is trained for dot-product similarity, but inference uses negative Euclidean L2 distance; the paper should explain whether the fine-tuned triplet objective intentionally changes the similarity metric.","section":"§4.3"},{"comment":"The retrieval model is fine-tuned for only one epoch with no mention of validation, early stopping, or the number of training queries used from RACE; this detail matters for reproducibility.","section":"§4.3"},{"comment":"No error bars, standard deviations, or significance tests are reported, so the claimed improvements over the DPR and pre-trained-sentence-transformer baselines are not shown to be statistically reliable.","section":"Table 1"},{"comment":"The abstract contains the typo 'ORDA' instead of 'OADR', and the title has an unusual capitalization in 'Multiple-Choice query Answering'.","section":"Abstract"},{"comment":"The caption of Table 2 contains typographical errors ('over-lap', 'sentences retrieved sentences fromOQ') and does not define the abbreviations Q, OAQ, (OAQ)R, and OAQRQ; Figure 2 is based on a single randomly selected dev example and should be supplemented by aggregate distance statistics over the dev set.","section":"Table 2 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an IR and QA venue, but the missing code and ambiguous split make verification difficult. If the authors resolve the encoder-mismatch concern by clarifying the architecture and confirming that the experiments use a single fine-tuned encoder for both query and context (or by rerunning the experiments accordingly), the paper could be suitable for publication. I would urge the editor to require a clear statement of the evaluation split and, ideally, a code/data release, before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Options-aware triplet training is a real, cheap idea: fine-tune a sentence transformer so query-plus-options embeddings sit near query-plus-correct-answer embeddings, then use that for evidence retrieval. The construction is a sensible adaptation of SetFit-style contrastive learning and I don't see it in DPR or Sentence-BERT. Table 1 shows consistent gains across several MCQA heads, and the RACE→QuALITY transfer story is plausible. So the paper earns a serious referee.\n\nThe soft spot is load-bearing. Section 3.2 says context sentences are encoded with a pre-trained sentence transformer while the options-aware query goes through a fine-tuned one; Figure 1 draws that same split. The triplet training in Section 3.1 only involves query-side strings, so the fine-tuned weights never train against document sentences. If the context encoder stays frozen, L2 distances rank sentences across two different embedding spaces, which is not semantically calibrated. If the context encoder is meant to be fine-tuned too, the text and figure are misleading and the reported experiments need to be rerun with that clarified. Either way, the central claim is unsupported as written. This is not a small ambiguity; it is the scoring mechanism.\n\nThe other issues are more ordinary: no code, no error bars, ambiguous evaluation split, and Table 2's overlap metric uses the same oracle target that created the training triplets, so it is self-referential. The t-SNE plot comparing embeddings from differently parameterized models does not establish calibration.\n\nI want to be fair. The core idea is cheap to test and the Table 1 gains are consistent enough that a corrected version might work. But as submitted, the method cannot be run from the description, and the main quantitative claim rests on an encoding choice that is either wrong or misdescribed. This is a conditional reject that a good referee could turn into a modest accept if the encoder question is resolved. I'd send it to review for exactly that reason, with a request for code or an explicit statement of what encodes the context.","headline":"A nice triplet idea undermined by an encoder mismatch between query and context that makes the main result uninterpretable as written.","tokens_in":6626,"tokens_out":3450,"would_cite":false,"duration_ms":29349,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning a sentence transformer so that question-plus-options embeddings mimic question-plus-correct-answer embeddings improves evidence retrieval and multiple-choice QA accuracy on long documents, reaching 59.3%…","keywords":["dense retrieval","multiple-choice question answering","sentence transformer","contrastive training","QuALITY","long-context QA","evidence retrieval","oracle query"],"falsifier":"Retrieve sentences for a long-context QA dataset that carries human-annotated supporting sentences, and compare OADR's retrieved set with the pre-trained sentence transformer's set against those annotations; if OADR shows no gain, the claim that answer-option triplets teach sentence-level relevance is falsified.","tokens_in":5665,"feed_emoji":"🔎","tokens_out":6820,"duration_ms":56033,"temperature":0.7,"pith_summary":"The paper tries to show that dense retrieval for long-context multiple-choice QA can be fine-tuned without sentence-level relevance labels. Its method, OADR, constructs contrastive training triplets from the question and the answer options, treating the question plus the correct answer as the anchor, the question plus all options as the positive, and the question plus the wrong options as the negative, then minimizes triplet loss over a sentence transformer. At inference the same transformer embeds the question-options query and ranks context sentences by Euclidean distance, and the top sentences are fed to a standard MCQA model. On QuALITY, the best configuration—OADR fine-tuned on RACE then QuALITY, with DeBERTaV3-large as the reader—reaches 59.3% on the full set and 48.9% on the hard subset, outperforming pre-trained ST and DPR baselines. The significance is that retrieval can be improved using only answer labels, which are plentiful in MCQA datasets.","feed_headline":"Answer-aware retriever lifts long-context QA to 59.3%","feed_subtitle":"Training retrieval on question-answer pairs, with no sentence labels, improves evidence selection on QuALITY.","key_machinery":"The central object is the options-aware query embedding, built by concatenating the question with the answer options, and the oracle query embedding, built by concatenating the question with the correct answer. The mechanism is a triplet loss over these embeddings: the anchor is the oracle query, the positive is the query plus all options, and the negative is the query plus all wrong options. Fine-tuning a sentence transformer with this loss makes options-aware query embeddings approximate oracle embeddings, and the same encoder is used at inference to score each context sentence against the options-aware query by negative Euclidean distance. The extracted sentences are then sorted in original order, truncated to 300 tokens, and passed to a pre-trained MCQA model (RoBERTa or DeBERTaV3) for answer selection.","core_discovery":"The central claim is that answer-label-derived supervision is sufficient to teach a dense retriever where the supporting evidence for a multiple-choice question lies. The authors show that when a sentence transformer is fine-tuned so that the embedding of the query concatenated with all answer options is pulled toward the embedding of the query concatenated with the correct answer, the transformer's sentence-level rankings align better with those of the oracle query. This improved alignment transfers to downstream accuracy: feeding the retrieved sentences to an MCQA reader yields substantial gains over retrieving with a pre-trained sentence transformer or DPR, and the gain is largest when the retriever is fine-tuned first on RACE and then on QuALITY.","pith_inferences":["Because OADR only needs question-answer pairs, the same triplet construction could be applied to other MCQA datasets with longer contexts, such as legal or medical exams, where sentence-level evidence labels are also missing.","If the oracle-query overlap metric is a fair proxy, then OADR's gains suggest that retrieval fine-tuning is a cheaper path to long-context QA improvements than scaling the reader model.","A natural next test would be to combine OADR with hard-negative mining or with relevance judgments when they are available, which might push retrieval quality further than the current fixed triplets.","The method's dependence on the correct answer means it cannot be applied directly to open-domain retrieval without an answer set; extending it to candidate-answer generation would be a separate step."],"forward_implications":["Training the retriever on RACE followed by QuALITY is essential: it produces the best results (59.3% full, 48.9% hard), showing that auxiliary MCQA data transfers to long-context retrieval.","A 300-token extracted passage is sufficient for DeBERTaV3-large to answer QuALITY questions accurately, meaning retrieval quality, not context length, is the bottleneck the method removes.","The same fine-tuned OADR retriever can be paired with different reader models (RoBERTa-base, DeBERTaV3-base, DeBERTaV3-large) and consistently outperforms the corresponding DPR and pre-trained-ST baselines.","Options-aware query embeddings produced by OADR align more closely with oracle query embeddings than pre-trained options-aware embeddings do, as shown by t-SNE and the overlap metric in Table 2."],"supporting_citations":[{"why":"Introduces the QuALITY benchmark and reports the oracle-query improvement that OADR aims to capture without labels.","marker":"(Bowman et al., 2022)"},{"why":"Dense Passage Retrieval, the baseline dense retrieval approach OADR modifies and compares against.","marker":"(Karpukhin et al., 2020)"},{"why":"Sentence-BERT supplies the sentence transformer encoder that OADR fine-tunes.","marker":"(Reimers and Gurevych, 2019)"},{"why":"SetFit motivates fine-tuning sentence transformers with contrastive triplet data in low-supervision settings.","marker":"(Tunstall et al., 2022)"},{"why":"RACE provides the large auxiliary MCQA training set used for knowledge transfer in the best configuration.","marker":"(Lai et al., 2017)"}],"fun_headline_variants":["Answer-aware retriever boosts long-context MCQA accuracy","Fine-tuning retrieval on answer options sharpens evidence selection","Options-aware retrieval improves multiple-choice QA on QuALITY","Teaching retrieval with answer options lifts QA performance","OADR: Options-aware dense retrieval beats baselines on QuALITY"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a question plus its correct answer is a good proxy for the set of sentences that support that answer, even though the retriever is never trained on sentence-level relevance labels and is evaluated only against the same oracle-query target.","fun_headline_variants_meta":{"raw":{"variants":["Answer-aware retriever boosts long-context MCQA accuracy","Fine-tuning retrieval on answer options sharpens evidence selection","Options-aware retrieval improves multiple-choice QA on QuALITY","Teaching retrieval with answer options lifts QA performance","OADR: Options-aware dense retrieval beats baselines on QuALITY"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2735,"prompt_tokens":844,"completion_tokens":1891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":1824}},"tokens_in":460,"tokens_out":1891,"duration_ms":12576,"temperature":1.0,"reasoning_tokens":1824,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:43:11.867289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrieve sentences for a long-context QA dataset that carries human-annotated supporting sentences, and compare OADR's retrieved set with the pre-trained sentence transformer's set against those annotations; if OADR shows no gain, the claim that answer-option triplets teach sentence-level relevance is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the QuALITY benchmark and reports the oracle-query improvement that OADR aims to capture without labels."}],"review_version":1}