{"id":"da4d1024-2f56-4b24-8333-e46b6a1ec4e5","arxiv_id":"2501.14998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-product RAG framework multiplies domain and passage relevance scores with a stochastic gate and reports gains over baselines on new Adobe QA data.","lead":"Researchers built a question-answering system for Adobe products that searches multiple product manuals at once, scoring each result by both which product it belongs to and how well it matches the question. The system beat three simpler baselines on its own tests, but those tests rely on the same data used to train it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split is described; evaluation may be on training data, so the claimed gains over baselines are not yet evidence of generalization.","rationale":"The method is internally coherent and the new Adobe datasets could be a useful contribution if released, but the empirical claim in the abstract is only as strong as the evaluation. Sections 3, 4, and 5 do not state that model training used a subset disjoint from evaluation; Section 4 describes a single dataset with GPT-4-generated queries and GPT-4 relevance labels, and Section 5.3 evaluates on that same dataset. Without a split, the reported Acc@Top1 and response quality values can reflect memorization rather than multi-domain search quality. The reader's weakest assumption identifies exactly this issue, and I agree. Additional concerns, such as the fairness of comparing a trained router against the zero-shot LFS baseline and the absence of seed averaging for the stochastic gate, are secondary but worth addressing in revision. A concrete partition and held-out evaluation would settle the question. I therefore maintain the reader's conditional acceptance: the contribution may be valid, but the evidence needs this fix before the central claim can be accepted as stated.","tokens_in":10426,"tokens_out":3977,"duration_ms":39849,"concrete_test":"Ask the authors to release or precisely describe an exact data partition and rerun Section 5 with all router and retriever parameters trained only on a training split, then evaluated on a disjoint query-level held-out split with no overlapping queries or golden documents, ideally with document URLs disjoint across splits. Report Acc@Top1 and response quality on that held-out split, averaged over at least 5 random seeds for the Bernoulli gate with 95% confidence intervals. If the held-out margin of MKP-QA over RFS is not significantly greater than zero, the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim rests on an evaluation that, as described, has no train/test separation. Section 3 fine-tunes the BERT domain router and the Sentence-BERT retriever on 'our multi-domain dataset'; Section 4 describes building that dataset from ExL pages, with GPT-4 generating queries and GPT-4 annotating relevance labels; Section 5.3 then reports Acc@Top1 and response quality on 'our datasets' without ever stating that a held-out split was used. Consequently, the same query-document pairs that define training labels appear to be the test set. Under this reading, memorization of golden documents and router exposure to query-domain labels can inflate Acc@Top1 and downstream faithfulness, so the reported margin over RFS and UIS is not evidence that MKP-QA generalizes to new queries. The missing ablation of the stochastic gating component makes it harder to attribute gains to the proposed mechanism, and the zero-shot LFS baseline is favorable to MKP-QA because the router is trained on the target domains, but the missing data split is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MKP-QA, a framework for multi-product retrieval-augmented question answering over enterprise product documentation. It combines a fine-tuned BERT domain router, an entropy-based adaptive threshold with stochastic gating, a Sentence-BERT bi-encoder retriever, and a federated aggregation score U = p_j·s_i^j that multiplies the domain probability by the retriever score. The authors also introduce and describe new query-document datasets for three Adobe products, with both uni-domain and cross-domain splits. Experiments compare retrieval accuracy (Acc@Top1) and response quality (relevancy and faithfulness) against unified search (UIS), router filter and search (RFS), and zero-shot LLM resource selection (LFS).","tokens_in":10683,"tokens_out":3999,"duration_ms":34345,"significance":"If the empirical claims were validated, the contribution would be a simple, practical mechanism for domain-aware federated retrieval in RAG pipelines, plus new benchmark datasets for an underexplored setting. The proposed architecture is straightforward and does not require domain-specific LLM fine-tuning. However, because the evaluation has fundamental validity gaps (no described train/test separation, LLM-based data creation and judgment, and an unfavorable zero-shot baseline), the significance cannot currently be assessed. The paper's strengths include a clear problem definition, a fairly detailed description of dataset construction, and the release of new datasets (pending approval).","major_comments":[{"comment":"The paper never states that the evaluation in Section 5 is performed on a held-out subset of the data. Section 3 fine-tunes the router and retriever on 'our multi-domain dataset,' Section 4 describes building that dataset from ExL pages, and Section 5.3 reports results on 'our datasets' without mentioning a split. If the same query-document pairs are used for training and testing, the reported Acc@Top1 and response-quality gains may reflect memorization rather than generalization, undermining the central claim in the abstract. The authors must explicitly describe the train/test split, or if none exists, redo the evaluation with a properly held-out set.","section":"5.3"},{"comment":"GPT-4 is used to generate queries, annotate query-document relevance, and judge the quality of generated responses, as shown in Figures 5-8. This creates a systematic evaluation circularity: the retriever is trained on GPT-4's relevance labels, and the same model family then measures whether retrieval and responses are correct. The reported gains could be inflated by the judge preferring content that matches GPT-4's own generation style. The authors should include a human evaluation on a sample, or at least use a different model for judging than for data creation.","section":"4.3/5.2"},{"comment":"The LFS baseline uses GPT-4 in a zero-shot fashion for domain selection, while MKP-QA's router is fine-tuned on the target domains with labeled queries. This is not an apples-to-apples comparison: the comparison conflates the contribution of the probabilistic federated search mechanism with the benefit of supervised training on the target domain. The paper should include a supervised version of LFS, or a zero-shot variant of MKP-QA, to isolate the effect of the proposed method.","section":"5.1"},{"comment":"No ablation is reported for the core stochastic gating mechanism or the adaptive threshold. The unified score U = p_j·s_i^j is a simple product, and it is unclear how much of the improvement comes from the gating stochasticity and entropy-based threshold versus the simple multiplicative re-ranking. The experiments should include a deterministic top-domain baseline, a fixed-threshold variant, and possibly a rank-fusion baseline to attribute the gains.","section":"3.2/3.4"},{"comment":"No error bars, confidence intervals, or significance tests are reported; all results appear to come from single runs. Given that the claimed improvements are described as 'significant,' the paper needs at least multiple seeds or bootstrap intervals to support this language.","section":"5.2"}],"minor_comments":[{"comment":"There are typos in this section: 'Bernouli' should be 'Bernoulli' and 'utilizie' should be 'utilize'.","section":"3.2"},{"comment":"The sentence beginning 'To effectively estimate the query-domain relevance scores.' ends with a period mid-sentence; the period should be a comma, and the sentence should continue without a break.","section":"3.1"},{"comment":"The captions for Figures 1 and 2 are minimal; they should indicate more explicitly what each panel shows, particularly which methods are compared and what the axes represent.","section":"Figures 1 and 2"},{"comment":"The 'Path to Deployment' section reads like an extended implications or future-work discussion; consider moving it to a shorter subsection or integrating it into the conclusion to keep the paper focused.","section":"Section 6"},{"comment":"The sentence 'This approach ensured that positive document pairs per query were designed to span different domains' is grammatically awkward; it should be clarified, for example, 'Each positive pair was designed to span different domains.'","section":"4.3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the missing train/test split, which could be an oversight in writing rather than a fundamental flaw; the authors should be asked to clarify. The comparison to LFS is also important. The 'Path to Deployment' section is unusual and may be trimmed. The dataset availability is conditional, which should be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper proposes a multi-product RAG pipeline that combines a fine-tuned BERT router, a Sentence-BERT retriever, and an entropy-based stochastic gate that decides which domains to search. The product of the router's domain probability and the retriever's passage score is a sensible way to combine evidence. That specific combination, with the adaptive threshold, is new to me, and it's a reasonable engineering idea. The Adobe datasets for three experience products are also potentially useful if they get released, though that's still pending approval.\n\nThe problem is the evaluation. Section 3 says the router and retriever are fine-tuned on 'our multi-domain dataset'; Section 4 describes building that dataset; Section 5 reports Acc@Top1 and response quality on those same datasets. There is no sentence anywhere describing a train/test split. So, as written, the model is evaluated on the same query-document pairs used for training. That would inflate retrieval accuracy and faithfulness in obvious ways. The stress-test note is right, and the paper does not answer it. This is the load-bearing flaw.\n\nThe evaluation also has secondary issues. The same GPT-4 process generates the queries, annotates relevance, and judges response quality, which is a known circularity. The LFS baseline uses GPT-4 zero-shot for domain selection while the proposed router is fine-tuned on the target domains, so the comparison is not apples-to-apples. There are no error bars or significance tests, and no ablation of the gating mechanism, so we don't know which component drives the gains. On the plus side, the paper is honest about its own scope, and the related work is handled cleanly.\n\nBottom line: The framework is plausible and the datasets fill a gap, but the central empirical claim is not yet supported. This is exactly the kind of paper that should get peer review, because the issues are fixable with a proper evaluation: state the split, report confidence intervals, release code and data, and take the LFS baseline on equal terms. I'd send it to review with a clear request for major revision. I would not cite it yet.\n\nThe thinking is coherent and the writing is straightforward; this is a solid applied paper in need of a rigorous eval, not a shaky idea.","headline":"Plausible multi-product RAG framework with a clever gating idea, but the evaluation has no described train/test split and uses the same GPT-4 for label creation and judging, so the reported gains are not yet convincing.","tokens_in":11159,"tokens_out":2429,"would_cite":false,"duration_ms":20877,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Probabilistic federated search lifts multi-product RAG-QA","keywords":["retrieval augmented generation","federated search","multi-domain question answering","stochastic gating","domain router","enterprise product QA","retrieval accuracy","RAG evaluation"],"falsifier":"Re-run the evaluation on a held-out set of query-document pairs from the same three Adobe products that were excluded from fine-tuning the router and retriever. If MKP-QA's Acc@Top1 falls to the level of the unified-index baseline on that held-out set, the reported advantage is an artifact of training on the test data; a further check would test whether any evaluation query-document pair appears verbatim in the fine-tuning set.","tokens_in":10154,"feed_emoji":"🔎","tokens_out":6724,"duration_ms":49918,"temperature":0.7,"pith_summary":"This paper claims that multi-product retrieval-augmented question answering over enterprise documentation can be improved by replacing indiscriminate all-domain search and rigid single-domain routing with a probabilistic federated search. The proposed MKP-QA framework multiplies each domain's router probability by the retriever's document relevance score, selects domains through a stochastic gate with an entropy-adaptive threshold, and aggregates the resulting scores into a unified ranking. On new datasets built from documentation for three Adobe products (Experience Platform, Target, and Customer Journey Analytics), the paper reports higher top-1 retrieval accuracy and better LLM response faithfulness and relevancy than three baselines. The paper also contributes these datasets as multi-domain RAG-QA benchmarks for the three products. A sympathetic reader would care because the claimed approach improves cross-product answers without domain-specific LLM fine-tuning.","feed_headline":"Probabilistic federated search lifts multi-product RAG-QA","feed_subtitle":"Stochastic domain gating plus score aggregation beats unified search and rigid routing on Adobe product docs.","key_machinery":"The central object is the unified domain-aware retrieval score $U(j,q,d_i^j) = p_j \\cdot s_i^j$, the product of the domain router's sigmoid probability for domain $j$ and the bi-encoder retriever's dot-product similarity for document $d_i^j$. This product is the mechanism that fuses routing softness with passage relevance. Around it sits the stochastic gate $G(q,j) = \\mathrm{Bernoulli}(\\min(1, p_j/\\tau(q)))$, with adaptive threshold $\\tau(q) = \\tau_0(1 - H(q)/\\log m)$, where $H(q)$ is the entropy of the router's probability vector; the gate decides which domains are searched, providing exploration of low-confidence domains while still exploiting high-confidence ones. The federated search then takes the top-$k$ documents by $U$ across the active domains. This machinery is what the paper credits for both retrieval accuracy and response quality gains.","core_discovery":"The paper's central claim is that MKP-QA \"significantly boosts multi-product RAG-QA performance in terms of both retrieval accuracy and response quality\" relative to unified index search (UIS), hard router filtering (RFS), and zero-shot LLM resource selection (LFS). In MKP-QA, a fine-tuned BERT-based domain router outputs query-domain probabilities $p_1,\\dots,p_m$, and a Sentence-BERT bi-encoder trained with a symmetric InfoNCE loss produces query-document scores $s_i^j$. These are combined into a unified score $U(j,q,d_i^j)=p_j \\cdot s_i^j$, and an active set of domains is sampled as $G(q,j) \\sim \\mathrm{Bernoulli}(\\min(1, p_j/\\tau(q)))$, where the threshold $\\tau(q)$ shrinks as the router's entropy rises. The final top-$k$ documents are those with the largest $U$ across active domains. The paper reports that this construction outperforms all three baselines on both uni-domain and cross-domain subsets of the Adobe datasets, with the largest gaps on cross-domain queries.","pith_inferences":["The paper does not ablate the stochastic gate, so the marginal contribution of Bernoulli sampling versus deterministic thresholding is unmeasured; a follow-up ablation would tell whether exploration is the source of the cross-domain gain or whether the $p_j$ multiplication alone carries it.","Because negative pairs are sampled from the same or closely related documentation pages, the benchmark is easier than a real federated search with unrelated corpora; adding unrelated distractor domains would test the unified score's ability to reject irrelevant resources.","The entropy-adaptive threshold implies a precision-recall trade-off across domains that the paper does not plot; varying $\\tau_0$ and measuring Acc@Top1 against the number of active domains would map that frontier.","All response-quality judgments use GPT-4 or GPT-3.5; whether the gains persist under smaller open-weight LLMs or other enterprise product suites is an untested extrapolation."],"forward_implications":["On the Adobe datasets, MKP-QA's top-1 retrieval accuracy exceeds unified index search, hard router filtering, and zero-shot LLM resource selection, and the gap grows on cross-domain queries.","LLM response quality, scored by GPT-4 relevancy judgment and RAGAS faithfulness, is also higher under MKP-QA than under the three baselines.","The released datasets provide the first multi-product RAG-QA benchmarks for Adobe Experience Platform, Target, and Customer Journey Analytics, with both uni-domain and cross-domain query-document pairs.","The framework achieves its gains without domain-specific LLM fine-tuning or adaptive modules: only the domain router and the bi-encoder retriever are trained.","Zero-shot LLM resource selection (LFS) is the weakest baseline, suggesting that learned query-domain relevance is more reliable than a general-purpose LLM's domain judgment for these product domains."],"supporting_citations":[{"why":"supplies the federated search formulation that MKP-QA adapts for multi-domain retrieval","marker":"Shokouhi and Si, 2011"},{"why":"provides the Sentence-BERT bi-encoder architecture used as the query-document retriever","marker":"Reimers, 2019"},{"why":"establishes dense passage retrieval, the retrieval paradigm the bi-encoder inherits","marker":"Karpukhin et al., 2020"},{"why":"defines the contrastive InfoNCE loss that the retriever's symmetric training objective is built on","marker":"Oord et al., 2018"},{"why":"supplies the ReSLLM zero-shot resource selection method used as the LFS baseline","marker":"Wang et al., 2024b"},{"why":"provides the RAGAS faithfulness evaluation used to measure response quality","marker":"Es et al., 2023"},{"why":"grounds the GPT-4 relevancy-judge prompt used for response evaluation","marker":"Zheng et al., 2024"},{"why":"gives evidence that the top-ranked document most influences LLM responses, motivating Acc@Top1 as the retrieval metric","marker":"Liu et al., 2024"}],"fun_headline_variants":["Fusing domain and passage relevance boosts RAG QA","Stochastic domain gating sharpens multi-product retrieval","MKP-QA: probabilistic federated search for cross-domain QA","Better multi-domain answers via score fusion and active sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the queries and document pairs used for evaluation were not also used to train the domain router and retriever; the paper does not report a train/test split, so the reported Acc@Top1 gains could partly reflect memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["Fusing domain and passage relevance boosts RAG QA","Stochastic domain gating sharpens multi-product retrieval","MKP-QA: probabilistic federated search for cross-domain QA","Better multi-domain answers via score fusion and active sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1785,"prompt_tokens":945,"completion_tokens":840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":774}},"tokens_in":561,"tokens_out":840,"duration_ms":7576,"temperature":1.0,"reasoning_tokens":774,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:42:57.923624+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on a held-out set of query-document pairs from the same three Adobe products that were excluded from fine-tuning the router and retriever. If MKP-QA's Acc@Top1 falls to the level of the unified-index baseline on that held-out set, the reported advantage is an artifact of training on the test data; a further check would test whether any evaluation query-document pair appears verbatim in the fine-tuning set.","supporting_citations":[],"review_version":1}