{"id":"fdb9abbe-f22e-454b-aaa9-14b740aa5588","arxiv_id":"2607.03028","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HETERQA builds 857 answer-driven multi-source Yelp record-retrieval questions and finds hybrid and agentic methods still far from saturating Recall@10 and MRR@10.","lead":"HETERQA is a new 857-question benchmark for retrieving whole business records that must jointly satisfy constraints from five data types: tables, text, images, maps, and knowledge graphs. It shows current sparse, dense, hybrid, late-interaction, and agentic retrievers still leave most verified answers unrecovered.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Gold-label reliability is the load-bearing risk: automated V*_H may understate true recall and overstate challenge.","rationale":"The paper’s contribution is real: five-source record retrieval with missing-value recovery is under-served, the answer-driven workflow is documented, artifacts are public, and baselines are broad. The reader correctly isolates the weakest assumption—gold V*_H from LLM/VLM thresholds and limited human review—and that remains the single load-bearing concern for the claim that current methods are far from saturating a high-quality gold set. Human validation does not close this gap because it scores question quality, not answer-set correctness. Domain specificity (Yelp) and modest absolute scores are secondary. A targeted human re-label of answer sets would settle whether the challenge is genuine or partly label-induced; until then CONDITIONAL with moderate confidence is appropriate, and no stronger rejection is warranted.","tokens_in":22366,"tokens_out":586,"duration_ms":5149,"concrete_test":"Independently re-annotate a stratified sample of ~100 QA pairs (all ten source-composition subsets): for each q, present annotators with the full candidate pool C (or top recovered candidates) and require binary multi-source satisfaction labels without seeing automated V*_H. Report (i) precision/recall of automated V*_H vs human gold, (ii) inter-annotator agreement, and (iii) re-scored Recall@10/MRR@10 for the Table 3 hybrid-Llama and Self-RAG runs under human gold. If automated recall of true positives is <0.85 or metric deltas exceed ~5 absolute points, the saturation claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that HETERQA is a high-quality, challenging five-source record-retrieval benchmark rests on treating verified answer sets V*_H as reliable gold for Recall@10/MRR@10. Construction Steps 3–5 and Appendix B.2 decide membership via LLM/VLM support judgments (llm_judge_threshold=0.7), embedding/rerank thresholds, and contradiction ratios ρT/ρI with τc=0.15, plus manual review only for unresolved cases. Human validation (Table 2) rates question naturalness/diversity/practicality on 200 items, not whether every retained (or discarded) record truly satisfies all constraints in q. If judges systematically drop valid multi-source matches or retain weakly supported ones, the reported ceiling (hybrid Llama 32.78 Recall@10; Self-RAG 25.26 MRR@10) partly measures label noise and threshold conservatism rather than irreducible retrieval difficulty. That is the softest condition for the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces HETERQA, a benchmark of 857 natural-language QA pairs for target-record retrieval over five heterogeneous sources—relational tables, text documents, image repositories, spatial databases, and knowledge graphs—instantiated on Yelp business records. Construction is answer-driven: relational-field constraints initialize an answer set, missing-value recovery expands candidates from non-relational sources, source-specific constraints are instantiated and cross-verified, and contradiction detection plus limited human review produce verified answer sets V*_H. The authors report human ratings of question naturalness, diversity, and practicality on a stratified sample of 200 questions, release the dataset and code, and evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers under shared Hit/Recall/MRR metrics. Empirically, hybrid retrieval with Llama and reranking reaches 32.78 Recall@10 and Self-RAG reaches 25.26 MRR@10, which the authors interpret as evidence that current methods remain far from saturating five-source record retrieval.","tokens_in":22720,"tokens_out":1437,"duration_ms":17835,"significance":"If the verified answer sets are sufficiently reliable, HETERQA fills a clear gap relative to prior multi-source QA/retrieval benchmarks (Table 1), which cover at most three sources and typically omit missing-value recovery and full-collection target-record ranking. The answer-driven workflow, explicit missing-value recovery, public Hugging Face/GitHub release, and broad baseline suite (including hybrid fusion, ColBERT/ColPali, ReAct, and Self-RAG) are concrete strengths that make the resource usable as a testbed. Source-combination heatmaps and the RAG record-selection study further surface where methods fail (e.g., spatial–KG subsets and generator-side selection). The contribution is therefore primarily empirical and infrastructural rather than theoretical; its lasting value hinges on gold-label quality and on how far the Yelp instantiation generalizes.","major_comments":[{"comment":"§3 Steps 3–5 and Table 2: the central hardness claim (hybrid Recall@10 32.78; Self-RAG MRR@10 25.26) treats V*_H as gold, but human validation rates only question naturalness/diversity/practicality on 200 items—not whether retained or discarded records correctly satisfy all constraints in q. Membership in V*_H is decided largely by LLM/VLM support judgments (llm_judge_threshold=0.7), embedding/rerank thresholds, and contradiction ratios with τc=0.15 (Appendix B.2), with manual review only for unresolved cases. Without a human audit of answer-set precision/recall (or inter-annotator agreement on a stratified sample of (q,r) pairs), reported ceilings may partly reflect label conservatism or judge noise rather than irreducible retrieval difficulty. A load-bearing revision is a quantified answer-set validation study and, if needed, relabeling.","section":null},{"comment":"Appendix B.2 and Table 6: construction depends on several free thresholds (contradiction_ratio_thres=0.15, text/image coarse and rerank thresholds, recovery top_k=300, judge confidence 0.7) with no sensitivity analysis of how V*_H size or baseline metrics change under plausible alternatives. Because missing-value recovery and cross-source verification are advertised as defining features, the paper should show that the “challenging but high-quality” conclusion is stable under threshold perturbation, or else report confidence intervals / alternative gold sets.","section":null},{"comment":"§4.1 and Observation 7: spatial constraints are evaluated by embedding location metadata inside serialized relational fields rather than with a dedicated spatial index or geo operators at retrieval time, while agentic methods alone receive an explicit Geo_Filter tool (Table 9). This asymmetry weakens the claim that spatial–KG subsets are intrinsically hard for standard retrievers; low scores may partly measure representation choice. Either equip non-agentic baselines with comparable spatial operators, or reframe the spatial results as a serialization baseline and qualify Observation 7 accordingly.","section":null},{"comment":"§1, Table 1, and §6: the paper positions HETERQA as a comprehensive five-source record-retrieval benchmark, yet it is a single-domain Yelp instantiation with 857 pairs and average |V*_H|≈2.3. That scale is usable, but claims about “emerging systems” and extensibility should be tempered until at least one additional domain or a larger held-out construction pass is shown, or until the limitations section more sharply bounds external validity. Extending the workflow is asserted but not demonstrated.","section":null}],"minor_comments":[{"comment":"§5 Conclusion: duplicate wording “Source-combination and RAG analyses analyses further show…”.","section":null},{"comment":"Table 3 notes and §4.1: ColPali has no w/ R column and near-zero scores; briefly state whether page rendering (Figure 7) was tuned or is an off-the-shelf stress test so readers do not over-interpret the late-interaction comparison.","section":null},{"comment":"Figure 3 heatmaps: color scales are clipped at ±15 / ±31; state clipping explicitly in the caption body (not only the legend note) and consider reporting absolute subset Recall@10 in an appendix table for readability.","section":null},{"comment":"§2.2: typo “hterogeneous” → “heterogeneous”.","section":null},{"comment":"Table 1 and related work: briefly clarify how HETERQA’s record-level ranking task differs from MMQA’s question-specific multimodal contexts so the “five sources” novelty is not conflated with multimodal QA context packing.","section":null},{"comment":"Appendix A KG construction: report a small human check of feature extraction/canonicalization quality (beyond redundancy score), since KG constraints enter both construction and subset analysis.","section":null},{"comment":"§4.4 RAG setup: Macro-F1 is appropriate, but also report exact-set-match rates (mentioned in Appendix E) in the main table so the gap between retrieval Recall@10 and generator selection is fully transparent.","section":null}],"recommendation":"major_revision","confidential_remarks":"The contribution is real if gold labels hold; the main editorial risk is accepting a “challenging benchmark” narrative whose ceilings may be partly threshold artifacts. I would prioritize requiring a human answer-set audit over expanding domains. Fit for a solid IR/data-centric venue is good after revision; without label validation I would not treat the hardness numbers as definitive."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a solid evaluation paper, not a theory paper. What is new is a public 857-pair Yelp benchmark that forces full-collection record retrieval over five sources—tables, text, images, spatial, and a feature KG—with explicit missing-value recovery. Prior multi-source QA sets mostly stop at two or three modalities and often do not score ranked target records over a full collection. That gap is real, and they fill it with a clear answer-driven pipeline, released data/code, and a fair baseline sweep (sparse, dense, hybrid, late-interaction, agentic) under shared metrics.\n\nWhat they do well: construction is documented end-to-end (field tiers, recovery, support sets, contradiction audit, ranking score). Human checks on 200 stratified questions look decent for naturalness/diversity/practicality. Experiments are honest about non-saturation: best hybrid Recall@10 is only ~33, Self-RAG wins MRR but not recall and is slow, spatial+KG subsets stay hard, and RAG selection F1 stays low. Hybrid gains over pure dense are larger than reranking gains—that is a useful empirical takeaway. Citation coverage of HybridQA/OTT-QA/MMQA/STARK/text-to-SQL is appropriate; they are not inventing a vacuum.\n\nSoft spots, in proportion. The load-bearing risk is gold-label reliability: V*_H membership depends on embedding/rerank thresholds, LLM/VLM judges (0.7), and contradiction ratio 0.15, with humans mainly on unresolved cases and on question quality—not full audit of every retained/discarded record. So the “far from saturated” numbers partly measure label conservatism as well as task hardness. That does not sink the benchmark, but users should treat absolute ceilings carefully and ideally re-audit a slice. Scope is Yelp-local-business only; the workflow is extensible in principle, not yet multi-domain. Free parameters (thresholds, top_k=300, RRF settings) are many but disclosed. No math to break; this is empirical IR.\n\nWho it is for: people building multi-source retrieval/RAG for platforms, and anyone who needs a harder record-level test than table-text or STARK-style setups. I would bring it to reading group as a methods/eval paper, cite it when I need a multi-source record benchmark, and send it to peer review. It deserves referee time; ask for clearer gold-label error analysis and maybe a small human answer-set audit, not a rewrite.","headline":"Useful five-source record-retrieval benchmark with public data; hardness claims are real enough to use, but gold labels rest partly on LLM/VLM thresholds.","tokens_in":23296,"tokens_out":615,"would_cite":true,"duration_ms":7460,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"HETERQA shows current retrievers still recover only a fraction of records that must jointly satisfy five heterogeneous sources.","keywords":["record retrieval","heterogeneous sources","question answering","benchmark","hybrid retrieval","missing-value recovery","multi-source QA","agentic retrieval"],"falsifier":"Re-annotate a stratified sample of questions with independent human verification of the gold record sets, then re-score the same hybrid and Self-RAG systems: if Recall@10 and MRR@10 rise sharply under the human gold, the claim that current methods are far from saturating the task weakens; if the gold sets and low scores hold, the claim stands.","tokens_in":23272,"feed_emoji":"🔍","tokens_out":918,"duration_ms":17430,"temperature":0.7,"pith_summary":"HETERQA is an 857-question benchmark for retrieving whole business records when a natural-language query imposes constraints drawn from relational tables, reviews, photos, spatial data, and a knowledge graph at once. The authors build each question answer-first: they start from relational field constraints, recover additional valid records whose incomplete table fields are evidenced in other sources, add source-specific constraints, then keep the question only after cross-source verification and contradiction filtering leave a non-empty gold set. Human checks support the questions as largely natural, diverse, and practical. Under shared metrics, hybrid retrieval reaches the best Recall@10 near 33 percent and Self-RAG the best MRR@10 near 25 percent, while sparse, dense, late-interaction, and agentic systems all remain far from saturating the task. The paper therefore presents HETERQA as a hard, high-quality testbed for multi-source record retrieval and as evidence that existing methods leave substantial room for improvement.","feed_headline":"Five-source record retrieval stalls near 33% recall","feed_subtitle":"HETERQA’s 857 Yelp questions show hybrid and agentic methods still miss most multi-constraint answers.","key_machinery":"Answer-driven construction with missing-value recovery: initialize candidates via relational-field constraints, enrich them by recovering incomplete fields from text, image, and knowledge-graph evidence, instantiate additional source-specific constraints, then retain a question only after cross-source support checks and contradiction filtering produce a non-empty verified answer set.","core_discovery":"Existing QA and retrieval benchmarks cover at most three sources and ignore missing-value recovery, so they understate the difficulty of returning records that must satisfy every constraint in a heterogeneous bundle. HETERQA closes that gap with 857 verified Yelp-based questions over five sources, and the evaluated sparse, dense, hybrid, late-interaction, and agentic methods all recover only a limited share of the verified answer sets.","pith_inferences":["Progress may require explicit multi-source verification modules, not only stronger single-tower embeddings or longer agent loops.","The same answer-driven recipe could transfer to product catalogs or scientific entity records wherever fields are incomplete across modalities.","The low RAG selection scores suggest record-set selection under multi-source constraints is a separate bottleneck from retrieval itself.","Latency-aware designs that use sparse/dense hybrid indexing with selective tool calls may be more practical than long multi-turn agents for this task."],"forward_implications":["Hybrid fusion of sparse and dense signals recovers more verified multi-source records than dense or late-interaction retrieval alone.","Agentic methods can improve early ranking metrics but do not solve recall and incur large latency costs.","Questions that combine spatial constraints with knowledge-graph patterns remain especially hard for current systems.","Downstream generators still struggle to select the correct verified record set even when given top-30 retrieved candidates.","Benchmarks and systems that ignore incomplete relational fields will systematically exclude valid target records."],"fun_headline_variants":["HETERQA shows five-source record retrieval still far from saturated","857 multi-source Yelp QA pairs leave hybrid and agentic methods incomplete","Five heterogeneous sources keep top retrievers well below full recall","Record retrieval across tables text images and graphs remains unsolved","HETERQA five-source questions expose wide gaps in hybrid late-interaction retrieval"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The gold answer sets depend on language- and vision-model support judgments plus a fixed contradiction threshold; if those automated filters systematically mislabel which records truly satisfy the question, the ranking metrics no longer measure real multi-source correctness.","fun_headline_variants_meta":{"raw":{"variants":["HETERQA shows five-source record retrieval still far from saturated","857 multi-source Yelp QA pairs leave hybrid and agentic methods incomplete","Five heterogeneous sources keep top retrievers well below full recall","Record retrieval across tables text images and graphs remains unsolved","HETERQA five-source questions expose wide gaps in hybrid late-interaction retrieval"]},"model":"grok-4.5","effort":"low","cost_usd":0.007354,"raw_usage":{"total_tokens":1867,"prompt_tokens":866,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":73540000,"prompt_tokens_details":{"text_tokens":866,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":906,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":866,"tokens_out":95,"duration_ms":6469,"temperature":1.0,"reasoning_tokens":906,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T05:20:54.114370+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-annotate a stratified sample of questions with independent human verification of the gold record sets, then re-score the same hybrid and Self-RAG systems: if Recall@10 and MRR@10 rise sharply under the human gold, the claim that current methods are far from saturating the task weakens; if the gold sets and low scores hold, the claim stands.","supporting_citations":[],"review_version":1}