{"id":"0bae9021-ac08-463f-91ae-a6b9701acda6","arxiv_id":"2505.21849","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Xinyu, an integrated generative AI search engine with query decomposition, multi-source retrieval, and rich answer presentation, outperforms eight existing technologies in human evaluations.","lead":"This paper describes Xinyu, a generative AI search engine that breaks complex queries into sub-questions, retrieves from multiple sources, and presents answers with citations, timelines, and images. Human ratings on 300 queries put Xinyu above eight existing products on overall quality, especially comprehensiveness and insightfulness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Superiority claim rests on an unblinded, small-N human evaluation with no inter-annotator agreement or significance testing; the winning dimensions are exactly those Xinyu's presentation modules target, so the 9.235 vs 8.810 advantage may not generalize.","rationale":"The reader's weakest assumption—that the 300-query, expert-rated evaluation with no blinding, no inter-annotator agreement, and no significance testing is a valid and generalizable measure—is indeed the most load-bearing point. The central claim is a comparative superiority statement, and its only direct evidence is Table 2. That table is also where the most concrete threat lives: the dimensions with the largest Xinyu advantage are exactly the ones engineered into the system, making it plausible that the evaluation protocol rewards Xinyu's design choices rather than measuring search quality generically. The paper's other potential weaknesses (no artifacts, small ablations, LLM-judge correlation) are secondary, because even a perfect ablation does not establish superiority over deployed competitors. I agree with the reader's conditional verdict: the engineering contribution is plausible and the evaluation is suggestive, but the headline claim is not yet supported by rigorous evidence. The proposed blinded, plain-text re-evaluation with reliability and significance checks would directly settle whether the advantage is genuine or an artifact of evaluation conditions. Until such a check is run, 'UNCHANGED' (CONDITIONAL) is the appropriate verdict.","tokens_in":21836,"tokens_out":6697,"duration_ms":66121,"concrete_test":"Run a pre-registered, blinded evaluation on a held-out set of 50 queries sampled independently of the original 300. For each query, strip all answers to plain text (remove images, timelines, and markdown), randomize order, and have e.g. 3 independent journalism-trained raters score each system. Compute Krippendorff's alpha for inter-annotator agreement and a paired Wilcoxon signed-rank test for Xinyu vs. the best baseline on the composite average and on Comprehensiveness/Insightfulness. If the advantage is not significant at alpha=0.05 or the inter-annotator agreement is poor (alpha<0.6), the superiority claim should be downgraded to a conditional, presentation-dependent one.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.1 and Table 2 are the sole support for the headline claim that Xinyu outperforms eight existing systems. The evaluation uses 300 Chinese queries (Fig. 4) rated by journalism-trained experts, but the paper reports no inter-annotator agreement, no variance or confidence intervals, no pairwise significance tests, and no evidence of blinding. The two dimensions where Xinyu's advantage is largest—Comprehensiveness (9.143 vs 8.252) and Insightfulness (7.333 vs 6.796)—are precisely the dimensions addressed by Xinyu's novel presentation modules (timeline visualization, QDG, textual-visual choreography). The test-query domain distribution is heavily weighted toward news/history/military (≈36%), where timeline-heavy, structured answers are most useful. If raters could identify Xinyu's answers by their distinctive formatted presentation, or if the criteria systematically reward such presentation, the comparative claim is confounded. The paper also asserts 'significantly outperforms' without any statistical test. The load-bearing assumption is therefore that this convenience sample and unblinded rating protocol yield an unbiased, generalizable measure; nothing in the paper supports that assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Xinyu AI Search, a generative AI search engine that combines a query-decomposition graph (QDG), multi-source retrieval with query expansion, passage deduplication/selection/re-ranking, and three presentation components: fine-grained built-in citations, timeline visualization, and textual-visual choreography. The authors report a full-stack system description with prompts, fine-tuning data sizes, and ablations. The central empirical claim is that, on 300 recent Chinese queries rated by journalism-trained experts, Xinyu outperforms eight existing generative AI search engines and conversational LLM+RAG systems, with the largest advantages in comprehensiveness and insightfulness. The paper also reports component-level comparisons for citations, timelines, and image placement, and ablation studies for most sub-modules.","tokens_in":22152,"tokens_out":4209,"duration_ms":41777,"significance":"If the performance claims were rigorously established, this would be a valuable full-stack reference for generative AI search engine design. The paper is unusually transparent: it discloses concrete prompts (App. B), data-construction procedures and dataset sizes (App. D), content-filtering rules (App. C), and threshold choices. The ablation studies, while mostly LLM-evaluated, provide a reasonable first-order check of the contribution of individual modules. However, the headline comparative claim rests on a single human evaluation that lacks statistical testing, inter-annotator agreement, and blinding, and the winning dimensions coincide with the modules whose presentation style is most distinctive. The evidence as it stands is suggestive but not sufficient to establish that Xinyu 'outperforms' eight systems in the strong sense claimed.","major_comments":[{"comment":"The paper states that Xinyu 'significantly outperforms other methods' in comprehensiveness (9.143 vs. 8.252) and insightfulness (7.333 vs. 6.796), but no statistical tests, confidence intervals, or variance measures are reported. On two metrics, Conciseness (9.813 vs. 9.851) and Coherence (9.810 vs. 9.853), Xinyu scores below Perplexity AI, so the pattern is not uniform even in the aggregate. Without per-query scores, error bars, or pairwise significance tests, the observed differences cannot be distinguished from rating noise, and the central superiority claim is not established.","section":"Sec. 4.1.1, Table 2"},{"comment":"The human evaluation reports no measure of inter-annotator agreement. The paper does not state how many expert raters scored each answer, whether ratings were collected independently, or any agreement coefficient (e.g., Cohen's kappa or Krippendorff's alpha). For subjective criteria such as Insightfulness, unreported reliability leaves the numerical scores of unknown quality; this is load-bearing because the entire comparative claim derives from these ratings.","section":"Sec. 4.1.1"},{"comment":"No blinding procedure is described. Xinyu's answers are visually distinctive due to timeline panels and embedded images, so expert raters could readily identify which system produced which answer. The two dimensions where Xinyu's advantage is largest, Comprehensiveness and Insightfulness, are precisely the dimensions addressed by the timeline and textual-visual modules, so the comparison risks measuring presentation style rather than answer content. The authors should describe how they controlled for this (e.g., rendering all outputs in plain text or using a blinded crowd protocol).","section":"Sec. 4.1.1 and Fig. 4"},{"comment":"The word 'significantly' appears repeatedly (Tables 3, 4, and 5) without any accompanying statistical test, and Table 1 reports Pearson correlations between human and LLM scores that appear to be computed on the aggregate mean values of Tables 2 and 15 rather than on per-item pairs. This overstates the agreement between the two evaluation methods and is not an adequate substitute for per-answer validation. The authors should either provide proper statistical support or soften the 'outperforms' and 'significantly' claims to descriptive observations.","section":"Sec. 4.1.1, 4.1.2, and Table 1"}],"minor_comments":[{"comment":"The figure label 'Gropu2 Title' contains a typo; it should read 'Group2 Title'.","section":"Fig. 3"},{"comment":"Several thresholds are stated without sensitivity analysis: the chunk size of 350 with 25% overlap, the passage retention ratio of 70% (Sec. 3.4.2), the citation fallback cosine threshold of 0.6, and the image-relevance threshold of 0.3. Since these choices are referenced as design decisions, a brief robustness check would increase confidence in the pipeline's reported performance.","section":"Sec. 3.3.2 and Sec. 3.5.1"},{"comment":"The evaluation is conducted exclusively on Chinese queries, while the paper also claims to have an English version. No English-language evaluation is reported, so the external validity of the 'outperforms eight existing technologies' claim is limited to one language and one test-query distribution.","section":"Sec. 4.1.1"},{"comment":"The evaluation protocol for the fine-tuned reranker is unclear: the metrics Precision, Recall, and F1 are reported without a definition of the underlying task (e.g., relevant-passage ranking at a fixed cutoff, or binary classification). A precise task definition would make this ablation interpretable.","section":"Table 14 (App. E)"},{"comment":"The claim of being 'the first comprehensive framework for generative AI search engines' is strong and should be supported with a more explicit comparison to prior system descriptions and open-source projects, or softened to 'one of the first'.","section":"Abstract and Sec. 1.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript offers a commendably detailed disclosure of a deployed generative search system, which is likely to be useful to the community. However, the evaluative evidence for the central claim is thin: no significance tests, no inter-annotator agreement, no blinding, and several unsupported uses of 'significantly.' These are fixable in revision, but without them the numerical comparisons in Tables 2-5 fall short of the standard expected for a peer-reviewed claim of outperformance. I recommend major revision rather than rejection because the system description itself is valuable and the experiments could be made rigorous with reasonable additional effort."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as an engineering report, not a scientific claim. The Xinyu system is described in unusual detail: query decomposition graph, multi-source retrieval with dedup/selection/reranking, entity-based built-in citation decoupled from generation, timeline visualization, and text-image placement via CLIP plus Hungarian matching. They give prompt templates, data sizes for fine-tuning, thresholds, and ablation results for each module. That level of disclosure is rare for a commercial system and should be credited. The abstract's 'first comprehensive framework' is marketing; the contribution is the disclosed pipeline, not the title.\n\nThe weak part is the evaluation. Table 2's claim that Xinyu 'significantly outperforms' eight systems comes from 300 Chinese queries rated by journalism-trained experts, with no inter-annotator agreement, no error bars or significance tests, and no evidence of blinding. The largest gaps—Comprehensiveness 9.143 vs 8.252 and Insightfulness 7.333 vs 6.796—are exactly the dimensions that Xinyu's presentation modules emphasize, and the query set skews toward news/history/military where timelines matter. The rater bias risk is real and unrebutted. The word 'significantly' is doing no work; there is no statistical test.\n\nThat said, the paper is not misleading in every part. Ablations in Table 6 and Table 7 are based on LLM evaluations (GPT-4O) with correlation coefficients reported; the correlations for Insightfulness are lower (0.610), so those results are less certain, but the method is disclosed. No code or data is released, which limits reproducibility, but the textual detail is enough to inform a replication attempt.\n\nMy bottom line: this is a useful reference for anyone building a generative search engine, and the system description is worth peer review. The superiority claim should be treated as provisional until the eval is hardened—at minimum add significance testing, blinding, and a query distribution that isn't dominated by timeline-friendly topics.","headline":"A detailed engineering blueprint for a generative search engine; the system description is the real contribution, and the superiority claim is not statistically supported.","tokens_in":22695,"tokens_out":3436,"would_cite":false,"duration_ms":32066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Xinyu AI Search outperforms eight rival generative search technologies in human evaluations, with the highest scores for comprehensiveness and insightfulness.","keywords":["generative AI search engine","retrieval-augmented generation","query-decomposition graph","built-in citation","timeline visualization","textual-visual choreography","multi-source retrieval","human evaluation of search answers"],"falsifier":"Re-run the nine-system comparison on a new query set (for instance, 300 English or non-news queries) with raters who cannot see which system produced each answer, and report per-rater agreement; the claim would be refuted if the average gap in comprehensiveness and insightfulness disappears once answer length, timeline structure, images, and citation density are statistically controlled, or once ratings are blinded.","tokens_in":21671,"feed_emoji":"🔍","tokens_out":13461,"duration_ms":109882,"temperature":0.7,"pith_summary":"Generative AI search engines answer questions with synthesized text, but their answers lose relevance on complex queries, omit key facts, and present information in ways that are hard to trace. This paper presents Xinyu, a generative search engine that decomposes each complex query into a graph of dependent sub-queries, retrieves and reranks passages for every sub-query, and then composes a final answer supplemented with sentence-level citations, a timeline of events, and images matched to individual paragraphs. The paper claims that on 300 real-world Chinese queries rated by journalism-trained experts, Xinyu beats eight rival systems, achieving the highest overall score (9.235) and the largest advantages in comprehensiveness (9.143) and insightfulness (7.333). The paper also positions itself as the first full-stack disclosure of a deployed generative search engine, which matters because the winning design is an orchestration of fine-tuned open-source models rather than a single proprietary model, making the approach reproducible.","feed_headline":"Outscores eight rivals in human search-quality tests","feed_subtitle":"It splits queries into sub-questions and adds timelines, images, and sentence-level citations.","key_machinery":"The load-bearing object is the query-decomposition graph (QDG), a directed graph in which nodes are sub-queries and edges are dependencies; a fine-tuned LLM builds it by choosing among chain decomposition (sequential, where parent answers inform children), split decomposition (independent parallel sub-queries), or terminal (no decomposition needed). The QDG fixes the execution order of the whole pipeline: retrieval happens per sub-query, answers are generated parents-first with ancestor Q&A inserted as context, and the final response is assembled from all sub-answers. Around this core, three named presentation mechanisms carry the user-facing claims: the built-in citation scheme (a small model extracts entities sentence by sentence, a second small model attaches the best source document, with an embedding-similarity fallback), timeline visualization (event extraction with timestamps, embedding-based deduplication, grouping, and chronological sorting), and textual-visual choreography (image filtering, a three-way paragraph-image relevance score, and Hungarian-algorithm assignment of images to paragraphs).","core_discovery":"The central claim is that a search engine built around a query-decomposition graph resolves the three weaknesses of current generative AI search: insufficient relevance, incomplete coverage, and poor answer presentation. Xinyu breaks a complex query into interdependent sub-queries, retrieves and filters passages for each, generates sub-answers in dependency order so that child questions build on parent answers, and only then assembles the final response. On top of this pipeline it attaches citations at sentence granularity by extracting entities and matching them to retrieved documents, builds a timestamped event timeline from the retrieved passages, and places images next to the paragraphs they illustrate using a learned similarity score. Evaluated by journalism-trained raters on 300 Chinese queries spanning eight domains, Xinyu scores highest overall (9.235 versus 8.810 for the closest rival) and highest on comprehensiveness (9.143) and insightfulness (7.333). Ablation studies attribute the gains to the query-decomposition graph, passage reranking, and the presentation modules, and the authors report that removing any single module does not always improve every metric, with the integrated system winning overall.","pith_inferences":["The evaluation that carries the headline comparison — 300 Chinese queries, journalism-trained raters, no reported inter-annotator agreement, no variance or significance testing, and no evidence of blinding — leaves open the possibility that the 0.425-point average gap reflects preference for long, structured, presentation-rich answers rather than substantive superiority; a blinded cross-language r","The query-decomposition graph is a transferable device: any generation task that benefits from splitting a question into dependent sub-questions, such as legal analysis, medical triage, or competitive intelligence, could reuse the same graph construction and dependency-ordered generation.","The sentence-level citation scheme doubles as an audit mechanism: because every sentence must name a supporting document, the system makes each claim individually checkable, which is a stronger hallucination-containment property than paragraph-level citation and could be reused as a verification tool in other RAG systems.","The deduplication and citation thresholds (0.8 passage similarity, 0.9 timeline merge, 0.6 citation fallback) are set without sensitivity analysis, so a test that varies these constants would show how much of the presentation gain depends on their exact values."],"forward_implications":["A competitive generative search engine can be assembled from openly available models — Qwen2.5 variants of 3B to 72B parameters, bge-large-zh embeddings, and bge-reranker-v2-m3 — fine-tuned on datasets of 5K to 37K examples per task, rather than requiring a proprietary foundation model.","The presentation modules are not cosmetic: removing built-in citation, timeline visualization, or textual-visual choreography each measurably lowers comprehensiveness ratings, and removing images also lowers clarity.","Because the QDG records dependencies between sub-queries and generation follows parents-before-children order, the same pipeline absorbs more complex questions by adding nodes, without redesigning retrieval or generation.","The ablation results show the modules interact: omitting query expansion can actually improve relevance on some queries, yet the full system still wins overall, so the value of each component is in combination rather than in isolation.","A deployed English version, produced simply by translating the Chinese prompts, reportedly works 'surprisingly well' with the same multilingual models, suggesting the orchestration transfers across languages before any language-specific fine-tuning."],"supporting_citations":[{"why":"Retrieval-augmented generation (Lewis et al.), the paradigm the whole system extends, supplying the retrieval-plus-generation loop.","marker":"[37]"},{"why":"Least-to-most prompting (Zhou et al.), the strategy the query-decomposition graph's chain decomposition builds on.","marker":"[81]"},{"why":"Lost in the middle (Liu et al.), the finding that motivates passage re-ranking so key evidence sits at prompt edges.","marker":"[41]"},{"why":"Perplexity AI, the leading baseline in the nine-system comparison and the system whose paragraph-level citation Xinyu contrasts with.","marker":"[5]"},{"why":"Metaso, the baseline for the textual-visual choreography comparison.","marker":"[49]"},{"why":"CHRONOS (Wu et al.), the timeline-generation system Xinyu is compared against for the timeline visualization.","marker":"[71]"},{"why":"GPT-4O, the model used as Perplexity's backend and as the automated evaluator in the LLM-based comparisons.","marker":"[53]"},{"why":"bge-large-zh (Xiao et al.), the embedding model used for passage deduplication, citation fallback, and paragraph-image matching.","marker":"[72]"},{"why":"BGE M3 / bge-reranker-v2-m3 (Chen et al., Li et al.), the base reranker fine-tuned for passage ranking and image relevance scoring.","marker":"[17, 38]"},{"why":"The survey the paper cites to justify using an LLM (GPT-4O) as an automated judge in place of expert ratings.","marker":"[27]"}],"fun_headline_variants":["Breaks queries into sub-questions to outscore eight AI search rivals","Query-splitting search engine beats eight rivals in human tests","New AI search uses query decomposition to top eight alternatives","Splits complex queries, adds visuals, and outperforms eight search AIs","Generative search with decomposed queries wins over eight competitors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 300 Chinese queries rated by journalism-trained experts, without reported inter-annotator agreement, variance, significance testing, or evidence of blinding, measure search answer quality well enough to support the headline outperformance claim; if those raters systematically preferred long, structured, presentation-rich answers, the result would not generalize to other query distributions or languages.","fun_headline_variants_meta":{"raw":{"variants":["Breaks queries into sub-questions to outscore eight AI search rivals","Query-splitting search engine beats eight rivals in human tests","New AI search uses query decomposition to top eight alternatives","Splits complex queries, adds visuals, and outperforms eight search AIs","Generative search with decomposed queries wins over eight competitors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2419,"prompt_tokens":934,"completion_tokens":1485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1397}},"tokens_in":550,"tokens_out":1485,"duration_ms":10692,"temperature":1.0,"reasoning_tokens":1397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:21:15.199565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the nine-system comparison on a new query set (for instance, 300 English or non-news queries) with raters who cannot see which system produced each answer, and report per-rater agreement; the claim would be refuted if the average gap in comprehensiveness and insightfulness disappears once answer length, timeline structure, images, and citation density are statistically controlled, or once ratings are blinded.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Least-to-most prompting (Zhou et al.), the strategy the query-decomposition graph's chain decomposition builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Metaso, the baseline for the textual-visual choreography comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4O, the model used as Perplexity's backend and as the automated evaluator in the LLM-based comparisons."}],"review_version":1}