Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers

Francesco Pierri; Geng Liu; Li Feng; Mengxiao Zhu

REVIEW 5 major objections 5 minor 40 references

Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Factual reliability in Chinese search and AI depends less on being right when they answer and more on how often, and in which direction, they answer.

desk verdict A useful dataset-and-pipeline contribution, but the ground-truth validation gap is big enough that the headline accuracy numbers are not reliable as reported. read the letter →

arxiv 2602.22221 v2 pith:W2QCH6WI submitted 2025-12-15 cs.IR cs.AIcs.CLcs.CY

Geng Liu , Li Feng , Mengxiao Zhu , Francesco Pierri This is my paper

classification cs.IRcs.AIcs.CLcs.CY

keywords Chinesefact-checkingsearchengineslargelanguagemodelsAIoverviewanswerratepolaritygapBaiduIndexmisinformationexposure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the factual reliability of Chinese information-access systems—search engines, standalone chatbots, and an AI-powered overview—cannot be summarized by a single accuracy number. Using 12,161 real Chinese yes/no questions drawn from actual search logs, it finds that systems are similarly accurate when they do give a definitive answer (roughly 73–79% conditional accuracy in the abstract's framing), but they differ sharply in how often they answer at all: search engines answer over 83% of queries, while some AI models answer fewer than half. The paper also documents a systematic polarity gap, with most systems performing better on questions whose true answer is 'yes' than on those whose answer is 'no,' and topic-level errors vary widely, with health questions dominating the dataset. It then combines system accuracy with Baidu Index search-demand data to argue that potential exposure to wrong health information is concentrated in coastal provinces with high search volume. The sympathetic reader should care because the work reframes reliability as a coverage–accuracy tradeoff with concrete geographic consequences.

What carries the argument

The load-bearing mechanism is the pairing of each real user query with a Level-3 passage from a public Chinese search-log dataset that explicitly contains the answer, which lets the authors convert 12,161 queries into unequivocal yes/no ground-truth questions. On top of this, they build a three-paradigm evaluation pipeline: for search engines, the top-10 results are scraped, classified as yes/no/uncertain by an LLM, and aggregated by majority vote; for chatbots, a zero-shot prompt forces a yes/no; for the AI overview, the generated summary is classified by another LLM. The pipeline's 'definitive answer vs. uncertain/no answer' distinction produces the answer-rate dimension, and the yes/no la

What would settle it

Manually annotate a random sample of, say, 500 of the 12,161 query–passage pairs with independent human yes/no/uncertain labels and compare to the LLM-generated labels; if agreement on the binary labels falls much below the reported ~95% (especially on the 'no' labels), the polarity gap and the 73–79% conditional accuracy range would need revision. Separately, re-run Bing queries through a direct query-forwarding method instead of the self-hosted proxy that the paper says 'occasionally failed to forward complete Chinese queries'—if Bing's reported 5.6% yes-accuracy rises, the extreme polarity

Watch

Extended reading notes

Core claim

The paper's central discovery is that 'how accurate when answering' and 'how often answering' are separate dimensions of reliability, and that systems occupy different points on these dimensions. Conditional accuracy—accuracy on queries where the system returns a definitive yes/no—falls in a narrow band (73.2% to 78.9% in the abstract; 45% to 69.8% depending on subset in the full text), while answer rates vary widely: traditional search engines answer definitively on at least 83% of queries, whereas the Qwen-Max model does so on fewer than half. A second finding is a consistent polarity gap: in the abstract's account, every system performs better on yes-labeled queries than on no-labeled one

Load-bearing premise

The automatically generated yes/no ground-truth labels for the 12,161 queries—created by an LLM and manually checked on only about 100–200 examples—are correct; the paper itself notes that reliance on LLM-assisted labeling in the preprocessing phase may introduce (circular) noise, and if the labels are wrong, every accuracy, answer-rate, and polarity comparison inherits those errors.

Editorial extensions

If this is right

A user comparing systems should weight answer rate as much as accuracy: a system that answers eight of ten correctly but only attempts half the queries may be worse in practice than one that attempts every query at 75% accuracy.
Platform auditors should report reliability as a coverage–accuracy pair rather than a single percentage, and should monitor the yes/no polarity gap because it reveals systematic bias against negative claims.
Health-related queries dominate the dataset and are answered with only moderate accuracy, so AI answers in Chinese medical-search contexts carry a measurable risk of wrong answers in high-stakes situations.
The province-level exposure metric, multiplying local search demand by system error, implies that misinformation mitigation efforts in China should be geographically targeted toward high-demand coastal provinces.
The dataset and pipeline can be reused to benchmark future systems, making it possible to track whether new Chinese search or AI products improve on answer rate and polarity balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the answer-rate gap is real, a natural testable extension is to see whether fine-tuning or prompt design can push LLMs to answer more often without sacrificing conditional accuracy—treating the two dimensions as a coverage–accuracy frontier.
The polarity gap, if it holds up under human labeling, points to a training-data imbalance in which 'no' answers are underrepresented; one could test this by measuring performance on a balanced set of yes/no claims rather than the naturally skewed distribution in search logs.
Because the ground-truth labels were assigned by the same family of models being evaluated, a fully human-annotated version of even a few hundred queries would be a cheap way to check whether the reported asymmetries are real or an artifact of circular labeling.
The geographic exposure result is likely sensitive to population size since raw Baidu Index volumes track population; normalizing by provincial population could reveal whether the coastal concentration is a density effect or a genuine per-capita risk difference.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: this is a genuinely new empirical resource—the first yes/no fact-checking evaluation of Chinese search engines, LLMs, and AI Overviews on real user query logs—and the pipeline is sensible. But the ground truth is built on LLM labeling with a measured precision of roughly 68% for the “factual” label, and the paper still reports every accuracy number against it. That is a load-bearing flaw, not a caveat.

What it does well: the choice of T2Ranking logs gives real information needs, the three-paradigm comparison is well motivated, and the authors are candid about several limitations (Bing retrieval failures, population-size confounds, circular labeling noise). The core observation that systems differ more in how often they answer than in conditional accuracy is plausible and worth pursuing. The consistency of the yes-over-no gap across systems is interesting, though less convincing once you account for potentially systematic label noise.

Where it falls short. First, the manual validation in §3.1.4 shows 24 of 50 non-factual queries were labeled factual. That means the 12,161-query dataset likely contains a large share of opinion or subjective questions, and every figure in Figures 3–9 inherits that noise. The authors only removed queries the LLM flagged as non-factual; they kept the false positives. Their own Section 5 admits circular noise, but they don't quantify or correct for it. Second, the abstract and the body disagree on central numbers: the abstract reports conditional accuracy 73.2–78.9%, search engines answering 83%+ queries, and Qwen-Max under 50%, but the body gives overall accuracy 45–70%, per-engine conditional numbers like Bing at 34.5%, and uses Qwen-Plus, not Qwen-Max. That inconsistency matters for a paper whose main selling point is empirical measurement. Third, the exposure metric R = I_p × A_m is mislabeled: multiplying search demand by accuracy gives a “reliable information” index, not a misinformation exposure index. Higher accuracy should reduce exposure to incorrect answers, not increase it. The geographic maps may still be informative, but the framing needs rework.

These are addressable. Release the data and code, re-validate labels on a larger human sample, fix the abstract/body mismatch, and reframe the exposure index. But as it stands, the central quantitative claims are not trustworthy enough to cite.

My take: this deserves a serious referee—the dataset and question are worth the field's attention—but only with major revision. Send it to peer review, but expect the review to insist on the fixes above.

Referee Report

5 major / 5 minor

Summary. The paper constructs a dataset of 12,161 Chinese Yes/No fact-checking questions from T2Ranking search logs, using DeepSeek-assisted classification and answer labeling with a small manual validation subset. It then compares Baidu, Bing, Sogou, DeepSeek, Qwen-Plus, LLaMA-3-8B, and Baidu AI Overview in a unified evaluation pipeline, reporting accuracy by system, by answer polarity, and by topic. It also combines health-query accuracy with province-level Baidu Index data to compute an exposure index. The headline claims are that systems differ mainly in how often they answer rather than in conditional accuracy, and that there is a consistent yes-over-no polarity gap. The full-text results, however, report only percentage-of-correct-predictions on uneven coverage subsets and show mixed polarity effects. The paper acknowledges limitations involving LLM-assisted labeling and unreliable Bing retrieval.

Significance. The dataset built from real Chinese search logs is a potentially valuable resource, and the cross-paradigm comparison addresses an understudied non-English ecosystem. The authors are transparent about several limitations, and the manual validation subset is a useful start. However, the central empirical claims are not currently supported: the ground-truth construction has an estimated precision of only about 68% for the factual label, the stated yes-better polarity gap is contradicted by the paper's own results for DeepSeek, LLaMA, and Bing, and the exposure metric R = I × A is mis-specified for measuring misinformation exposure. The paper's contribution is therefore important in intent but needs substantial revision before its conclusions can be relied upon.

major comments (5)

[§3.1.4 and §5] The fact-checking filter has a serious false-positive problem. In the 100-query validation, 24 of 50 non-factual queries were labeled as factual, giving an estimated precision of 50/(50+24) ≈ 68% for the factual label. Because only LLM-labeled non-fact-checking queries were removed, these false positives remain in the 12,161-query dataset. Every accuracy and polarity result in Figures 3–9 is computed against ground-truth answers for queries that may not be factual. The Section 5 admission that 'reliance on LLM-assisted labeling... may introduce (circular) noise' is not sufficient; the paper needs a robustness analysis on the manually validated factual subset, or a re-estimation with bounds. This issue is load-bearing for the central accuracy comparisons.
[§4.3, Fig. 6, and Abstract] The claim of a 'consistent polarity gap' with 'all systems performing better on yes-labeled queries' is contradicted by the reported data. DeepSeek is more accurate on No questions (78.3%) than Yes questions (56.6%); LLaMA is also better on No (55.6% vs 40.2%). In §4.2, Bing is overwhelmingly better on No (98.5% vs 5.6%). Only Qwen and AI Overview show the yes-over-no pattern. The abstract and discussion need to be rewritten to describe the actual mixed polarity effects, or the metric needs to be redefined.
[§3.3, Eq. (R = I_p × A_m)] The exposure metric is mis-specified for the stated goal. R_p,m = I_p × A_m, where A_m is factual accuracy, means that higher accuracy produces higher exposure. But the text says the goal is to identify 'greater potential exposure to misinformation' when high user interest 'coincides with lower-accuracy tools.' The correct composite for misinformation exposure should be I_p × (1 - A_m) (or an explicit inverse relationship). As written, Figure 10 and the province-level conclusions do not measure misinformation exposure; they mostly reflect search volume. This is a central error for RQ2.
[Abstract vs. §§3–4] The abstract's central quantitative claims do not appear in the body. The abstract reports answer rates ('search engines answer at least 83%', 'Qwen-Max answers fewer than half') and conditional accuracy ranges (73.2–78.9%), but the results sections report only overall 'percentage of correct predictions' on different coverage subsets (1454 common queries in §4.1; 9212, 5339, and 1455 engine-specific queries in §4.2; 1932 AI Overview responses in §4.4). Neither 'answer rate' nor 'conditional accuracy' is defined in the body, and 'Qwen-Max' does not appear anywhere; the body evaluates 'Qwen' (Qwen-Plus). The paper should either include and clearly define these metrics, or the abstract must be revised to match the actual analysis.
[§5 and §4.2] The Bing results are explicitly acknowledged as unreliable: the self-hosted SearXNG instance 'occasionally failed to forward complete Chinese queries, producing incomplete or mismatched retrieval.' Yet Bing is included in the main cross-system comparison and drives the extreme polarity imbalance (98.5% correct on No vs 5.6% on Yes). This is likely an artifact of retrieval failure rather than a property of the Bing engine. The paper should either exclude the Bing data, re-collect it with a reliable retrieval pipeline, or report all cross-system results with and without Bing as a sensitivity check.

minor comments (5)

[§3.2.3] There is a placeholder 'Appendix??' that should be replaced with the correct appendix reference.
[Throughout] Terminology is inconsistent: 'percentage of correct predictions', 'accuracy rate', and 'conditional accuracy' are used interchangeably. Define one term and use it throughout.
[Abstract and §1] The paper says 'nine systems' in the abstract but the body lists seven systems (Baidu, Bing, Sogou; DeepSeek, Qwen, LLaMA; Baidu AI Overview). This discrepancy should be corrected.
[Appendix C] The paired t-test reports p = 0.066 and states there is 'no statistically significant difference.' This is a failure to reject the null, not evidence of equivalence. Report an agreement metric or a confidence interval for the difference instead.
[§3.3] The text says 'province-level time series for all 34' without specifying 34 what; it should say '34 provincial-level administrative regions.'

Circularity Check

3 steps flagged · score 5.0 of 10

Ground truth is partly authored by DeepSeek, which is also an evaluated system; the fact-checking filter further retains a large share of non-factual queries, so several headline comparisons reduce to self-agreement.

self definitional [§3.1 Data Collection; §3.2.3 Large Language Models]
"In particular, we employed DeepSeek 1, to assist with Yes/No identification, fact-checking classification, and answer labeling during preprocessing. ... To evaluate the fact-checking capabilities of Chinese-based LLMs, we employed a zero-shot prompting approach using the deepseek-v3[ 8], Qwen-Plus 7 and Llama-3-8B-Instruct[ 14] models."

DeepSeek is both the author of the ground-truth labels for the 12,161 queries (answer labeling) and one of the systems scored against those labels. Its reported accuracy therefore measures self-agreement between its labeling pass and its zero-shot answering pass, not an externally grounded factuality. The manual validation of only 100 answer labels does not remove the dependence for the remaining ~12,000 queries, so the DeepSeek accuracy figures and the yes/no polarity gap are partly constructed by the labeler.
self definitional [§3.2.4 Baidu AI Overview]
"Because Baidu AI Overview responses often contain explanatory text rather than explicit binary answers, we use the DeepSeek model to infer the overall answer."

The same model that generated the ground-truth yes/no labels is used to convert AI Overview summaries into yes/no decisions. Comparing those decisions to the DeepSeek-derived ground truth makes the AI Overview result a DeepSeek-to-DeepSeek agreement measure rather than an independent factual evaluation, especially for the reported yes/no accuracy asymmetry.

1 more flagged steps

other [§3.1.4 Manual Evaluation of LLM Annotations for Pre-processing]
"all factual labels matched the LLM output, but 24 of the non-factual queries were incorrectly labeled as factual. Given this substantial error rate, all LLM-labeled non–fact-checking queries were removed from subsequent analyses."

The validation shows that the 'factual' filter has an estimated precision of only ~68%, yet the authors removed only queries the LLM labeled as non-fact-checking, retaining all false-positive non-factual queries. For those retained non-factual queries there is no objective yes/no truth; the 'correct' answer is whatever DeepSeek assigned. Thus the 12,161-query 'verified fact-checking' set, and every accuracy and polarity figure computed on it, is partially defined by the labeler's own output.

full rationale

The most serious circularity is a construction chain rather than a citation chain: DeepSeek supplies the yes/no ground-truth labels and is then evaluated against them, and DeepSeek also converts Baidu AI Overview outputs into the labels that are compared with that same ground truth. The paper's own Limitations section concedes that 'reliance on LLM-assisted labeling in the preprocessing phase may introduce (circular) noise.' Manual validation attenuates but does not break the circle: only 100 queries were checked for answer labeling, while roughly 12,000 labels come from DeepSeek. Additionally, the fact-checking validation found that 24 of 50 non-factual queries were retained as factual; because only predicted non-fact-checking queries were dropped, the final set likely contains a large fraction (estimated ~32% of predicted-factual queries) of non-factual questions whose yes/no labels are not independently defined. This contaminates every accuracy figure and the headline yes-over-no polarity gap. The use of Qwen as the passage annotator for search engines is a separate annotator and is not circular by itself; the self-citations (e.g., [12]) are not load-bearing. Overall, the empirical comparisons are not fully reduced to a single fitted parameter, and the T2Ranking passages and partial human annotation provide some independent content, so the circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its load-bearing assumptions are all methodological: LLM-generated ground truth, LLM-judged passage stances, binary reduction of real queries, and a simple product as an exposure index. These choices are acknowledged in the limitations section, which explicitly flags circular noise, uneven coverage, and population-size confounds.

free parameters (4)

DeepSeek-assisted ground-truth labels (yes/no) = 12,161 labels
The labels are generated by prompting DeepSeek on T2Ranking passages and are the target values every accuracy number is compared against; only ~300 samples were manually validated, with a known 24/50 error rate in one filtering step.
Qwen-Plus passage classifier threshold/vote = majority vote over top-10 results
Search-engine accuracy is computed by a Qwen-Plus yes/no classification of each retrieved page aggregated by simple majority vote; the choice of classifier and vote rule is a free methodological parameter that can move engine accuracies.
Baidu Index keyword sets and 5-year window = 2020-09-19 to 2025-09-19, representative keywords per topic
The provincial exposure metric depends on which keywords were chosen for each topic and the selected multi-year window; these are researcher choices that determine the health attention I_p values.
Exposure metric R = I_p × A_m = unitless product
The simple product of average Baidu Index and a single health-accuracy number is an ad hoc composite; no validation is provided that it measures actual misinformation exposure rather than population size.

assumptions (6)

domain assumption T2Ranking Level-3 passages contain the correct yes/no answer for the query
Section 3.1 restricts the dataset to queries whose associated passages are labeled Level 3 and assumes those passages objectively resolve the query; if the relevance label or the passage content is wrong, ground truth is wrong.
domain assumption A binary yes/no framing captures factual reliability
The entire evaluation reduces complex factual queries to yes/no propositions, excluding multi-hop, time-sensitive, or partially answerable questions, which the paper itself lists as future work in Section 5.
ad hoc to paper DeepSeek's filtering and labeling behavior is reliable enough for ground-truth construction
DeepSeek is used for query classification, fact-checking classification, and answer labeling; the manual check found a substantial error on non-factual queries (24/50) which the authors patch by deleting LLM-labeled non-fact-checking queries, but the whole pipeline is built on this model's outputs.
domain assumption Qwen-Plus's reading of retrieved pages is a valid oracle for what a search engine answers
Section 3.2.1: the engine's stance is determined by Qwen-Plus labels on each returned page; this equates 'the search engine answered' with 'Qwen-Plus says the page answers', ignoring ranking position, snippet text, and user attention.
domain assumption Baidu Index volumes reflect true health information demand rather than population or confounds
Section 5 concedes 'Baidu Index values may reflect population size as well as information demand'; the exposure map is therefore largely a map of population/search volume.
ad hoc to paper AI Overview text can be reduced to a yes/no stance by DeepSeek
Section 3.2.4 uses DeepSeek to classify each AI Overview summary as yes/no; the paper itself shows a failure case (Appendix D.3) where a conditional answer is forced into yes/no, so this reduction loses information and can create errors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers." pith.science (2026). https://pith.science/paper/W2QCH6WI

@misc{pith2026260222221,
  author       = {Pith},
  title        = {Pith review of: Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W2QCH6WI}},
  note         = {Machine review of arXiv:2602.22221}
}

read the original abstract

Search engines and AI-powered systems increasingly mediate access to factual information, yet their reliability remains difficult to evaluate in realistic information-seeking settings. We study this problem in the Chinese web ecosystem by constructing a query-based fact-checking dataset from real Chinese search logs and comparing nine systems across traditional search engines, standalone large language models, and search-integrated AI Overviews. Focusing on factual Chinese-language factual Yes/No questions, we evaluate whether systems provide correct, incorrect, or uncertain decisions against evidence-derived ground truth. We find that systems are similarly accurate when they provide definitive answers, but differ sharply in how often they do so. Conditional accuracy ranges from 73.2% to 78.9%, yet search engines answer definitively on over 83% of queries, while Qwen-Max does so on fewer than half. We also find a consistent polarity gap: all systems perform better on yes-labeled queries than on no-labeled queries. We also use Baidu Index data to identify Chinese provinces with higher health-related search attention, which may indicate greater potential exposure to misinformation. Overall, our results show that reliability depends not only on whether systems are correct when they answer, but also on how often they answer, how they handle negative claims, and where information demand may increase exposure risks.

Figures

Figures reproduced from arXiv: 2602.22221 by the authors.

**Figure 2.** Distribution of collected fact-checking queries [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Percentage of correct predictions across search en [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 5.** Figure 5: Topic-level percentage of correct predictions for [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 8.** Figure 8: The percentage of correct predictions for AI [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 7.** Figure 7: Topic-level percentage of correct predictions for [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 10.** Figure 10: Provincial exposure index for Baidu AI Overview. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

**Figure 11.** Figure 11: Province-level exposure index computed using the [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

**Figure 12.** Figure 12: Comparison of topic-level percentage of correct [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 linked inside Pith

[1]

Ruben L Bach and Alexander Wenz. 2020. Studying health-related internet and mobile device use using web logs and smartphone records.PloS one15, 6 (2020), e0234663

2020
[2]

Marko Bachl, Elena Link, Frank Mangold, and Sebastian Stier. 2024. Search engine use for health-related purposes: Behavioral data on online health information- seeking in Germany.Health Communication39, 8 (2024), 1651–1664

2024
[3]

Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(Virtual Event, Canada)(FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. doi:10.114...

arXiv 2021
[4]

Hannah C Cai, Leanne E King, and Johanna T Dwyer. 2021. Using the Google™ search engine for health information: is there a problem? Case study: supplements for cancer.Current Developments in Nutrition5, 2 (2021), nzab002

2021
[5]

Shui-Lung Chuang and Lee-Feng Chien. 2003. Enriching web taxonomies through subject categorization of query terms from search engine logs.Decision Support Systems35, 1 (2003), 113–127

2003
[6]

Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021. The echo chamber effect on social media.Proceedings of the national academy of sciences118, 9 (2021), e2023301118

2021
[7]

Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. arXiv:2304.08177 [cs.CL] https://arxiv.org/abs/ 2304.08177

arXiv 2024
[8]

Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

arXiv 2025

Show all 40 references

[9]

Matthew R DeVerna, Kai-Cheng Yang, Harry Yaojun Yan, and Filippo Menczer
[10]

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. InProceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. ...

2024
[11]

Marcos Fernández-Pichel, Juan C Pichel, and David E Losada. 2025. Evaluating search engines and large language models for answering health questions.npj Digital Medicine8, 1 (2025), 153

2025
[12]

Nicolo’ Fontana, Francesco Corso, Enrico Zuccolotto, and Francesco Pierri. 2025. Evaluating open-source Large Language Models for automated fact-checking. arXiv preprint arXiv:2503.05565(2025)

2025 arXiv
[13]

Xue Gong, Yangyang Han, Mengchi Hou, and Rui Guo. 2020. Online public attention during the early days of the COVID-19 pandemic: Infoveillance study based on Baidu index.JMIR public health and surveillance6, 4 (2020), e23098

2020
[14]

Aaron Grattafiori et al . 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

2024 arXiv
[15]

Anat Hashavit, Tamar Stern, Hongning Wang, and Sarit Kraus. 2024. The Impact of Snippet Reliability on Misinformation in Online Health Search.arXiv preprint arXiv:2401.15720(2024)

2024 arXiv
[16]

Jansen and Danielle Booth

Bernard J. Jansen and Danielle Booth. 2010. Classifying web queries by topic and user intent. InCHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA)(CHI EA ’10). Association for Computing Machinery, New York, NY, USA, 4285–4290. doi:10.1145/...

2010
[17]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation.ACM Comput. Surv.55, 12, Article 248 (March 2023), 38 pages. doi:10.1145/3571730

2023 doi
[18]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

2023
[19]

David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Penny- cook, David Rothschild, et al. 2018. The science of fake news.Science359, 6380 (2018), 1094–1096

2018
[20]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicen...

2022 doi
[21]

Guangxiang Leon Liu, Ron Darvin, and Chaojun Ma. 2025. Exploring AI-mediated informal digital learning of English (AI-IDLE): A mixed-method investigation of Chinese EFL learners’ AI adoption and experiences.Computer Assisted Language Learning38, 7 (2025), 1632–1660

2025
[22]

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On Faithfulness and Factuality in Abstractive Summarization. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Te...

2020 doi
[23]

Preslav Nakov, Giovanni Da San Martino, Alberto Barrón-Cedeño, Tamer Elsayed, Shaden Shaar, Firoj Alam, Francesco Haouari, Motaz Hasanain, Wesam Mansour, Kairit Takahashi, Arijit Mitra, Georgi Karadzhov, Alex Nikolov, Shammur Absar Chowdhury, Hamdy Mubarak, Wajdi Zaghouani, an...

2021
[24]

Houston C Nelson, Morgan T Beauchamp, April A Pace, and Houston Nelson
[25]

Shaurya Rohatgi, C Lee Giles, and Jian Wu. 2021. What were people searching for? A query log analysis of an academic search engine. In2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE, 342–343

2021
[26]

The Reliability Gap: How Traditional Search Engines Outperform Artificial Intelligence (AI) Chatbots in Rosacea Public Health Information Quality.Cureus Conference’17, July 2017, Washington, DC, USA Geng Liu, Junjie Mu, Li Feng, Mengxiao Zhu, and Francesco Pierri 17, 6 (2025)

2017
[27]

Xin Sun, Rongjun Ma, Xiaochang Zhao, Zhuying Li, Janne Lindqvist, Abdallah El Ali, and Jos A Bosch. 2024. Trusting the search: unraveling human trust in health information from Google and ChatGPT.arXiv preprint arXiv:2403.09987(2024)

2024 arXiv
[28]

Zacchary Sadeddine, Winston Maxwell, Gaël Varoquaux, and Fabian M Suchanek
[29]

InACM SIGIR Forum, Vol

Large Language Models as Search Engines: Societal Challenges. InACM SIGIR Forum, Vol. 59. ACM New York, NY, USA, 1–35
[30]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal
[31]

Qinyi Tan, Fang He, and Luyan Teng. 2022. Using Baidu Index to Understand the Public Concern of Children’s Mental Health in Mainland China in the Context of COVID-19 Epidemic.Journal of Education, Health and Sport12, 10 (2022), 189–198

2022
[32]

Deependra K Thapa, Denis C Visentin, Rachel Kornhaber, Sancia West, and Michelle Cleary. 2021. The influence of online health information on health decisions: a systematic review.Patient education and counseling104, 4 (2021), 770–784

2021
[33]

Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, et al . 2023. T2ranking: A large-scale chinese benchmark for passage ranking. InProceedings of the 46th International ACM SIGIR Conference on Research and ...

2023
[34]

Ziyi Ye, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. Investigating COVID-19-Related query logs of Chinese search engine users.Proceedings of the Association for Information Science and Technology57, 1 (2020), e424

2020
[35]

Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online.science359, 6380 (2018), 1146–1151

2018
[36]

Zhicheng Wang, Hong Xiao, Leesa Lin, Kun Tang, and Joseph M Unger. 2022. Geographic social inequalities in information-seeking response to the COVID-19 pandemic in China: longitudinal analysis of Baidu Index.Scientific Reports12, 1 (2022), 12243

2022
[39]

Yukun Zheng, Zhen Fan, Yiqun Liu, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. Sogou-QCL: A New Dataset with Click Relevance Label. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval(Ann Arbor, MI, USA)(SIGIR ’18). ACM, New Yor...

2018
[40]

是/否问题”。定义：是 /否问题是一类可以用简单的 “是

Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korhonen, D...

2019
[2018]

FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Marilyn Walker, Heng Ji, and Amanda Ste...

2018 doi
[2025]

Large Language Models Require Curated Context for Reliable Political Fact- Checking–Even with Reasoning and Web Search.arXiv preprint arXiv:2511.18749 (2025)

2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Ruben L Bach and Alexander Wenz. 2020. Studying health-related internet and mobile device use using web logs and smartphone records.PloS one15, 6 (2020), e0234663

2020

[2] [2]

Marko Bachl, Elena Link, Frank Mangold, and Sebastian Stier. 2024. Search engine use for health-related purposes: Behavioral data on online health information- seeking in Germany.Health Communication39, 8 (2024), 1651–1664

2024

[3] [3]

Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(Virtual Event, Canada)(FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. doi:10.114...

arXiv 2021

[4] [4]

Hannah C Cai, Leanne E King, and Johanna T Dwyer. 2021. Using the Google™ search engine for health information: is there a problem? Case study: supplements for cancer.Current Developments in Nutrition5, 2 (2021), nzab002

2021

[5] [5]

Shui-Lung Chuang and Lee-Feng Chien. 2003. Enriching web taxonomies through subject categorization of query terms from search engine logs.Decision Support Systems35, 1 (2003), 113–127

2003

[6] [6]

Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021. The echo chamber effect on social media.Proceedings of the national academy of sciences118, 9 (2021), e2023301118

2021

[7] [7]

Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. arXiv:2304.08177 [cs.CL] https://arxiv.org/abs/ 2304.08177

arXiv 2024

[8] [8]

Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

arXiv 2025