REVIEW 3 major objections 5 minor 8 references
AI Answer Engine Citation Behavior An Empirical Analysis of the GEO16 Framework
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A 16-pillar page-quality score predicts whether AI answer engines cite a page, with a 4.2 odds ratio per standard deviation and a 78% citation rate at the G≥0.70, ≥12-hit operating point.
desk verdict Useful descriptive audit of cited pages, but the headline odds ratio and thresholds rest on data with no non-cited control set, so the central inference isn't identified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The GEO-16 framework is the central mechanism. It converts page features into 16 pillar scores (0–3 bands), a pillar hit indicator (band≥2), a hit count H, and a normalized overall score G = (sum of bands)/48. The pillars operationalize six principles: people-first answers, structured data, provenance, freshness, risk controls, and RAG fit. The framework provides both interpretable per-pillar diagnostics and a single composite threshold that carries the paper's predictive claims.
What would settle it
A randomized experiment or natural experiment where high-GEO pages are deliberately altered (e.g., schema removed, dates removed) and citation rates measured would directly test causality. Alternatively, a regression that includes domain-level fixed effects or backlink proxies and still finds no GEO coefficient would falsify the claim.
Extended reading notes
Core claim
On its own terms, the paper establishes that a composite on-page quality measure, GEO-16, is strongly associated with citation likelihood in AI answer engines. The central claim is that pages meeting a simple threshold—G≥0.70 and at least 12 of 16 pillar hits—are cited at 78% across Brave Summary, Google AI Overviews, and Perplexity, versus much lower rates below the threshold. Logistic regression with domain-clustered standard errors yields an odds ratio of 4.2 for G per standard deviation increase. Pillar-level correlations single out Metadata & Freshness (r=0.68), Semantic HTML (r=0.65), and Structured Data (r=0.63) as the strongest on-page signals. These findings are presented as the fir
Load-bearing premise
The paper assumes that the fully rendered page content at fetch time is what the AI engines see, so that no unmeasured off-page signals (like domain authority or brand reputation) are driving both the GEO score and citation decisions.
Editorial extensions
If this is right
- Publishers can use G≥0.70 and ≥12 pillar hits as a concrete benchmark for AI citation likelihood.
- Prioritizing Metadata & Freshness, Semantic HTML, and Structured Data should yield the largest gains.
- Engine-specific differences imply that a single optimization strategy may not suffice; Perplexity's lower G among cited pages suggests it may rely on different signals or lower-quality sources.
- Cross-engine citations (pages cited by more than one engine) have ~71% higher quality scores, suggesting quality begets multi-engine visibility.
Reading between the lines
- If the GEO-16 score is a causal driver rather than a correlate, then schema and freshness interventions could be tested experimentally (e.g., before/after schema additions) to confirm.
- The strong engine differences hint that Perplexity may incorporate non-DOM signals (domain authority, reputation) or treat structured data differently; the framework's DOM-only assumption might understate Perplexity's reliance on off-page factors.
- The threshold result suggests a possible 'quality cliff' rather than linear returns; a randomized study varying G within the threshold region could localize the inflection point.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces GEO-16, a 16-pillar scoring framework for on-page quality, and applies it to 1,702 citations collected from three AI answer engines (Brave Summary, Google AI Overviews, Perplexity) using 70 B2B SaaS prompts. The authors audit 1,100 unique cited URLs, compute a normalized GEO score G in [0,1], and report that cited pages differ across engines in mean GEO score, that pillars such as Metadata & Freshness, Semantic HTML, and Structured Data correlate strongly with citation, and that a logistic model yields an odds ratio of 4.2 per standard deviation of G. They propose an operating point of G ≥ 0.70 with at least 12 pillar hits, claiming a 78% cross-engine citation rate, and translate these findings into publisher recommendations. The study is explicitly observational and discusses limitations such as unobserved confounding and restricted generalizability.
Significance. If the central claim were valid, the paper would provide a practical, actionable bridge between on-page quality auditing and citation behavior in generative AI engines, a topic of substantial current interest. Strengths include the multi-engine design, the relatively large corpus of citations and audited URLs, the detailed pillar structure, and the stated intention to release an anonymized artifact with reproduction scripts. However, the paper's core inferential claims—the odds ratio, sensitivity/specificity, and the operating point—rest on a binary outcome that is not identifiable from the described data because no non-cited control set is collected. This is not a minor caveat; it undermines the main quantitative results. The paper also appears to select its thresholds in-sample without a clearly reported out-of-sample validation, raising the risk of optimism bias. The descriptive engine comparisons and pillar-level summaries retain some value, but the central predictive claim is not supported as presented.
major comments (3)
- [Sections 4.1 and 5.4] The logistic regression and threshold metrics require a binary outcome Y(u) with both cited and non-cited URLs. The methodology defines U as the set of audited unique URLs collected from citations (Section 4.3), so Y=1 for all audited pages. No procedure is described for sampling or identifying a negative set of URLs that could have been cited but were not. Consequently, the reported odds ratio (OR=4.2, 95% CI [3.1, 5.7]) and the specificity (0.84) in Section 5.4 are not identified from the described data. The paper must explicitly define the control population, the sampling scheme, and the outcome coding; otherwise the central quantitative claims are unsupported.
- [Sections 4.5 and 5.4] The threshold analysis selects g* and h* by maximizing Youden's J, but the paper does not report how the validation split was created, used, or whether the reported threshold performance is from the training or validation partition. The sentence 'Given a validation split' in Section 4.5 is not operationalized. If the thresholds are chosen and evaluated on the same data, the reported sensitivity, specificity, and 78% figure are in-sample results and likely optimistic. The authors should provide a clear train/validation/test decomposition, or cross-validated estimates, and report performance on held-out data.
- [Table 2] The 'Citation Rate' column in Table 2 is undefined. If all audited URLs are cited by at least one prompt-engine pair, then a 'rate' such as 78% for Brave lacks a denominator: is it the fraction of prompts for which Brave cited at least one page meeting a threshold, the fraction of audited pages cited by Brave, or something else? Without a defined denominator, these numbers cannot be interpreted or compared across engines. This is directly relevant to the operating-point claim in Section 5.4.
minor comments (5)
- [Section 4.4] The indicator notation in Section 4.1 uses '⊮' instead of the standard '𝟙' or '1{...}', making the equations hard to read. Please use a conventional indicator notation throughout.
- [Section 4.5] The logistic regression section does not specify how the outcome variable is defined for the model. If Y_any(u) is used, the model includes only cited pages; if engine-specific Y_e is used, the paper should clarify how non-cited URLs enter the sample. This is related to Major Comment 1 but also needs clarification in the statistical methods.
- [Section 5.5] Figures 1–5 are referenced but the actual images are not included in the manuscript text provided for review. The captions are detailed, but the figures themselves should be present for a complete evaluation.
- [References] References [5]–[8] (DPR, ColBERT, RAG, Self-RAG) are listed but not cited in the body of the paper. Either remove them or integrate them into the related work discussion.
- [Limitations] The limitations section mentions unobserved confounding and construct validity but does not acknowledge that the absence of a non-cited control group prevents the estimation of citation likelihood entirely. This omission should be corrected if the paper is revised.
Circularity Check
Central operating point and its performance metrics are in-sample fits, not predictions; the logistic 'citation' outcome lacks a non-cited control set as described.
-
fitted input called prediction
[Sections 4.1, 4.5, 5.4]
"threshold identification— find operating points( g,h )for a classifier fg,h(u) = ⊮{G (u)≥g∧H (u)≥h} that optimise a utility criterion (e.g., Youden’sJ, precision–recall F1, or lift). ... Thresholds forG andH are selected by maximizing Youden’sJ or micro-averaged F1. ... GEO≥ 0.70marks a strong inflection: sensitivity= 0.78, specificity= 0.84, with Youden’s indexJ = 0.62."
The operating point (G≥0.70, H≥12) is selected as the maximizer of Youden's J on the same corpus, and the reported J=0.62, sensitivity, specificity, and the associated 78% citation rate are the optimized objective and in-sample confusion-matrix values. The paper then presents this fitted threshold as an 'emerging' practical operating point and as an actionable benchmark. Because no validation split or external benchmark is described, these numbers are properties of the fitted classifier, not predictions of citation behavior.
-
self definitional
[Abstract and Sections 4.1, 4.3]
"Using 70 product intent prompts, we collected 1,702 citations across three engines ... and audited 1,100 unique URLs. ... LetU the set of audited URLs (|U| = 1 100). ... For each(p,e )∈P×E , letRp,e⊆U be the set of URLs cited by enginee for promptp after normalization/deduplication. ... For each(p,e )we collect Rp,e from Brave Summary, Google AIO and Perplexity (sonar-pro)."
The audit set U is constructed entirely from URLs that were cited at least once (1,100 unique URLs from 1,702 citations), so by construction Y_any(u)=1 for every audited URL. The logistic regression outcome is therefore not 'cited vs. not cited'; any negative example is only a URL cited by a different engine. Consequently, the reported OR=4.2 for GEO is a contrast among already-cited pages (engine preference), not an estimate of citation likelihood. The paper's own limitations section acknowledges unobserved confounding but not this more fundamental absence of a non-cited control set.
full rationale
The GEO-16 framework itself is a descriptive scoring instrument; computing G and correlating it with citation indicators is not circular. The circularity is concentrated in the threshold analysis and the construction of the outcome sample. First, the operating point (G≥0.70, H≥12) is chosen by maximizing Youden's J on the same data, and the reported J, sensitivity, specificity, and 78% citation rate are in-sample optimized values, not out-of-sample predictions. Second, the audit design defines U only from cited URLs, so the binary outcome is degenerate for Y_any and the logistic 'citation likelihood' is really a conditional engine-preference model; the central OR=4.2 is not identified as a predictor of citation versus non-citation from the described data. No load-bearing self-citations are present: references [1] and [4] are by Aggarwal et al., not the present authors. The paper honestly notes its observational design and unobserved confounding in Section 6, but that limitation statement does not cover the missing negative set or the in-sample threshold selection. These issues make the headline findings partially circular, though the pillar-level correlations retain independent descriptive content.
Assumptions & free parameters
free parameters (3)
- GEO score threshold g* =
0.70
- Pillar hit threshold h* =
12
- Sub-signal weights w_j,i
assumptions (2)
- domain assumption The fully rendered DOM at fetch time approximates crawler-visible content for the studied engines
- domain assumption Citation outcomes are stable and URL normalization/deduplication does not introduce systematic bias
invented entities (1)
-
GEO-16 framework
Cite this review
Pith. "Pith review of AI Answer Engine Citation Behavior An Empirical Analysis of the GEO16 Framework." pith.science (2026). https://pith.science/paper/52J2WZE5
@misc{pith2026250910762,
author = {Pith},
title = {Pith review of: AI Answer Engine Citation Behavior An Empirical Analysis of the GEO16 Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/52J2WZE5}},
note = {Machine review of arXiv:2509.10762}
}
read the original abstract
AI answer engines increasingly mediate access to domain knowledge by generating responses and citing web sources. We introduce GEO-16, a 16 pillar auditing framework that converts on page quality signals into banded pillar scores and a normalized GEO score G that ranges from 0 to 1. Using 70 product intent prompts, we collected 1,702 citations across three engines (Brave Summary, Google AI Overviews, and Perplexity) and audited 1,100 unique URLs. In our corpus, the engines differed in the GEO quality of the pages they cited, and pillars related to Metadata and Freshness, Semantic HTML, and Structured Data showed the strongest associations with citation. Logistic models with domain clustered standard errors indicate that overall page quality is a strong predictor of citation, and simple operating points (for example, G at least 0.70 combined with at least 12 pillar hits) align with substantially higher citation rates in our data. We report per engine contrasts, vertical effects, threshold analysis, and diagnostics, then translate findings into a practical playbook for publishers. The study is observational and focuses on English language B2B SaaS pages; we discuss limitations, threats to validity, and reproducibility considerations.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
GEO: Generative Engine Optimization
Aggarwal, P., et al. “GEO: Generative Engine Optimization.”arXiv preprintarXiv:2311.09735 (2023)
arXiv 2023
-
[2]
Prompt engineering for search visibility: Challenges and limitations
Pfrommer, M., Chen, A., & Rodriguez, C. “Prompt engineering for search visibility: Challenges and limitations.”NeurIPS AI Safety Workshop(2024)
2024
-
[3]
LLM-driven content generation for search visibility
Samarah, S., Kim, J., & Wang, M. “LLM-driven content generation for search visibility.”The Web Conference(2024)
2024
-
[4]
Generative Engine Optimization: How to Dominate AI Search
Aggarwal, P., et al. “Generative Engine Optimization: How to Dominate AI Search.”arXiv preprintarXiv:2509.08919 (2025)
arXiv 2025
-
[5]
Dense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., et al. “Dense Passage Retrieval for Open-Domain Question Answering.”EMNLP (2020)
2020
-
[6]
ColBERT: Efficient and Effective Passage Search via Contextual- ized Late Interaction over BERT
Khattab, O., & Zaharia, M. “ColBERT: Efficient and Effective Passage Search via Contextual- ized Late Interaction over BERT.”SIGIR(2020)
2020
-
[7]
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS33 (2020)
2020
-
[8]
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Asai, A., et al. “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.” arXiv preprintarXiv:2310.11511 (2023). 12
arXiv 2023
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.