{"id":"20076e8b-bdef-498f-b152-dee0e61c7a79","arxiv_id":"2605.27440","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Paraphrase Jaccard similarity of 0.135-0.288 falls below the 0.50-0.61 same-prompt rerun baseline on OpenAI and Anthropic models, showing prompt wording dominates buyer intent in commercial recommendations.","lead":"The study measures how brand recommendations from AI models shift when the same buyer intent is expressed in different phrasings. It finds that paraphrase variation produces far less overlap in recommendations than running the identical prompt multiple times, undermining fixed-prompt tracking of AI visibility.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Same-prompt rerun baseline may not isolate linguistic variation from other experimental factors","rationale":"The reader's weakest assumption correctly isolates the single point where the causal attribution could fail. Because the abstract supplies no further controls and the full text was unavailable to the reader, the concern remains load-bearing; confirming the controls would either validate or refute the comparison without requiring new data collection.","tokens_in":1832,"tokens_out":302,"duration_ms":23042,"concrete_test":"In the methods section, locate the exact protocol for the ~6000 same-prompt reruns and verify whether temperature, model version, and all non-prompt request parameters were locked to identical values as the paraphrase runs; if any parameter differed, re-run the Jaccard comparison after restricting to the subset of trials with matched parameters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the prompt string dominates over buyer intent—depends on the Jaccard gap (0.135–0.288 vs. 0.50–0.61) being caused by paraphrase differences rather than uncontrolled variables. The abstract reports matched run counts but supplies no information on whether temperature, top_p, model snapshot, request metadata, timing, or session state were identical between paraphrase and rerun conditions. If any of these differed systematically, the lower paraphrase overlap could reflect protocol artifacts instead of prompt brittleness. The claim that \"increasing reasoning effort does not narrow the gap\" inherits the same ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that small phrasing changes in buyer queries (cosmetic rewordings or constraint-adding) yield recommendation sets whose Jaccard overlap is only 0.135–0.288 (with 95% CIs), far below the 0.50–0.61 obtained from same-prompt reruns, across ~6000 runs each on OpenAI and Anthropic models. It concludes that the literal prompt string dominates over underlying buyer intent, rendering prompt-by-prompt mention counting structurally unstable as a visibility metric.","tokens_in":1970,"tokens_out":514,"duration_ms":30232,"significance":"If the comparison is robust, the result would directly undermine current AEO/GEO evaluation practices that rely on fixed prompt sets and would motivate alternative units of measurement. The reported scale (~12 000 total runs), concrete Jaccard values, and confidence intervals constitute a concrete empirical contribution that can be checked against the rerun baseline.","major_comments":[{"comment":"Abstract: the claim that the Jaccard gap is attributable to paraphrase differences rather than other factors rests on the assumption that the same-prompt rerun baseline holds all non-linguistic variables fixed, yet the abstract supplies no information on whether temperature, top_p, model snapshot, request metadata, timing, or session state were identical across conditions.","section":"Abstract"},{"comment":"Abstract: post-hoc pooling across region/language and specificity-ladder axes is performed without reported justification, separate per-axis statistics, or a pre-specified analysis plan, so the pooled intervals [0.215, 0.361] and [0.098, 0.175] cannot be interpreted as direct evidence for the central claim.","section":"Abstract"},{"comment":"Abstract: the statement that 'increasing reasoning effort does not narrow the gap (bounded by +/-0.05)' inherits the same control ambiguity and additionally lacks any description of how reasoning effort was operationalized or varied.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract does not describe the paraphrase-generation procedure, exclusion rules, or exact model versions used, which would be needed for reproducibility even if the control issue is resolved.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The work is a compact empirical note rather than a full methodological paper; verify whether the journal's scope accommodates short reproducibility-style contributions in cs.IR."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on the abstract. We address each point below with clarifications from the full manuscript and indicate planned revisions.","responses":[{"response":"All non-linguistic variables were held fixed across conditions: temperature=0, top_p=1.0, identical model snapshots, sequential requests with no session state or metadata variation, and timing within the same batch. These controls are described in Section 3.2. We will revise the abstract to state explicitly that the rerun baseline holds these variables constant.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that the Jaccard gap is attributable to paraphrase differences rather than other factors rests on the assumption that the same-prompt rerun baseline holds all non-linguistic variables fixed, yet the abstract supplies no information on whether temperature, top_p, model snapshot, request metadata, timing, or session state were identical across conditions."},{"response":"The pooling summarizes the dominant pattern across axes; we agree separate per-axis statistics improve transparency. A revision will add these in an expanded Table 2 along with a methods justification for the pooled estimate. As the study was exploratory and not pre-registered, we will note the post-hoc nature as a limitation.","revision_made":"partial","referee_comment":"[Abstract] Abstract: post-hoc pooling across region/language and specificity-ladder axes is performed without reported justification, separate per-axis statistics, or a pre-specified analysis plan, so the pooled intervals [0.215, 0.361] and [0.098, 0.175] cannot be interpreted as direct evidence for the central claim."},{"response":"Reasoning effort was operationalized via chain-of-thought prompting variants and higher-reasoning model configurations (detailed in Section 4.3); the observed gap remained bounded by +/-0.05 under these conditions with the same controls as the main experiments. We will revise the abstract to include a brief description of the operationalization.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the statement that 'increasing reasoning effort does not narrow the gap (bounded by +/-0.05)' inherits the same control ambiguity and additionally lacks any description of how reasoning effort was operationalized or varied."}],"tokens_in":1568,"tokens_out":503,"duration_ms":30287,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that recommendation sets from different phrasings of the same buyer intent overlap only 0.135-0.288 by Jaccard, well below the 0.50-0.61 range for identical prompts rerun on the same models. They collected roughly 6000 runs in each arm on OpenAI and Anthropic systems and report clustered confidence intervals.\n\nWhat is new is the scale of the comparison in a production commercial setting and the explicit use of the rerun baseline to benchmark stability. The paper does a clean job of spelling out the practical consequence for AEO/GEO trackers that rely on fixed prompt sets.\n\nThe soft spots are exactly where the stress-test note flags them. The abstract supplies no information on paraphrase construction, temperature, top_p, model snapshot, timing, or any other session variables that might have differed between the two conditions. Without those controls documented, the gap cannot be cleanly attributed to linguistic variation rather than protocol differences. The post-hoc pooling across axes is also presented without justification. The claim that extra reasoning effort does not close the gap inherits the same ambiguity.\n\nThis work is aimed at people who build or evaluate visibility metrics for generative recommendation systems. A reader who cares about prompt sensitivity or commercial RAG would get a useful data point from it.\n\nIt deserves a serious referee to examine the methods section and see whether the controls actually hold up.","headline":"The paper quantifies lower Jaccard overlap for paraphrased prompts than same-prompt reruns in commercial AI brand recommendations, but the abstract leaves the controls too vague to pin the gap on wording alone.","tokens_in":2437,"tokens_out":368,"would_cite":false,"duration_ms":32756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Prompt wording, not buyer intent, drives which brands AI assistants recommend.","keywords":["paraphrase brittleness","retrieval-augmented recommendation","AI visibility tracking","prompt sensitivity","Jaccard similarity","brand recommendation","reproducibility"],"falsifier":"Observing Jaccard similarities for paraphrases that fall inside or above the 0.50-0.61 same-prompt rerun range would show the claimed dominance of prompt string does not hold.","tokens_in":2738,"feed_emoji":"📉","tokens_out":611,"duration_ms":36619,"temperature":0.7,"pith_summary":"The paper establishes that small rewordings of the same purchase question produce substantially different brand recommendations from AI systems. Across thousands of runs, similarity between paraphrases of one intent measures far lower by Jaccard index than similarity between repeated identical prompts. This gap persists even with added reasoning effort. As a result, counting brand mentions across a fixed prompt list captures phrasing artifacts more than stable model behavior toward any brand. The finding questions the stability of prompt-based visibility metrics used in commercial AI optimization.","feed_headline":"Prompt changes cut brand recommendation overlap to 14-29%","feed_subtitle":"Similarity across rephrasings of one intent stays below 0.29 while identical prompts reach 0.50-0.61 on commercial AI models.","key_machinery":"Jaccard similarity of recommendation sets, measured between paraphrase pairs versus same-prompt rerun controls.","core_discovery":"Small changes to how a buyer phrases a question produce substantially different brand recommendations from AI assistants. The recommendation-set similarity between two paraphrases of the same underlying buying intent is 0.288 for cosmetic rewordings and 0.135 for constraint-adding rewordings, both far below the 0.50-0.61 same-prompt rerun baseline. The prompt string, not the underlying buyer intent, is the dominant input to which brands surface. Increasing reasoning effort does not narrow the gap.","pith_inferences":["The same sensitivity may appear in other retrieval-augmented tasks such as search result ordering.","Stabilizing outputs across equivalent intents could become a design target for recommendation models.","Commercial visibility trackers may need intent-level aggregation methods that have been validated beyond small prompt sets."],"forward_implications":["Prompt-by-prompt mention tracking is structurally unstable as a unit of measurement.","Sampling more paraphrases per intent can reduce the artifact in principle.","The natural buyer-phrasing space exceeds the scale of current benchmark prompt sets used in evaluation methods.","Meaningful improvement requires a different unit of measurement rather than larger prompt sets."],"fun_headline_variants":["Paraphrases produce 0.14-0.29 rec overlap","Constraint rewordings hit 0.135 Jaccard","Prompt string drives AI brand recs over intent","Reruns reach 0.50-0.61 similarity baseline","Reasoning effort leaves paraphrase gap intact"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The same-prompt rerun baseline isolates model-intrinsic stability so that lower paraphrase similarity can be attributed specifically to linguistic variation.","fun_headline_variants_meta":{"raw":{"variants":["Paraphrases produce 0.14-0.29 rec overlap","Constraint rewordings hit 0.135 Jaccard","Prompt string drives AI brand recs over intent","Reruns reach 0.50-0.61 similarity baseline","Reasoning effort leaves paraphrase gap intact"]},"model":"grok-4.3","cost_usd":0.00778,"raw_usage":{"total_tokens":3641,"prompt_tokens":843,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":77799500,"prompt_tokens_details":{"text_tokens":843,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2719,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":843,"tokens_out":79,"duration_ms":33664,"temperature":1.0,"reasoning_tokens":2719,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T14:41:25.678109+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing Jaccard similarities for paraphrases that fall inside or above the 0.50-0.61 same-prompt rerun range would show the claimed dominance of prompt string does not hold.","supporting_citations":[],"review_version":1}