{"id":"c70ac2db-7d12-4083-9551-85edffd0413a","arxiv_id":"2411.13813","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Text of analyst reports, represented by LLM embeddings, explains 10.19% of three-day abnormal returns out-of-sample, exceeding the 9.01% from quantitative forecasts.","lead":"A large study of 122,252 sell-side analyst reports finds that the qualitative text, encoded with LLMs, explains more of contemporaneous stock returns out-of-sample than the analysts' quantitative forecasts. Readers see which topics move markets most, and what early access to analyst reports might be worth in dollars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's headline comparison may not be on a common sample: EFREV and TPREV are missing for a large share of reports (Table A3), and no imputation or sample sizes are reported, so text-only versus revision-only R2_oos can reflect different test sets.","rationale":"I read the paper in good faith and identify a more fundamental threat to the central claim than the Shapley computation highlighted by the reader. The primary headline — text embeddings explain 10.19% of three-day CAR versus 9.01% for forecast revisions — is only meaningful if the two Ridge models are estimated and evaluated on the same set of reports. The paper does not establish this. Section 3.1 requires three I/B/E/S revision variables, and Table A3 shows substantial missingness in EFREV (90,625 of 122,252) and TPREV (84,108). Table 3 reports no sample sizes and no imputation procedure. If the revision-only model drops missing rows, its OOS test set differs from the text-only model's test set, so the reported R2 gap and the incremental gain of the combined model are not identified. This is an internal consistency concern, not a disagreement with consensus. The reader's Shapley concern is valid and should be addressed — Eq. 3 says nothing about how 131,072 Ridge regressions on 5,120-dimensional embeddings were computed — but it is secondary to the primary R2 comparison and to the abstract's central claim. I also note the reader's correct observation that the DM t-statistic of 1.66 is not significant; unequal samples would make even that comparison unreliable. The paper has genuine strengths: out-of-sample design with expanding window, multiple embeddings and ML algorithms, a 2023-only lookahead test, and a same-sample horse race in Table 4 (N=28,837) showing both text and revisions are significant. Those strengths support a conditional acceptance pending clarification of the sample. My verdict therefore remains UNCHANGED relative to the reader's CONDITIONAL, but for a different, more central reason.","tokens_in":39200,"tokens_out":7905,"duration_ms":71931,"concrete_test":"Re-estimate the four configurations in Table 3 on the common subsample of reports with nonmissing RECREV, EFREV, and TPREV (and available text embeddings). Also run a robustness version that imputes missing revisions (e.g., zero or median) to retain the full sample. Recompute the overall 2015–2023 R2_oos and DM test statistics for Revision-only, Text-only, and Rev+text on the identical test set. If Text-only no longer exceeds Revision-only, or if the Rev+text increment over Text-only shrinks materially, the headline comparison is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim — that text embeddings explain 10.19% of three-day CAR out-of-sample versus 9.01% for forecast revisions — is identified only if both models are evaluated on the same observations. Section 3.1 defines the revision inputs as RECREV, EFREV, and TPREV, each requiring a prior forecast in I/B/E/S for the same analyst–firm pair. Table A3 reports N=120,673 for RECREV, 90,625 for EFREV, and 84,108 for TPREV against 122,252 reports in the full sample. A Ridge regression with three revision regressors will drop any report missing one of these fields; a text-embedding regression has no such requirement. Table 3 does not report subsample sizes, and the text does not describe imputation for missing forecast revisions. If the revision-only and Rev+text models are estimated on the roughly 84k-report subsample while text-only uses the full roughly 122k sample, the 1.18 percentage-point gap and the 2.09pp incremental gain from 'Rev + text' over 'Text only' are not identified: they conflate information content with sample composition. The DM t-statistic of 1.66 for (5)-(1) is already far from significance; unequal samples make even that test's validity questionable. The reader's Shapley-value computational concern (Eq. 3, 2^17 subsets) is real and should be disclosed, but it affects the topic-decomposition interpretation, not the primary R2 comparison. The sample-comparability issue directly targets the abstract's 'more economically significant than quantitative forecasts.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the information content of sell-side analyst reports by combining LLM-based text embeddings with out-of-sample Ridge regressions. Using 122,252 Investext reports for S&P 100 stocks from 2000 to 2023, it reports that text embeddings explain 10.19% of three-day cumulative abnormal returns out-of-sample, versus 9.01% for quantitative forecast revisions, and that a combined model reaches 12.28%. The paper also introduces a 17-topic taxonomy (generated with ChatGPT-4o and a fine-tuned BERT classifier) and uses a Shapley value decomposition (Eq. 3) to attribute the text-based R2_OOS across topics, finding that Income Statement Analysis dominates. Economic significance is assessed by translating the predictive R2 into dollar values through the Kadan and Manela (2025) framework, yielding an estimated annualized information value of about $6.89 million for an average S&P 100 stock. The main contribution is the claim that qualitative narrative in analyst reports carries information beyond the quantitative forecasts, with the decomposition and dollar-value estimates as secondary contributions.","tokens_in":39471,"tokens_out":4471,"duration_ms":39639,"significance":"If the central claim holds, the paper provides novel, large-sample evidence on the value of qualitative analyst text, complementing the established literature on quantitative revisions and sentiment. The empirical design has notable strengths: the out-of-sample evaluation uses an expanding-window scheme with a 2015-2023 test period; the main result is robust across several LLMs (BERT, OpenAI, LLaMA-3), multiple ML algorithms (PLS, XGBoost, neural networks), alternative CAR windows, and a 'numbers removed' analysis; and the 2023-only test addresses look-ahead leakage in a credible way. The economic translation through an external theoretical framework adds a practical dimension that is rare in this literature. The topic-decomposition and dollar-value results, however, rest on computational and sample-comparability assumptions that are not fully disclosed, as detailed in the major comments. With those points resolved, the paper would be a valuable contribution to the analyst-information and AI-in-finance literatures.","major_comments":[{"comment":"The headline comparison between 'Text only' (R2_OOS = 10.19%) and 'Revision only' (R2_OOS = 9.01%) may not be evaluated on a common sample. Table A3 shows that EFREV has N=90,625 and TPREV has N=84,108, while the full report sample is N=122,252 and RECREV has N=120,673. The text-only model requires no forecast-revision fields, while the revision-only and 'Rev + text' models will drop any report missing RECREV, EFREV, or TPREV unless explicit imputation is performed. Table 3 does not report subsample sizes, and the text does not describe imputation for missing revisions. If the revision-based models are estimated on the roughly 84k-report subsample while text-only uses the full 122k sample, the 1.18 percentage-point gap and the 2.09pp incremental gain from adding text to revisions are not identified, because they conflate information content with sample composition. Please report all R2_OOS values and DM tests on a common set of observations, or clarify the imputation procedure.","section":"Section 3.2.1, Table 3 and Table A3"},{"comment":"The Shapley value decomposition as written in Eq. (3) requires computing out-of-sample R2 for every subset of the 17 topics, i.e., 2^17 = 131,072 Ridge regressions on 5,120-dimensional embeddings, each with cross-validated regularization over an expanding window. The paper reports exact-looking Shapley values in Figure 2 and claims that Income Statement Analysis accounts for 67% of total R2_OOS, yet it does not describe any approximation, such as Monte Carlo Shapley, truncation, or a linearity assumption. Without a feasible and disclosed approximation, the numerical topic contributions and the central decomposition result in Section 3.2.3 are not verifiable. Please describe the actual computational procedure, the approximation error bounds, or provide code/reproducibility details.","section":"Section 2.3, Eq. (3) and Section 3.2.3, Figure 2"},{"comment":"The abstract states that qualitative information is 'more economically significant than quantitative forecasts,' but the direct DM test comparing 'Text only' and 'Revision only' yields a t-statistic of 1.66 (Table 3, row 'Overall', column '(5)-(1)'), which is not statistically significant at conventional levels. While the body text appropriately says 'comparable or marginally larger,' the abstract overstates the evidence. Please temper the abstract claim or provide a same-sample test that would support a stronger statement.","section":"Abstract and Section 3.2.1"}],"minor_comments":[{"comment":"The word 'expriment' in the sentence 'I expriment a standard K-Means algorithm' should be 'experiment'.","section":"Section 2.2"},{"comment":"The description of the 'CLNV' series in Figure A8 says it uses the trade-signing algorithm of Lee and Ready (1991), but the main analysis already uses Lee and Ready, and the table labels 'CLNV' as the algorithm of Chakrabarty et al. (2007). Please clarify which method corresponds to which series, and fix the spelling of 'volatility' in Table A8.","section":"Table A8 and Figure A8"},{"comment":"The Shapley values are described as 'relative contribution' to total R2_OOS, but with 17 topics some Shapley values are negative (e.g., 'minimal or even negative'). Please state explicitly how negative Shapley values are normalized or whether the reported percentages are computed over the sum of positive values only.","section":"Section 3.2.3"},{"comment":"The label 'ToneIncome,NB/BERT' in the table header is ambiguous about the two separate measures; consider writing 'ToneIncome,NB' and 'ToneIncome,BERT' explicitly to match the reporting in the text and panel notes.","section":"Section 3.2.5, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The two main issues are (1) sample comparability for the central Table 3 comparison and (2) the feasibility and disclosure of the Shapley computation. Both are fixable with additional analysis and transparency, but they are essential before the paper's headline claims can be accepted. I also note that the topic taxonomy ultimately rests on ChatGPT-generated categories and a classifier trained on GPT-4 labels without human annotation, which is a limitation worth acknowledging in the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe headline result of this paper is probably not as clean as the abstract suggests. The main comparison — text embeddings explaining 10.19% out-of-sample versus 9.01% for forecast revisions — is likely not estimated on a common sample. Table A3 shows that the revision variables RECREV, EFREV, and TPREV are missing for a large share of reports (120,673, 90,625, 84,108 observations vs. 122,252 total), and the paper never says it imputes or restricts to the overlapping subsample. So “Text only” may be evaluated on many more reports than “Revision only” or “Rev + text,” and the 1.18 percentage point gap becomes an artifact of sample selection. The DM t-statistic of 1.66 already fails to reject equality; with non-overlapping samples, the comparison is even less informative. This directly weakens the abstract’s claim that text is “more economically significant” than numbers.\n\nThe second issue is the Shapley decomposition. Equation (3) sums over all 2^17 subsets, each requiring a Ridge regression on 5120-dimensional embeddings. That is 131,072 regressions, and no approximation is disclosed. As written, the computation is infeasible; the reported topic importance (income statement > 50%) is therefore unsupported without an explicit approximation.\n\nThat said, the paper has a real core. The overall out-of-sample R² of around 10% from LLM embeddings is a meaningful finding, and the robustness across different LLMs, ML algorithms, and event windows (Tables 3B, A4–A6) gives me some confidence that the effect is not a fluke. The horse race in Table 4, with controls for earnings surprise and prior returns, is a sensible way to separate text from public signals. The idea of applying Shapley values to LLM embeddings for interpretability is genuinely useful, if the implementation becomes transparent.\n\nThe economic-value section is speculative but clearly labeled; I don’t flag it as a major flaw, but readers should keep in mind it’s a transformation of the same R², not independent evidence.\n\nOverall: this deserves peer review, but only after major revision. The authors must report common-sample estimates, disclose the Shapley approximation (or use an exact method on a smaller topic set), and soften the abstract. It would be a useful reading-group paper for discussing ML evaluation pitfalls in finance.","headline":"The paper's headline text-vs-numbers comparison likely conflates samples, and the Shapley decomposition as written is infeasible; the robust out-of-sample text signal still deserves peer review after major fixes.","tokens_in":40065,"tokens_out":4742,"would_cite":false,"duration_ms":41352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the narrative text in sell-side analyst reports explains 10.19% of three-day abnormal stock returns out-of-sample, more than the 9.01% explained by quantitative forecast revisions, and that combining text with…","keywords":["Sell-side analyst reports","Value of information","Large language models","Text embeddings","Out-of-sample R-squared","Shapley value decomposition","Abnormal stock returns","Earnings announcements"],"falsifier":"Re-run the exact 131,072-subset Shapley computation on the published topic embeddings and check whether income-statement analysis contributes more than half of the total out-of-sample $R^2$; if it does not, the paper's central topic-importance claim fails.","tokens_in":38887,"feed_emoji":"📈","tokens_out":7507,"duration_ms":73898,"temperature":0.7,"pith_summary":"This paper tries to establish that what analysts write matters more to the stock market than the numbers they publish. Using large-language-model embeddings of 122,252 analyst reports on S&P 100 firms, it finds that report text alone explains 10.19% of three-day abnormal returns out-of-sample, beating the 9.01% explained by forecast revisions, and that the two together reach 12.28%. If this is right, investors reading analysts' narratives are getting the main information product, not a redundant commentary on the numbers. The paper further claims that income-statement interpretation accounts for more than half of the text's explanatory power, and that early access to this information is worth about $0.47 million per three-day event, or $6.89 million annually for an average large-cap stock.","feed_headline":"Analyst report text explains returns better than forecasts","feed_subtitle":"Narrative embeddings explain 10.2% of three-day abnormal returns; forecast revisions explain 9.0%; together, 12.3%.","key_machinery":"The key machinery is a three-stage pipeline: first, each report is mapped to a 5,120-dimensional embedding by averaging token and layer vectors from a large language model, with a sentence-segmented variant used to limit cross-topic contamination; second, out-of-sample ridge regressions of three-day cumulative abnormal returns on these embeddings produce $R^2_{\\mathrm{oos}}$ values in an expanding-window design from 2000 through 2023; third, a Shapley value decomposition allocates the total out-of-sample $R^2$ across 17 predefined report topics by averaging each topic's marginal contribution over all subsets of topics, thereby attributing explanatory power while accounting for topic interactions.","core_discovery":"The central discovery is that the qualitative content of analyst reports, represented by LLM embeddings, has greater out-of-sample explanatory power for contemporaneous stock returns than the quantitative forecasts in the same reports. Text alone gives an out-of-sample $R^2$ of 10.19% for three-day cumulative abnormal returns, compared with 9.01% for forecast revisions and 9.08% for a broader set of 17 numerical measures; combining text and revisions raises the $R^2$ to 12.28%. This result survives removing all numbers from the text, which actually raises the text-only $R^2$ to 10.95%, and it is not explained by simple sentiment measures, which reach at most about 3.7%. A Shapley value decomposition attributes roughly two-thirds of the text's explanatory power to income-statement analysis, and within that topic, interpretation of realized income contributes about three times as much as reporting raw data. Translating the explained return variance into dollar terms through an imperfect-competition trading model, the paper estimates that early acquisition of analyst reports yields $0.38 million per event from text, $0.34 million from revisions, and $0.47 million combined.","pith_inferences":["The paper does not build a tradeable strategy; a natural extension is to construct a long-short portfolio sorted on the text-only predicted abnormal return and measure net returns after transaction costs, testing whether the $0.47 million gross value survives trading frictions.","Because the sample is restricted to S&P 100 stocks, the size-liquidity channel found in the paper may not generalize; for smaller, less liquid firms the information channel could dominate and make analyst text relatively more valuable per dollar traded.","The topic hierarchy is defined relative to one set of 17 ChatGPT-generated categories; a different topic taxonomy might redistribute credit, so the 'income statement is half the value' conclusion should be tested with an independently constructed labeling scheme.","A controlled experiment that presents investors with number-only summaries versus full interpretive narratives for the same earnings outcome would directly test the paper's claim that interpretation, not information acquisition, drives the market reaction."],"forward_implications":["If analyst narratives carry more value-relevant information than forecast numbers, then consensus estimates and recommendation ratings are incomplete summaries of analyst output, and investors who read only the numbers miss a distinct, somewhat larger source of return predictability.","The 12.28% combined $R^2$ being significantly higher than either input alone implies that quantitative forecasts and qualitative text are complements rather than substitutes.","Because the text-only signal survives when all digits are stripped from the reports, the market is reacting to narrative framing and reasoning, not merely to the numbers embedded in the prose.","The sharp peak in information content in the week after earnings announcements means the timing of report dissemination is economically important, not just its content.","The dollar-value estimates imply that selective early distribution of analyst reports is worth millions of dollars per large-cap stock each year, providing a concrete measure of the incentive behind tipping practices."],"supporting_citations":[{"why":"Defines report information content through $R^2$ and documents the price impact of analyst reports, providing the paper's baseline framework for comparing text versus quantitative information.","marker":"Asquith et al., 2005"},{"why":"Supplies the Naive Bayes tone measure and the forecast-linking window used to match reports to I/B/E/S revisions; the paper's sentiment comparison target.","marker":"Huang et al., 2014"},{"why":"Provides the out-of-sample $R^2$ definition and the Diebold-Mariano-type test used for all statistical comparisons in the paper.","marker":"Gu et al., 2020"},{"why":"Establishes analysts' information discovery versus interpretation roles and motivates the topic-level decomposition of report value.","marker":"Huang et al., 2018"},{"why":"Defines the cooperative-game Shapley value used to allocate total explanatory power across the 17 report topics.","marker":"Shapley, 1953"},{"why":"Provides the strategic-information-value model and lower-bound interpretation used to convert explained return variance into dollar profits.","marker":"Kadan and Manela, 2025"},{"why":"Supports the use of large language model embeddings to measure textual information relevant to asset pricing, which the paper adapts to analyst reports.","marker":"Chen et al., 2022"},{"why":"Supplies the analyst-name matching and BERT-based boilerplate removal procedures used to construct the matched report sample.","marker":"Li et al., 2024"},{"why":"Frames the look-ahead bias concern and motivates the 2023 out-of-sample evaluation with pre-2023-trained language models.","marker":"Sarkar and Vafa, 2024"}],"fun_headline_variants":["Why analyst narratives beat their numbers","Analyst words move markets more than forecasts","Text from analysts explains stock returns more than numbers","Qualitative analyst info outpredicts quantitative forecasts","LLM embeddings show analyst text explains more than revisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The topic-importance result rests on the unstated feasibility of the Shapley computation: the paper reports exact-looking contributions from all 131,072 subsets of 17 topics without describing an approximation, so the whole decomposition depends on an undisclosed computational step that must reproduce those numbers.","fun_headline_variants_meta":{"raw":{"variants":["Why analyst narratives beat their numbers","Analyst words move markets more than forecasts","Text from analysts explains stock returns more than numbers","Qualitative analyst info outpredicts quantitative forecasts","LLM embeddings show analyst text explains more than revisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1938,"prompt_tokens":914,"completion_tokens":1024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":956}},"tokens_in":530,"tokens_out":1024,"duration_ms":8089,"temperature":1.0,"reasoning_tokens":956,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:51:07.555554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact 131,072-subset Shapley computation on the published topic embeddings and check whether income-statement analysis contributes more than half of the total out-of-sample $R^2$; if it does not, the paper's central topic-importance claim fails.","supporting_citations":[],"review_version":1}