{"id":"c0ce01f8-6266-4536-9add-59920306d434","arxiv_id":"2508.00961","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"FinKario combines an event-enhanced financial knowledge graph dataset with a two-stage graph retrieval method, reporting 18.81% better stock-trend prediction than financial LLM baselines.","lead":"The paper presents FinKario, a financial knowledge graph built from company reports and market events, along with a graph-based retrieval method for large language models. It claims these improve stock-trend prediction accuracy by 18.81% over financial LLMs in backtests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central backtest claim is unverifiable from the submitted text: the full manuscript is corrupted and the abstract alone does not document transaction costs, temporal split, or stock universe.","rationale":"I agree with the reader's assessment. The strongest claim is quantitative and depends entirely on the validity of the backtest, which cannot be inspected because the manuscript text is corrupted beyond the abstract. The reported improvements over financial LLMs and institutional strategies are the entire basis for the paper's claimed contribution, and the abstract alone omits the details needed to rule out lookahead bias, survivorship bias, or unrealistic cost assumptions. The paper may be perfectly sound, but the current submission provides no way to tell. Therefore the appropriate verdict remains UNVERDICTED, and my read does not change the reader's verdict.","tokens_in":11676,"tokens_out":2364,"duration_ms":23031,"concrete_test":"Obtain the intact LaTeX/PDF source and reproduce the backtest in the experimental section from the released data and code, using only point-in-time data and charging stated transaction costs; if the 18.81% and 17.85% margins drop below noise or change sign under a strict temporal train/test split, the abstract's implied superiority claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that FinKario with FinKario-RAG outperforms financial LLMs by 18.81% and institutional strategies by 17.85% in backtesting—can only be true if the backtest is fair. The manuscript body supplied for review is an undecodable encoding artifact, so the experimental section, tables beyond the abstract, and any ablation or sensitivity analyses cannot be inspected. This leaves the reported margins exposed to the standard financial-evaluation failure modes: lookahead bias (e.g., using fundamentals or market events dated after the prediction target), survivorship bias in the stock universe, and unrealistic transaction-cost or slippage assumptions. The abstract reports 'accuracy' and 'backtesting' without stating the prediction horizon, benchmark definitions, statistical significance, or whether the institutional strategies are out-of-sample. None of this is an accusation of error; it is simply that the submitted artifact does not contain enough intact information to test the premise. Since the whole quantitative contribution rests on these two percentages, the paper cannot currently be verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinKario, a dataset for automatically constructing financial knowledge graphs from equity research reports, enriched with real-time company fundamentals and market events via prompt-driven extraction guided by institutional templates. It also introduces FinKario-RAG, a two-stage graph-based retrieval strategy intended to provide timely, structured financial knowledge to LLMs. The central empirical claim, stated in the abstract, is that FinKario with FinKario-RAG achieves superior stock trend prediction accuracy, outperforming financial LLMs by 18.81% and institutional strategies by 17.85% on average in backtesting. However, the submitted full text is a corrupted, undecodable encoding artifact; only the abstract and fragments of the paper are readable, so the experimental setup, tables, ablations, and baseline definitions cannot be inspected.","tokens_in":11859,"tokens_out":4433,"duration_ms":39012,"significance":"The paper addresses a real gap in financial LLM applications: the rapid evolution of market events versus the slow update cycles of knowledge bases, and the long-form unstructured nature of financial reports. If the claims hold, the FinKario dataset (with 305,360 entities, 9,625 relational triples, and 19 relation types) and the FinKario-RAG retrieval strategy could be useful contributions to automated financial analysis. The paper proposes a concrete, plausible approach to event-enhanced knowledge graph construction and retrieval. However, the headline empirical results are entirely unverifiable from the submitted artifact: no methodology, error bars, or significance tests are visible, and no code or dataset release is described in the accessible text. The significance of the contribution therefore cannot be assessed until a readable manuscript with full experimental details is provided.","major_comments":[{"comment":"The paper's central empirical claim—that FinKario with FinKario-RAG achieves 18.81% average improvement over financial LLMs and 17.85% over institutional strategies in backtesting—is stated without any description of the evaluation protocol. The submitted full text is a corrupted, undecodable encoding, so no experimental details (data period, stock universe, prediction horizon, transaction costs, benchmark definitions, error bars, or significance tests) are available for inspection. Because the quantitative contribution rests entirely on these two percentages, the authors must resubmit a readable manuscript and supply the experimental setup in full before the claim can be assessed.","section":"Abstract"},{"comment":"The knowledge graph is constructed from equity research reports, and the same reports appear to underlie the stock prediction evaluation. This creates a concrete risk of label leakage or evaluation circularity if any report content or market event dated after the prediction target is used in the graph at inference time. The authors should specify the temporal ordering between graph construction, report timestamps, and prediction dates, and should report an out-of-time evaluation (e.g., a holdout period after the knowledge cutoff) as a leakage test.","section":"Evaluation design (inferred from abstract)"},{"comment":"The abstract reports 305,360 entities and 9,625 relational triples, which is a strikingly low triple-to-entity ratio (about 0.03 triples per entity) for a financial knowledge graph and may indicate a typographical error. The authors should clarify the exact scale of the graph and report the distribution of the 19 relation types, as this affects the dataset's utility and the credibility of the 'over 305,360 entities' claim.","section":"Dataset statistics (abstract)"}],"minor_comments":[{"comment":"The abstract should state the prediction horizon (e.g., daily, weekly) and the exact metric used for 'accuracy' (e.g., directional accuracy, F1, Sharpe ratio) to make the headline improvements interpretable.","section":"Abstract"},{"comment":"The comparison against 'institutional strategies' is undefined; the authors should name the specific strategies and their source (e.g., published backtests, benchmark indexes) in the resubmission.","section":"Abstract"},{"comment":"The phrase 'on average' should be qualified with the number of stocks/periods and the dispersion across them, since average improvements can hide large variance.","section":"Abstract"},{"comment":"The full text supplied for review is corrupted and undecodable; the authors should ensure the resubmitted PDF is intact and includes readable tables and figures for all experimental results.","section":"Full text (submitted artifact)"}],"recommendation":"major_revision","confidential_remarks":"The uploaded manuscript is a corrupted encoding artifact; the technical content beyond the abstract cannot be reviewed. I recommend asking the authors to resubmit a readable PDF before sending the paper for a full technical review. No judgment about the validity of the claims is possible from the current artifact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the idea has legs, but the submitted manuscript is a mojibake artifact beyond the abstract. I can only assess the abstract, so the headline backtest gains are unverified.\n\nWhat looks genuinely new: FinKario as a dataset artifact—305k entities, 9,625 relational triples, 19 relation types—built by prompt-driven extraction guided by institutional templates, and FinKario-RAG as a two-stage graph retrieval strategy on top of it. That combination (event-enhanced KG construction plus two-stage retrieval for financial LLMs) is a reasonable extension of KG/RAG work into finance. The problem framing is sensible: equity research reports are long, unstructured, and update more slowly than market events.\n\nWhat I cannot check: the experimental section, tables, ablations, code, or dataset. The abstract reports 18.81% and 17.85% improvements over financial LLMs and institutional strategies in backtesting, but leaves out prediction horizon, benchmark definitions, transaction costs, statistical significance, and whether the institutional strategies are out-of-sample. The stress-test concerns about lookahead and survivorship bias are legitimate, not because the authors did anything wrong, but because the artifact contains no information to rule them out. The two percentages are load-bearing; they deserve scrutiny before citation.\n\nOne more soft spot: the graph is built from the same reports used for evaluation. That opens a possible circularity/leakage channel—if the KG contains future-relative information, retrieval gains could be inflated. Not a claim of error; just unanswerable from the text.\n\nWho this is for: financial NLP researchers working on RAG for investor-facing applications, and anyone building public KG resources. If the dataset and code are released and the backtest design is clean, this could be a solid resource paper.\n\nRecommendation: don't desk-reject the line of work—ask for a clean, readable manuscript plus code/data availability, then send it out. As submitted, no referee can responsibly evaluate the central claim.","headline":"Plausible financial KG/RAG pipeline with a real dataset artifact, but the submitted full text is unreadable, so the headline backtest gains are unverified.","tokens_in":12316,"tokens_out":2909,"would_cite":false,"duration_ms":28317,"reading_group":"no","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinKario gives LLMs event-fresh financial knowledge and beats baseline predictors in backtests.","keywords":["financial knowledge graph","large language models","stock trend prediction","retrieval-augmented generation","event extraction","equity research reports","backtesting"],"falsifier":"Re-run the evaluation with point-in-time data, a fixed universe that includes stocks that later delisted, and realistic transaction costs. If the reported 18.81% and 17.85% average gains shrink to statistical noise, the central claim fails; a simpler audit is to check whether any triple used in a prediction carries a date after the prediction date.","tokens_in":11533,"feed_emoji":"📈","tokens_out":3622,"duration_ms":31860,"temperature":0.7,"pith_summary":"The paper claims that stock-trend prediction by large language models can be materially improved by giving the models a financial knowledge graph that is built automatically from equity research reports and refreshed with real-time market events. That claim is backed by a dataset, FinKario, with over 305,360 entities, 9,625 relational triples, and 19 relation types, plus a two-stage graph-based retrieval method, FinKario-RAG, that selects the most relevant knowledge for each query. In backtests, the paper reports average accuracy gains of 18.81% over financial LLMs and 17.85% over institutional strategies. If those numbers hold, the contribution is a practical recipe: structure fast-moving financial information so LLMs can actually use it, without needing to retrain the models.","feed_headline":"Event-updated knowledge graph lifts LLM stock predictions","feed_subtitle":"FinKario's two-stage retrieval beats financial LLMs by 18.81% and institutional strategies by 17.85% in backtests.","key_machinery":"The central object is the FinKario knowledge graph: more than 305,360 entities connected by 9,625 relational triples of 19 types, capturing company fundamentals and market events. It is constructed automatically by prompting LLMs to extract structured facts from long-form equity research reports using professional institutional templates. The second load-bearing piece is FinKario-RAG, a two-stage, graph-based retrieval strategy that first narrows the candidate set and then retrieves the most relevant triples for a stock-trend question, so the LLM sees current, structured evidence rather than raw reports.","core_discovery":"The paper's central claim is that the bottleneck for LLM-based stock analysis is not model capability but knowledge access: equity research reports are long, unstructured, and slow to update, while market events move faster than knowledge bases. FinKario addresses this by automatically converting company fundamentals and market events into a structured graph through prompt-driven extraction guided by professional institutional templates. FinKario-RAG then retrieves from this graph in two stages, ensuring that the LLM receives timely, context-aware evidence at prediction time. The paper reports that this combination outperforms financial LLMs by 18.81% and institutional strategies by 17.85% on average in backtesting.","pith_inferences":["A natural next test is whether the same event-graph-plus-two-stage-retrieval design transfers to other fast-moving, document-heavy domains such as regulatory filings or clinical trial news.","The reported margins may compress under realistic trading costs and capacity limits; backtests rarely price slippage, so live or simulation-based verification is the honest follow-up.","Because the graph has far more entities than triples, one could test whether relational density, rather than event freshness alone, drives the accuracy gain."],"forward_implications":["LLM stock prediction can be improved by changing what knowledge is stored and how it is retrieved, not by retraining or scaling the model.","Professional report templates make automated extraction of company fundamentals and events feasible at scale, keeping the graph aligned with how analysts actually reason.","Two-stage graph retrieval gives a concrete way to balance freshness and precision when knowledge is large and evolving.","If the backtest is sound, the method offers individual investors a defensible LLM-based analysis pipeline that closes part of the gap with institutional research."],"supporting_citations":[],"fun_headline_variants":["Event-aware knowledge graph boosts LLM stock pick accuracy","FinKario graph turns news into market edge for LLMs","Two-stage graph retrieval lifts LLM predictions by 18.8%","Automated financial knowledge graph beats human strategies","Real-time events feed LLM stock analysis, beating benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the backtest measures what a real investor could have earned: the model never sees information dated after the prediction moment, delisted stocks are not removed from the universe, and transaction costs are realistic.","fun_headline_variants_meta":{"raw":{"variants":["Event-aware knowledge graph boosts LLM stock pick accuracy","FinKario graph turns news into market edge for LLMs","Two-stage graph retrieval lifts LLM predictions by 18.8%","Automated financial knowledge graph beats human strategies","Real-time events feed LLM stock analysis, beating benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1223,"prompt_tokens":936,"completion_tokens":287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":552,"tokens_out":287,"duration_ms":3255,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:01:06.524419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with point-in-time data, a fixed universe that includes stocks that later delisted, and realistic transaction costs. If the reported 18.81% and 17.85% average gains shrink to statistical noise, the central claim fails; a simpler audit is to check whether any triple used in a prediction carries a date after the prediction date.","supporting_citations":[],"review_version":1}