{"id":"65cfbb9c-651d-4cc3-8b8a-899e5328dda7","arxiv_id":"2603.02702","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new dataset and pairing framework links stock prices to semantically relevant news at macro, sector, related-company, and target-company levels, improving stock forecast accuracy over keyword-based pairing.","lead":"This paper builds a financial dataset pairing news articles with stock prices using semantic matching and four information levels instead of simple keyword matching. The authors show that forecasts using their paired text outperform keyword-paired baselines across twelve time-series models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic-vs-keyword comparison in §4.2 is confounded with text volume; the headline claim that semantic relevance drives gains needs a volume-matched baseline.","rationale":"This is a dataset-construction paper whose pilot forecasting experiments are the main evidence for the value of the pairing method. The strongest evidence is Table 1, but the comparison is not controlled for the amount of text, and the paper's own description of forward filling indicates large coverage differences between conditions. The reader's conditional verdict already identifies this confound; my independent read agrees. The requested experiment is feasible using the released dataset and code and would settle whether semantic relevance or text volume drives the gains. I considered other potential concerns—train-only z-score normalization is not explicitly stated, error bars are absent, and the LLM-judge evaluation is subjective—but these are secondary: normalization ambiguity affects absolute values more than the semantic-vs-keyword contrast, and the LLM-judge table is correlational. Because the central unresolved issue is exactly the confound the reader identified, no change to the reader's conditional verdict is needed, but the condition should be resolved before the headline claim is accepted without qualification.","tokens_in":15562,"tokens_out":2749,"duration_ms":29795,"concrete_test":"Add a volume-matched control to §4.2: for each company-date, record the number of tokens/articles in the FNSPID keyword condition and in the Semantic condition, then (a) truncate or randomly subsample Semantic text to the keyword token budget, and (b) augment Keyword with randomly selected non-company news to the Semantic token budget. Re-run all 12 models with three seeds. If Semantic's advantage over Keyword shrinks to within noise in (a) or random augmentation closes the gap in (b), the headline pairing-strategy claim is unsupported; if the advantage persists after matching volume, the semantic explanation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that forecasting gains come from semantic relevance and multi-level structure rather than from simply supplying more text. Table 1 compares Semantic against FNSPID keyword pairing, but the two conditions differ in both selection mechanism and in the amount/consistency of text: the authors note keyword pairing leaves many company-days with no news and they apply forward filling (§4.2), while Semantic retrieves from approximately one million articles at four levels. Likewise, Figure 4's progressive additions add text at each step, so monotonic improvement is compatible with a volume effect. No baseline matches the token count, article count, or coverage of the semantic condition, so the 'consistently outperforms' result does not identify the cause. This is load-bearing because the paper's stated contribution is the pairing strategy, not a larger corpus. If volume alone drives the gains, the dataset may still be useful, but the specific claim about semantic-based and multi-level pairing would be unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinTexTS, a financial text-paired time-series dataset covering 100 U.S. companies over 2019–2023, built by a semantic-based and multi-level pairing framework. Company-specific context is extracted from SEC filings with an LLM, news articles are retrieved via a fine-tuned embedding model, and articles are classified into macro, sector, related-company, and target-company levels using LLMs, followed by LLM-based summarization. The dataset is evaluated as text augmentation for stock price forecasting with 12 time-series models against a keyword-based FNSPID baseline, a no-text baseline, and a proprietary news variant. The paper claims that semantic-based pairing consistently outperforms keyword-based pairing and that multi-level text progressively improves forecasting, with additional backtesting and quality analyses.","tokens_in":15847,"tokens_out":2711,"duration_ms":26754,"significance":"If the central claims hold, FinTexTS is a substantial resource for financial text–time-series research: it is large-scale, publicly released with code and a fine-tuned embedding model, and the evaluation spans 12 diverse forecasting architectures with consistent directional gains. The proposed framework addresses a real limitation of keyword-based pairing and the multi-level structure is a useful organizing principle. The paper also ships machine-checked reproducibility artifacts (dataset URLs, code, HuggingFace model) and includes a backtest, which are strengths. However, the load-bearing attribution of gains to semantic relevance and multi-level structure is currently confounded with text volume and coverage, and several evaluation components rely on LLM-generated labels without external validation.","major_comments":[{"comment":"The headline claim that 'semantic-based pairing consistently outperforms the keyword-based approach across all 12 forecasting models' is confounded with text volume and coverage. The keyword condition (FNSPID) pairs only articles with explicit company mentions, has many company-days with no news (handled by forward filling), and supplies at most one level of text. The semantic condition retrieves from ~1 million articles across four levels, resulting in far more and more consistently available text per day. No baseline matches the token count, article count, or coverage of the semantic condition. The observed gains may therefore be due to quantity or availability of text rather than semantic relevance. A volume-matched baseline is needed—e.g., keyword-paired text augmented with random or coverage-matched articles per day—to isolate the effect of semantic selection.","section":"§4.2, Table 1"},{"comment":"The multi-level ablation progressively adds text levels, so each step increases the amount of text fed to the model. The monotonic improvement shown in Figure 4 is compatible with a simple volume effect: more text (regardless of its semantic level or relevance) may improve forecasts through increased information supply. To support the claim that multi-level structure itself is beneficial, the paper should compare against adding an equivalent volume of text distributed at random, or hold total token count roughly constant across configurations (e.g., by subsampling articles per level). Without such a control, the 'complementary signals' interpretation is not established.","section":"§4.3, Figure 4"},{"comment":"The retrieval quality evaluation uses hit-rate against 'target company-level news article' labels that are themselves generated by the LLM classification pipeline described in Section 3.3. This creates an internal feedback loop: the ground truth is produced by the same type of LLM used to filter retrieved articles. The comparison between Base and Fine-Tuned embeddings is still informative, but the absolute hit-rate numbers and the claim of 'improved retrieval performance' need external validation—e.g., human-annotated relevance judgments on a sample, or at least a baseline such as random retrieval. Reporting the chance-level hit-rate would also help interpret the improvements.","section":"§5.2, Table 3"},{"comment":"The paper states that z-score normalization is applied to stock prices, but it does not state whether the normalization statistics are computed on the training split only. In time-series forecasting with a temporal split, using full-sample statistics for normalization is a form of leakage that can inflate apparent performance. The authors should explicitly state that normalization is fit on the training set and applied to validation/test, or correct the pipeline if this was not the case. This is a load-bearing detail for the forecasting evaluation.","section":"§4.1"}],"minor_comments":[{"comment":"The paper reports averages over three random seeds but omits standard deviations or confidence intervals. Given the small differences between some conditions (e.g., DLinear, FiLM in Table 4), error bars are needed to assess whether the claimed improvements are meaningful. Please add per-seed standard deviations.","section":"Tables 1 and 4"},{"comment":"The text says 'consistent improvements' across 10 companies, but COST shows +0.0%p improvement. This should be acknowledged, and significance testing (or paired tests) across companies would strengthen the claim.","section":"Table 3"},{"comment":"The LLM-as-judge evaluation compares paired texts that were generated with LLM-based summarization. The judge may favor summaries that match its own stylistic preferences. A small human-annotated sample for each criterion would increase confidence in the quality comparison.","section":"§5.5"},{"comment":"The x-axis label shows '0 10 15 20'; it would be clearer to state that N=0 means no company-level news is used, not zero retrieval size in the standard sense. Also, the caption references Figure 4 for normalization but the procedure is slightly different (across N settings); please clarify.","section":"Figure 5"},{"comment":"The backtesting experiment is a nice addition, but the portfolio construction (top 20 by predicted 3-day returns, rebalancing every 3 days) should specify transaction costs, liquidity considerations, and whether returns are gross or net. Also, the 2023 evaluation period is a single year; a multi-year backtest would be more robust.","section":"§4.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset and framework are potentially valuable contributions to the KDD community, and the public release of code and data is commendable. The main issue is the confound between pairing strategy and text volume/coverage in the central forecasting comparison; this is fixable with additional controlled ablations, but it is load-bearing and cannot be waved away. The LLM-generated evaluation labels are a secondary concern. I recommend major revision rather than rejection because the core idea is sound and the missing controls are within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nFinTexTS is a solid, reusable resource: 100 companies, five years, roughly a million news articles paired at four levels via SEC-filing-derived queries and a fine-tuned embedding model. The pipeline is credible and the public artifacts (dataset, embedding model, code) are a real contribution. The pilot forecasting study across 12 architectures is more thorough than most dataset papers, and the consistent win over the keyword baseline is suggestive.\n\nBut the central causal claim — that semantic-based and multi-level pairing drives the gains — is not actually isolated. The keyword baseline from FNSPID pairs only articles with explicit company mentions, and the authors themselves note many company-days have no news, so they forward-fill. The semantic condition retrieves from a much larger pool and supplies text every day across four levels. No baseline matches the volume or coverage of paired text. So the 'consistently outperforms' result in Table 1 is compatible with a simpler explanation: more text, more consistently available. The multi-level progression in Figure 4 has the same issue, since every added level also adds tokens. To support the paper's stated contribution, they need a volume-matched baseline — for example, keyword-based retrieval with the same number of articles per day, or a downsampled semantic condition.\n\nTwo smaller soft spots. Tables 1 and 4 report averages over three seeds but no error bars, so we can't tell whether the margins are meaningful. And the paper says z-score normalization is applied but doesn't state explicitly that it's computed on the training split only; that distinction matters for leakage. The LLM-as-judge quality table is heavily internal (LLM grading LLM summaries), but the gap between FNSPID and FinTexTS is large enough that the direction is believable.\n\nNone of this kills the paper. The dataset is valuable regardless of which mechanism causes the performance gain, and the framework is genuinely more principled than keyword matching. The volume confound undercuts the strong claim, not the resource. This deserves a serious referee — ideally one who pushes for the matched baseline and error bars before publication.\n\nFor a reading group: useful if anyone works on financial multimodal forecasting. I'd cite it if I needed a benchmark.\n\nBest,","headline":"A genuinely useful financial text–TS dataset with a sensible pairing pipeline, but the headline claim that semantic relevance beats keyword matching is confounded by text volume and needs a matched baseline.","tokens_in":16268,"tokens_out":1800,"would_cite":true,"duration_ms":17696,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic, multi-level news pairing consistently beats keyword matching for stock forecasting.","keywords":["financial forecasting","text-time-series dataset","semantic pairing","multi-level pairing","stock price prediction","embedding-based retrieval","LLM classification","multimodal time series"],"falsifier":"A matched-volume experiment where the keyword baseline receives the same number of articles per day as the semantic condition—for example by sampling all articles mentioning the company plus a random same-sector sample—would settle the causal claim; if keyword pairing then matches or beats semantic pairing, the paper's central attribution fails.","tokens_in":15527,"feed_emoji":"📈","tokens_out":5395,"duration_ms":46401,"temperature":0.7,"pith_summary":"This paper argues that the way text is paired with financial time series matters as much as the text itself. It proposes a pairing framework that uses SEC filings as company-specific context to retrieve news articles by meaning rather than by explicit company mentions, and classifies each article into one of four levels—macro, sector, related-company, and target-company—so a stock is paired with a structured, multi-level text summary each day. On a new dataset covering 100 companies over five years, the authors report that semantic pairing outperforms keyword pairing across all 12 forecasting models under both MSE and MAE, and that adding levels progressively improves accuracy. If correct, this gives the time-series and finance communities a better-aligned resource for text-based forecasting and a reusable recipe for building such datasets.","feed_headline":"Semantic news pairing beats keyword matching for stock forecasts","feed_subtitle":"A four-level pairing strategy improves forecast accuracy across all 12 models tested on a new 1-million-article dataset.","key_machinery":"The central mechanism is the pairing framework itself, built from four components: (1) an LLM-based SEC filing parser that extracts five categories of company context (overview, strategy, governance/risks, financials, recent events) and forward-fills them to daily level; (2) an LLM-based news classifier that first assigns articles to macro, sector, or company level and then refines sector-level articles into specific industry sectors and company-level articles into target vs. related company; (3) an embedding-based retrieval module that uses parsed filing components as queries and a sector-fine-tuned embedding model to retrieve the top-N relevant company-level articles per day; and (4) an LL","core_discovery":"The paper's central claim is that semantic-based and multi-level pairing of news articles with stock price time series yields consistently better stock-forecasting performance than the standard keyword-based approach. Using SEC filing content as a query, a fine-tuned embedding model retrieves company-relevant news even when the company name is absent; an LLM classifier then assigns each article to macro, sector, related-company, or target-company level, and an LLM summarizer condenses each level into distinct event categories. In a pilot study across 12 time-series models, the semantic condition outperforms the keyword condition on every model under both evaluation metrics, and progressive a","pith_inferences":["A controlled comparison that matches per-day text volume between the keyword and semantic conditions would separate the effect of semantic relevance from the effect of simply having more text; the paper does not provide such an ablation.","The multi-level text structure could be exploited by level-specific attention or by predicting each level's marginal contribution, which the current pilot does not attempt.","The framework's reliance on SEC filings and U.S.-listed companies means the recipe may not transfer directly to markets with less standardized filings; extending it to other geographies would require an analogous authoritative context source.","The LLM-as-judge quality scores suggest the pairing improves coverage and diversity, so the approach could also be used as a data-cleaning step for other financial-text tasks such as sentiment analysis or event extraction."],"forward_implications":["Across 12 time-series architectures, the semantic condition lowers MSE and MAE relative to keyword pairing for every model—a direct result reported in the paper.","Adding macro, then sector, then related-company, then target-company text progressively lowers normalized MSE, and the same holds when levels are added in reverse order.","Backtesting with a simple equal-weighted top-20 portfolio gives higher cumulative return and Sharpe ratio than keyword-paired or text-free alternatives.","Fine-tuning the embedding model on sector classification improves retrieval hit-rate by 1.4–5.9 percentage points across ten sampled companies.","Applying the framework to a proprietary, well-curated news source improves forecasting over public news on most of the 12 models."],"fun_headline_variants":["Four-level news pairing beats keyword-only for stock forecasts","Semantic news pairing lifts stock forecasting across 12 models","SEC-filing context pulls better news for stock prediction","Multi-level news pairing wins over keyword matching","News beyond company name: semantic pairing for stock data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The experiments assume the improved forecasts stem from semantic relevance and level structure rather than from the semantic condition's larger and more consistent daily text volume, since the keyword baseline leaves many days with no paired news and no matching-volume control is run.","fun_headline_variants_meta":{"raw":{"variants":["Four-level news pairing beats keyword-only for stock forecasts","Semantic news pairing lifts stock forecasting across 12 models","SEC-filing context pulls better news for stock prediction","Multi-level news pairing wins over keyword matching","News beyond company name: semantic pairing for stock data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1085,"prompt_tokens":789,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":235}},"tokens_in":533,"tokens_out":296,"duration_ms":3143,"temperature":1.0,"reasoning_tokens":235,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:15:57.235031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched-volume experiment where the keyword baseline receives the same number of articles per day as the semantic condition—for example by sampling all articles mentioning the company plus a random same-sector sample—would settle the causal claim; if keyword pairing then matches or beats semantic pairing, the paper's central attribution fails.","supporting_citations":[],"review_version":1}