{"id":"8b1b9cf6-b496-435f-af7e-c5efc9e49664","arxiv_id":"2504.15683","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"FinTextSim, a sentence transformer fine-tuned on keyword-labeled financial sentences, is claimed to improve BERTopic topic modeling on S&P 500 10-K Item 7/7A text, but the evaluation is largely circular.","lead":"This paper fine-tunes a sentence embedding model, FinTextSim, on 10-K financial text and reports that it makes BERTopic topic clusters much more coherent and separated than the general-purpose all-MiniLM-L6-v2 model. The headline gains rest on an evaluation metric built from the same keyword list used to train and guide the model, so the results are largely circular.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FinTextSim vs. AM comparison is confounded by base model: FinTextSim is fine-tuned ModernBERT while AM is all-MiniLM-L6-v2; without an untuned ModernBERT baseline, the 81% and 100% improvements cannot be attributed to financial fine-tuning.","rationale":"The reader's central concern about circular topic-precision is valid and contributes to the rejection, but the more load-bearing flaw is the uncontrolled base model. The paper explicitly aims to 'isolate the effect of the selected sentence-transformer' (Section 1), yet compares a fine-tuned ModernBERT against all-MiniLM-L6-v2. Since ModernBERT is a recently released, larger, and more capable encoder, any performance gap could arise from the architecture alone. This confound undermines not only the topic-precision-weighted BERTopic results but also the unweighted similarity metrics in Tables 5 and 6, which the reader treated as non-circular evidence in the paper's favor. For example, FinTextSim's unweighted intratopic similarity of 0.925 vs AM's 0.661 might simply reflect ModernBERT's stronger embeddings. The paper does not provide an untuned ModernBERT baseline, and no supplementary material or code is released to check this. The concern is directly testable: add a ModernBERT sentence-embedding baseline and/or fine-tune AM with the same data and loss. If the advantage disappears, the paper's central causal claim—that domain-specific fine-tuning is necessary—is unsupported. Because the paper's stated contribution and conclusion depend on this causal interpretation, the REJECT verdict stands. This stress-test agrees with the reader's rejection but routes it through an internal-validity flaw in the experimental design, which is distinct from (and more fundamental than) the metric circularity.","tokens_in":25437,"tokens_out":12650,"duration_ms":107035,"concrete_test":"Add an off-the-shelf, non-fine-tuned ModernBERT sentence-embedding baseline (e.g., a published sentence-transformers checkpoint) and fine-tune AM on the same 180k labeled sentences with identical loss and hyperparameters; then rerun Tables 2, 5, and 6. If the untuned ModernBERT matches FinTextSim's performance, or if fine-tuned AM closes the gap, the reported advantage is an architecture effect rather than finance-specific fine-tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that finance-specific fine-tuning boosts clustering (81% higher intratopic, 100% lower intertopic similarity) and that BERTopic only forms clear economic topics with FinTextSim—is built on a comparison that does not control for base architecture. FinTextSim is a fine-tuned ModernBERT (Warner et al., 2024) with mean pooling and normalization (§3.3); AM is all-MiniLM-L6-v2, an older, much smaller distilled model. The authors state they 'isolate the effect of the selected sentence-transformer' by keeping all other model parameters identical (Section 1, §3.4), but the two models differ not only in fine-tuning but in architecture, parameter count, training data, and release date. Consequently, the observed differences in intratopic/intertopic similarity (Tables 2, 5, 6), outlier counts, and downstream BERTopic quality could be due to ModernBERT's general-purpose superiority rather than to the financial fine-tuning that the paper claims to demonstrate. No baseline with an untuned ModernBERT sentence embedding model is reported, and no ablation fine-tunes AM on the same financial data. This confound affects even the non-circular metrics (unweighted similarity improvements and qualitative examples), so it is more foundational than the metric circularity noted by the reader. Without controlling for base model, the central claim 'domain-specific fine-tuning is crucial' (Conclusion) is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinTextSim, a sentence-transformer model fine-tuned on financial text (based on ModernBERT), and evaluates BERTopic topic modeling on Item 7 and Item 7A of 10-K filings from S&P 500 companies. The authors compare FinTextSim against all-MiniLM-L6-v2 (AM), reporting that FinTextSim increases intratopic similarity by 81% and reduces intertopic similarity by 100%, and that BERTopic forms clear economic topic clusters only when using FinTextSim embeddings. The evaluation relies on a topic-precision metric (Eq. 1) computed from the same 14-topic keyword list used to create FinTextSim's training labels, to select refined sentences for BERTopic, and to seed BERTopic's c-tfidf with a multiplier of 50.","tokens_in":25734,"tokens_out":3746,"duration_ms":34730,"significance":"If the claims were sound, the paper would make a useful applied contribution by showing that domain-specific fine-tuning of sentence transformers materially improves topic modeling on financial disclosures. The work addresses a relevant gap in applying neural topic models to MD&A text, and it includes qualitative examples and a detailed pipeline. However, the central quantitative evidence is undermined by two load-bearing problems: the main evaluation metric is circular with respect to the training and seeding procedure, and the model comparison conflates architectural differences with the effect of fine-tuning. These issues affect the paper's main conclusions, not just peripheral details.","major_comments":[{"comment":"The topic-precision metric is defined from the same 14-topic keyword list that is used to create FinTextSim's training labels (§3.3), to select refined sentences (§3.4), and to seed BERTopic's c-tfidf with a multiplier of 50 (§3.4). Consequently, the perfect topic-precision score of 1.0 in Table 3 is largely by construction: the model is rewarded for reproducing the authors' own taxonomy, not for discovering independently meaningful financial topics. This circularity is load-bearing because topic-precision is then used to weight NPMI coherence and intra/intertopic similarity in Tables 4-6. The unweighted NPMI coherence reported in parentheses in Table 4 actually favors AM (0.341 vs. 0.279 for sentences; 0.376 vs. 0.301 for refined sentences), so the standard metric contradicts the paper's topic-quality conclusion. The authors' dismissal of raw coherence in §4.2 is not justified by evidence, and it cannot compensate for the fact that the reported advantage vanishes when the evaluation is decoupled from the keyword list.","section":"§3.5.3, Eq. (1); Tables 3 and 4"},{"comment":"The comparison between FinTextSim and AM is confounded by the base model. FinTextSim is fine-tuned from ModernBERT, a recent, larger encoder model, while AM is all-MiniLM-L6-v2, an older and substantially smaller distilled model. The statement in §1 and §3.4 that 'keeping all other model parameters identical' isolates the effect of the sentence transformer is inaccurate, because the models differ in architecture, parameter count, training data, and release date. Without an untuned ModernBERT baseline or a fine-tuned AM model, the reported improvements in intratopic similarity, intertopic similarity, and outlier counts cannot be attributed to financial fine-tuning rather than to general superiority of the base architecture. This confound affects even the non-circular similarity metrics, so the central conclusion that 'domain-specific fine-tuning is crucial' is not established.","section":"§3.3 and §3.4; Tables 2, 5, and 6"},{"comment":"The use of seed words with a multiplier of 50 in c-tfidf directly injects the evaluation keyword list into BERTopic's topic-word extraction. Since topic-precision is computed by matching topic words against that same list, this step further guarantees high topic-precision for FinTextSim, independently of the embedding quality. The paper does not report any sensitivity analysis for the seed multiplier or for the refined-sentence selection rule, both of which are free parameters that could drive the reported results. A comparison without seed words, or with an independently constructed evaluation vocabulary, is needed to support the claim that FinTextSim improves topic quality.","section":"§3.4, c-tfidf seed words"}],"minor_comments":[{"comment":"The abstract claims that FinTextSim 'reduces intertopic similarity by 100%', but Table 2 reports 0.0002 vs. 0.4647, which is a 99.96% reduction. In addition, §4.1 states the FinTextSim intertopic similarity is '0.002', while Table 2 reports 0.0002; this inconsistency should be corrected.","section":"Abstract and §4.1, Table 2"},{"comment":"Figure A.9 is captioned 'Wordcloud - FinTextSim - Refined Sentences', but given the sequence of Figures A.6-A.9, it appears the caption should read 'AM - Refined Sentences'.","section":"Appendix A, Figure A.9 caption"},{"comment":"The fine-tuning procedure for FinTextSim does not report several hyperparameters needed for reproducibility, such as the number of training epochs, learning rate, and the exact ModernBERT variant used. The adaptive circle loss description also omits the pairing strategy for positive and negative sentence pairs.","section":"§3.3"},{"comment":"The sentence 'We find that BERTopic is highly on financial text when combined with FinTextSim' appears incomplete; it should probably read 'highly effective on financial text'.","section":"§4.4"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claims are not supported by the evidence as presented. The topic-precision metric is circular, the model comparison is confounded by base architecture, and the only non-circular standard metric (raw NPMI) favors the baseline. Even a substantial revision with an untuned ModernBERT baseline and an independent evaluation protocol would essentially change the paper's focus; I do not see a path within the current manuscript's framework to support the title-level conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, but not a good paper as-is. The idea is straightforward: fine-tune a sentence transformer on 10-K text and use it with BERTopic on Item 7/7A. That is a legitimate research question, and the authors did real work: 1,439 S&P 500 filings, a 180k-sentence labeled training set, a ModernBERT-based model fine-tuned with adaptive circle loss, and a BERTopic pipeline. The unweighted intra/inter-topic similarity numbers (about 40% higher intra, 84% lower inter at sentence level) are at least non-circular evidence that FinTextSim clusters the test sentences more tightly. The three wordcloud examples are illustrative, though cherry-picked.\n\nThe soft spots are serious. First, the comparison is confounded. FinTextSim is built on ModernBERT; the baseline all-MiniLM-L6-v2 is an older, much smaller distilled model. The paper says it 'isolates the effect of the selected sentence-transformer' by keeping other BERTopic parameters identical, but that only isolates the embedding model as a whole—not the fine-tuning. Without an untuned ModernBERT baseline or an AM model fine-tuned on the same data, you cannot attribute the 81% and 100% improvements to financial fine-tuning. Architectural general-purpose superiority is a live alternative explanation. This confound affects even the non-circular metrics, so it is the load-bearing flaw.\n\nSecond, the topic-precision metric is built from the same 14-topic keyword list that labels FinTextSim's training data, selects the refined sentences, and seeds BERTopic's c-tfidf with a multiplier of 50. Scoring 1.0 on that metric is close to by construction. The authors do report raw NPMI coherence, but it favors AM—and they dismiss it with an untested story about financial vocabulary and outliers. That dismissal needs evidence, not assertion.\n\nThird, no code, no data, no error bars, and the abstract's '100% reduction' is an overstatement even against their own Table 2 (it is ~99.96%). Minor: the text says 0.002 and the table says 0.0002 for FinTextSim intertopic similarity.\n\nThe central claim—that BERTopic only forms clear economic topics with FinTextSim, and that domain fine-tuning is 'crucial'—is not established. But the paper is not incoherent or junk; the research question is sensible, the dataset is real, and an ablation that controls for base model could change the story. I would send it to peer review expecting heavy revision: add an untuned ModernBERT baseline, fine-tune AM on the same financial sentences, and report topic precision on an external or human-annotated evaluation rather than the training keyword list.","headline":"A sensible domain-tuned embedding idea with real dataset work, but the FinTextSim-vs-AM comparison is confounded by base architecture and the headline metric is circular; worth a serious referee, not desk rejection, but it needs major revision.","tokens_in":26374,"tokens_out":3220,"would_cite":false,"duration_ms":28585,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERTopic, a neural topic model, produces clear, distinct economic topic clusters from Item 7 and Item 7A of S&P 500 10-K filings only when paired with FinTextSim, a sentence-transformer fine-tuned on financial text; FinTextSim raises…","keywords":["FinTextSim","BERTopic","topic modeling","10-K filings","financial text analysis","sentence transformers","MD&A","topic-precision"],"falsifier":"Re-run the BERTopic comparison with topic-precision computed from a held-out keyword list that played no role in training or topic seeding, or have independent financial analysts label the inferred topics; alternatively, feed FinTextSim embeddings of an out-of-domain financial corpus such as analyst reports or earnings-call transcripts into BERTopic and check whether the clusters stay as clean as on 10-K text. If the gap over the baseline narrows or vanishes, the reported advantage comes from the shared keyword scheme rather than the embeddings.","tokens_in":25080,"feed_emoji":"📊","tokens_out":6941,"duration_ms":52570,"temperature":0.7,"pith_summary":"This paper claims that domain-specific fine-tuning of sentence transformers is required for reliable topic modeling of financial disclosures. It introduces FinTextSim, a sentence-transformer fine-tuned on 180,435 keyword-labeled sentences from 10-K Item 7/Item 7A and other annual and sustainability reports, and reports that BERTopic only forms clear, distinct economic topic clusters when fed FinTextSim's embeddings rather than those of the widely used all-MiniLM-L6-v2. On the authors' 14-topic evaluation, FinTextSim achieves perfect topic-precision of 1.0 while the baseline misses up to ten of the fourteen topics, and it improves intratopic similarity by 81% while reducing intertopic similarity by 100%. The authors also report that raw NPMI coherence favors the baseline, and argue that coherence measures mislead for financial text because domain terms rarely co-occur in small windows. A sympathetic reader would take away that embedding choice drives topic quality more than any other BERTopic component.","feed_headline":"Finetuned embeddings give BERTopic all 14 financial topics","feed_subtitle":"General embeddings miss 10 of 14 10-K topics; the finetuned model finds all, the paper reports.","key_machinery":"The load-bearing object is FinTextSim, a sentence-transformer built on ModernBERT with mean pooling and normalization, trained with adaptive circle loss, a curriculum variant of circle loss that shifts margin and scale during training. Its training data are 180,435 sentences labeled by a 14-topic financial keyword list derived from prior accounting and economics literature, augmented with Energy, ESG, and COVID-19 topics. The same keyword list labels the training data, seeds BERTopic's topic-word extraction with a multiplier of 50, and defines topic-precision (Equation 1), the weighting factor multiplied into NPMI coherence and similarity scores and divided out of intertopic similarity. The BERTopic pipeline uses UMAP to ten components, HDBSCAN with minimum cluster size 1,250, CountVectorizer with the Loughran-McDonald stopword list, and c-tfidf weighting. The paper's comparisons hold every BERTopic component fixed except the embedding model.","core_discovery":"The central claim is that the quality of BERTopic's topics on Item 7 and Item 7A of S&P 500 10-K filings is determined by the embedding model, and that general-purpose embeddings fail at this task while FinTextSim succeeds. With FinTextSim, all fourteen economic topics in the authors' scheme are recovered with topic-precision 1.0 on both sentence and refined-sentence input; intratopic similarity reaches 0.9972 versus 0.5498 for all-MiniLM-L6-v2, and intertopic similarity drops to 0.0002 versus 0.4647. The paper also reports that the baseline model leaves 226,605 of 687,959 sentences as BERTopic outliers, versus 184,470 for FinTextSim. Notably, raw NPMI coherence is higher for the baseline in every configuration; the authors attribute this to the many outlier sentences simplifying topic compression and to financial vocabulary not co-occurring within sliding windows, and conclude that coherence without domain-weighting is insufficient for evaluating financial topics.","pith_inferences":["The near-total collapse of intertopic similarity from 0.4647 to 0.0002 suggests FinTextSim's embedding space may be collapsed onto the 14 training labels; a testable consequence is that topics outside the keyword scheme would be poorly separated, which a reader could check with a held-out topic.","Because the same keyword list trains the model and defines the evaluation, the perfect 1.0 topic-precision should be read as an upper bound; an independent annotation study would be the natural way to confirm the paper's main claim.","The reported 19% reduction in BERTopic outliers suggests the fine-tuned geometry is denser around the labeled topics; an interesting extension is measuring whether that density transfers to unlabeled financial genres like conference-call transcripts."],"forward_implications":["FinTextSim's reported gains imply that embedding choice, not clustering or weighting, is the main lever on topic quality for 10-K MD&A text.","Downstream tasks that depend on topic separation, such as aspect-based sentiment analysis of filings, business valuation, and stock-price prediction, would inherit cleaner inputs if the reported separation holds.","Raw NPMI coherence would no longer be treated as sufficient evidence of topic quality in finance; precision-weighted metrics would be needed to avoid misleading rankings.","If the approach generalizes beyond Item 7/7A, the same fine-tuning recipe could be applied to other 10-K sections and to other regulated disclosure documents."],"supporting_citations":[{"why":"introduces BERTopic, the five-step embedding-to-c-tfidf pipeline that the paper evaluates with both embedding models.","marker":"(Grootendorst, 2022)"},{"why":"supplies the original 11-topic economic anchorword list for 10-K/10-Q disclosures that grows into the paper's 14-topic keyword scheme.","marker":"(Li, 2010a)"},{"why":"augments the anchorword list into a topic-word list via Word2Vec on MD&A sections, the basis of FinTextSim's labeling vocabulary.","marker":"(Fengler and Phan, 2023)"},{"why":"defines the sentence-transformer training approach (siamese/triplet structure, mean pooling) that FinTextSim adapts.","marker":"(Reimers and Gurevych, 2019)"},{"why":"provides ModernBERT, the encoder-only base model FinTextSim is fine-tuned from.","marker":"(Warner et al., 2024)"},{"why":"introduces circle loss, whose adaptive variant is FinTextSim's training objective.","marker":"(Sun et al., 2020)"},{"why":"provides HDBSCAN, the soft-clustering algorithm that assigns BERTopic's topics and outliers.","marker":"(McInnes and Healy, 2017)"},{"why":"supplies the window-based NPMI coherence measure used for topic quality evaluation.","marker":"(Röder et al., 2015)"}],"fun_headline_variants":["FinTextSim embeddings give BERTopic perfect 14-topic recovery","Domain-specific vectors turn BERTopic into a financial topic finder","General sentence embeddings fail 10-K topics; FinTextSim fixes it","FinTextSim: 81% tighter topics, 100% less overlap for BERTopic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that topic-precision, computed from the same 14-topic keyword list used to label FinTextSim's training sentences, seed BERTopic's topic words, and score the results, is a valid measure of financial topic quality; if that metric does not reflect real topic quality, the perfect 1.0 score collapses because the model is being graded on labels it was built and guided to reproduce.","fun_headline_variants_meta":{"raw":{"variants":["FinTextSim embeddings give BERTopic perfect 14-topic recovery","Domain-specific vectors turn BERTopic into a financial topic finder","General sentence embeddings fail 10-K topics; FinTextSim fixes it","FinTextSim: 81% tighter topics, 100% less overlap for BERTopic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001,"raw_usage":{"total_tokens":4284,"prompt_tokens":1046,"completion_tokens":3238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":3155}},"tokens_in":662,"tokens_out":3238,"duration_ms":20383,"temperature":1.0,"reasoning_tokens":3155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:20:30.430700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the BERTopic comparison with topic-precision computed from a held-out keyword list that played no role in training or topic seeding, or have independent financial analysts label the inferred topics; alternatively, feed FinTextSim embeddings of an out-of-domain financial corpus such as analyst reports or earnings-call transcripts into BERTopic and check whether the clusters stay as clean as on 10-K text. If the gap over the baseline narrows or vanishes, the reported advantage comes from the shared keyword scheme rather than the embeddings.","supporting_citations":[{"cited_title":", author Phan, M.T","cited_arxiv_id":null,"evidence_quote":"augments the anchorword list into a topic-word list via Word2Vec on MD&A sections, the basis of FinTextSim's labeling vocabulary."},{"cited_title":", author Cheng, C","cited_arxiv_id":null,"evidence_quote":"introduces circle loss, whose adaptive variant is FinTextSim's training objective."}],"review_version":1}