Pith. sign in

REVIEW 4 major objections 7 minor 50 references

FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A compact bidirectional encoder pretrained on 32 billion tokens of Chinese financial text outperforms much larger LLMs on financial classification and tops general-purpose embedders on financial retrieval.

desk verdict Credible, well-controlled case for a Chinese financial BERT, but the headline margins rest on private test sets and test-set-based selection, so the exact numbers need a decontamination check before being taken at face value. read the letter →

arxiv 2506.06335 v2 pith:J3U6C5UL submitted 2025-05-31 cs.IR cs.AIcs.CEcs.CL

Xuan Xu , Fufang Wen , Beilin Chu , Zhibing Fu , Qinhong Lin , Jiaqi Liu , Binjie Fei , Yu Li
show 2 more authors
Linna Zhou Zhongliang Yang
This is my paper
classification cs.IRcs.AIcs.CEcs.CL
keywords FinBERT2financialNLPdomain-specificpretrainingdenseretrievaltopicmodelingChinesecorpuscontrastivelearningRAG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a relatively small, encoder-only language model can still be the right tool for financial NLP whenever the job is classification, retrieval, or feature extraction. It introduces FinBERT2, a RoBERTa-style model pretrained on 32 billion tokens of Chinese financial text, and shows that fine-tuned versions beat both older financial BERTs and much larger LLMs such as GPT-4 and Qwen2 on five financial labeling tasks. The same backbone, trained with contrastive learning, outperforms general-purpose retrieval embedders on five financial retrieval benchmarks, and a topic-modeling pipeline built on its embeddings produces more coherent financial topics. The practical stake is cost and accuracy: if true, enterprises can deploy a small local encoder for discriminative work and reserve LLMs for generation.

What carries the argument

The load-bearing machinery is Fin-Corpus, a 32B-token Chinese financial pretraining corpus assembled from analyst reports, company announcements, and the Duxiaoman open-source news corpus; a finance-customized WordPiece tokenizer that adds about 14,000 domain terms and company names; and RoBERTa-style masked-language-modeling pretraining with dynamic and whole-word masking and no next-sentence prediction. Downstream, Fin-Labelers are produced by task-specific fine-tuning, Fin-Retrievers by contrastive learning with InfoNCE, mined hard negatives, and a 2:8 positive-to-negative ratio, and Fin-TopicModel by UMAP, HDBSCAN clustering, and c-TF-IDF on FinBERT2 embeddings. The argument is that each stage transfers domain knowledge from the corpus into the encoder, and the ablations tie gains to pretraining data volume and vocabulary expansion.

What would settle it

Compute n-gram or embedding-level overlap between each downstream test set and Fin-Corpus, remove near-duplicate test examples, and re-run the Fin-Labeler and Fin-Retriever benchmarks; if the reported margins over baselines shrink or disappear, the gains reflect memorization of pretraining data rather than domain understanding.

Watch

Extended reading notes

Core claim

FinBERT2 is a bidirectional encoder continued-pretrained from Chinese-RoBERTa-wwm-ext on Fin-Corpus, a curated 32B-token collection of analyst reports, company announcements, and financial news, with a finance-expanded WordPiece vocabulary. The paper's central claim is that this domain-adapted backbone, after task-specific fine-tuning, outperforms other (Fin)BERT variants by 0.4%-3.3% and leading LLMs by 9.7%-12.3% on average across five financial classification tasks; that contrastively fine-tuned Fin-Retrievers beat BGE-base-zh by 6.8% and a proprietary text-embedding-3-large by 4.2% on average across five financial retrieval tasks; and that the Fin-TopicModel built on FinBERT2 embeddings yields better clustering and topic representation for financial titles. The paper presents this as evidence that small specialized encoders remain a strong default for discriminative and feature-based financial NLP in the LLM era, complementing rather than being replaced by LLMs.

Load-bearing premise

The evaluation assumes that the downstream test sets contain no near-duplicates of pretraining texts, because the test data and the 32B-token corpus come from the same text sources and no decontamination step is reported.

Editorial extensions

If this is right

  • Fine-tuned FinBERT2 can serve as a low-cost, high-accuracy labeler for large-scale financial text, replacing LLM prompting in production.
  • Fin-Retriever provides a finance-specific dense retriever for RAG systems, improving recall on reports, announcements, and indicators over general embedders.
  • Fin-TopicModel enables unsupervised organization of large financial-title corpora with better coherence and lower outlier rates.
  • The base variant offers most of the benefit of the large variant at lower cost, making it a practical default for deployment.
  • Ablations show pretraining data volume and domain vocabulary are the main drivers of downstream gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains survive a decontamination check, the same recipe — large curated domain corpus, domain tokenizer, continued pretraining — could transfer to other high-jargon verticals such as legal, biomedical, or technical support text.
  • The paper compares LLMs via few-shot prompting only; the fairer competitive test for the 'LLM era' claim would be fine-tuning a similarly sized or larger LLM on the same five tasks, which the paper does not report.
  • The 2:8 positive-to-negative ratio and hard-negative mining protocol for Fin-Retriever is a reusable recipe for building domain retrievers from any encoder backbone.
  • The Fin-TopicModel evaluation leans on LLM-as-judge scores; a human-annotated topic coherence study would verify that the LLM preference tracks actual usability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces FinBERT2, a RoBERTa-style bidirectional encoder pretrained on a 32B-token Chinese financial corpus composed of analyst reports, company announcements, and financial news. It also describes Fin-Tokenizer (an expanded financial vocabulary), Fin-Labelers (task-specific fine-tuned classifiers), Fin-Retrievers (contrastive fine-tuned dense retrievers), and Fin-TopicModel (a BERTopic-style pipeline). The central claim is that FinBERT2 outperforms existing (Fin)BERT variants by 0.4%–3.3% and leading LLMs by 9.7%–12.3% on five financial classification tasks, and that Fin-Retrievers surpass open-source and proprietary embedders on five financial retrieval tasks. The paper includes ablation studies on pretraining data volume and vocabulary expansion, and it includes a same-data fine-tuning control against BGE-base-zh. The evaluation rests on privately constructed test sets: five Fin-Labeler tasks and FIR-Bench are extracted from financial terminal systems.

Significance. If the central claims hold, the paper provides a strong practical result: small specialized encoders remain competitive or superior for discriminative and feature-based financial NLP, and they can serve as efficient backbones for RAG and topic modeling. The manuscript has several genuine strengths: it uses the same fine-tuning protocol for BERT-based baselines, includes a same-data fine-tuning control against BGE in Sec. 4.4.3, reports scaling ablations over pretraining corpus volume (Tables 8–9), and evaluates on a multi-task suite. However, the headline comparisons with LLMs and retrieval baselines are only meaningful if the downstream test sets are free of overlap with the pretraining corpus. The paper does not describe any decontamination or overlap analysis, and the data descriptions in Secs. 3.2.1 and 3.3.1/3.4.3 show the same text sources (analyst reports, announcements, news) on both sides. This is the main load-bearing issue; it can be addressed with additional experiments, but as it stands it limits the confidence in the reported margins.

major comments (4)
  1. [Sec. 3.2.1 vs Sec. 3.3.1/3.4.3; Tables 3–4] The pretraining corpus (Sec. 3.2.1: analyst reports, company announcements, financial news) and the downstream evaluation sets (Sec. 3.3.1: IC/MSC/NER from financial terminal systems; Sec. 3.4.3: FIR-Bench with research reports and announcements) are drawn from the same kinds of texts, and the paper does not report any decontamination step. If test examples or near-duplicates appeared in the 32B pretraining tokens, the reported gains over LLMs and over BGE (especially the large margins on Research Reports and Announcements in Table 4 and Table 11) would partly reflect memorization rather than domain learning. The authors should provide an explicit overlap analysis (e.g., exact and near-duplicate n-gram overlap between pretraining data and each test set), describe any deduplication performed, and ideally hold out the test sources from pretraining. This is load-bearing for the paper's central claim.
  2. [Sec. 4.1, paragraph 1] The paper states: 'Model performance was assessed throughout the training process, with the optimal test results being recorded.' This is a form of test-set-based checkpoint selection. If the best checkpoint on the test set is selected for every model, the reported numbers may be optimistically biased; if the procedure is not applied identically to all baselines, the comparisons in Table 3 are unfair. The authors should specify whether a validation set was used for checkpoint selection, and confirm that the same selection rule was applied to every BERT-based baseline.
  3. [Sec. 4.1, Table 3] The comparison against LLMs is between zero-shot/few-shot prompted LLMs and fine-tuned BERT models, not between fine-tuned models. The abstract and introduction state that Fin-Labelers 'outperform leading LLMs by 9.7%–12.3%', which conflates architectural advantage with the absence of LLM fine-tuning. The paper should either temper the claim or include fine-tuned LLM baselines; at minimum, the caption and Sec. 4.1 should explicitly state that the LLM numbers are zero-shot/few-shot with majority voting, consistent with the prompt description in Appendix A.
  4. [Sec. 4.4.3, Table 11] The same-data fine-tuning control against BGE-base-zh addresses whether the retrieval advantage comes from fine-tuning budget, but it does not address pretraining leakage. BGE-base-zh was not pretrained on the financial corpus, so the large remaining margin on Research Reports (R@10 0.987 vs 0.965) and Announcements (R@20 0.642 vs 0.442) is exactly where corpus overlap is most plausible. This control should be accompanied by an overlap analysis between FIR-Bench documents and the Fin-Corpus.
minor comments (7)
  1. [Sec. 1, footnote 2] The remark that BERT is 'now seen as a Tiny LM' is informal; consider replacing it with a precise parameter-based definition or removing it.
  2. [Sec. 3.2.2] The description of the quality-filter classifier is under-specified: the paper reports 'over 99% accuracy' on a 4K-instance set but does not state the validation metric or variance; adding precision/recall for the two classes would be more informative.
  3. [Sec. 3.3.1, Table 1] The MSC(4 labels) row lists '1280/400' train/test samples; this is much smaller than other classification tasks. The implication for the reported margins should be discussed, especially because the paper highlights this task as a key differentiator.
  4. [Sec. 4.2.1] The text refers to 'text-embedding-ada-002-large', but the actual model name in Table 4 is 'text-embedding-3-large'; please unify the naming.
  5. [Sec. 4.2.2, Table 5] The paper reports that Fin-Retriever 'approaches' text-embedding-3-large (0.781) with average 0.752, but the gap is 2.9 points; consider using a more neutral phrasing.
  6. [Sec. 4.3] The topic-modeling evaluation relies on LLM scoring (Qwen-max) for coherence, conciseness, and informativity; the paper should report the prompt template or scoring rubric, as it does for the retrieval and classification tasks.
  7. [Appendix C, Tables 8–11] The ablation tables are only presented in the appendix, but Sec. 4.4 references them as if in the main text; consider moving them to the main body or clearly labeling them as appendix tables in the narrative.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: FinBERT2's margins come from a self-contained empirical pipeline with external baselines; only minor self-citation and private-data validity caveats prevent a 0.

full rationale

The paper's load-bearing claims are empirical comparisons on held-out test portions of five private classification tasks, FIR-Bench retrieval tasks, and a C-MTEB subset, against external baselines (vanilla Chinese BERTs, FinBERT1, Mengzi-fin, BGE, BCE, OpenAI embedders, and three LLMs) all evaluated with identical fine-tuning or prompting protocols. No parameter is fit to the test set and then reported as a prediction; the only fitted quantities are model weights, and their evaluation is on separate held-out sets. The self-citation to the authors' own FinBERT1 [10] appears as a predecessor and a baseline, not as a load-bearing justification for FinBERT2's improvements, so it does not make the central derivation circular. The Fin-Retriever's hard-negative mining with FinBERT2 and the use of Qwen2.5-72B for data filtering and subjective topic scoring are internal engineering choices, not reductions of the evaluation to the training signal. The main caveat—private downstream sets may overlap in genre with the pretraining corpus, and no decontamination is described—is a benchmark-validity risk, not a circularity of the kind where a result equals its input by construction. Because the empirical evaluation is self-contained and includes external controls, the circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. FinBERT2 is a trained artifact, not an explanatory entity; the paper does not introduce a new force, particle, dimension, or conserved quantity.

free parameters (5)
  • corpus quality thresholds = score >8 high quality, <4 low quality
    Qwen2.5-72B ratings on a 100K-token sample were converted into a binary classifier; the cutoffs are hand-chosen and determine the 15% of pretraining data removed (§3.2.2).
  • positive:negative contrastive ratio = 2:8, with 10 or 15 negatives per query
    Authors report 'after extensive experimentation' this ratio delivered the best results (§3.4.1); the claimed retrieval margins depend on this tuned hyperparameter.
  • sliding window size and overlap = 400 words, 20-word overlap
    Chosen for optimal long-document retrieval performance (§3.4.2); directly affects FIR-Bench scores.
  • fine-tuning epochs and learning rate for labelers = lr 5e-5, epochs 1 for sequence and 5 for token tasks
    Fixed across BERT baselines for fairness, but not shown to be optimal for every baseline (§3.3.2).
  • topic-modeling hyperparameters = UMAP n_neighbors=15, min_dist=0.0; HDBSCAN min_cluster_size=2, min_samples=1
    Chosen to minimize outliers; topic quality scores are sensitive to these choices (§3.5.5).
assumptions (5)
  • domain assumption The LLM judge Qwen2.5-72B-Instruct provides reliable quality labels for corpus filtering and positive/negative filtering.
    Used in §3.2.2 and §3.4.1 without external validation of the labels.
  • domain assumption Financial terminal annotations and CITIC industry labels are the correct ground truth.
    The five classification datasets are annotated in-house; annotation quality and inter-annotator agreement are not reported (§3.3.1).
  • domain assumption No leakage exists between the 32B-token pretraining corpus and downstream test sets.
    Pretraining uses analyst reports, announcements, and news, the same genres as the test tasks; no decontamination is described (§3.2.1, §3.3.1, §3.4.3).
  • domain assumption Unsupervised metrics including LLM scoring, Silhouette, Calinski-Harabasz, and Davies-Bouldin measure topic quality validly.
    No human evaluation or labeled topics is provided for Fin-TopicModel (§3.5.2, Appendix B).
  • standard math InfoNCE contrastive loss and UMAP/HDBSCAN behave as standard in this setting.
    These are external algorithms; correctness is assumed from prior literature (§3.4.2, §3.5.5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models." pith.science (2026). https://pith.science/paper/J3U6C5UL

@misc{pith2026250606335,
  author       = {Pith},
  title        = {Pith review of: FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3U6C5UL}},
  note         = {Machine review of arXiv:2506.06335}
}
read the original abstract

In natural language processing (NLP), the focus has shifted from encoder-only tiny language models like BERT to decoder-only large language models(LLMs) such as GPT-3. However, LLMs' practical application in the financial sector has revealed three limitations: (1) LLMs often perform worse than fine-tuned BERT on discriminative tasks despite costing much higher computational resources, such as market sentiment analysis in financial reports; (2) Application on generative tasks heavily relies on retrieval augmented generation (RAG) methods to provide current and specialized information, with general retrievers showing suboptimal performance on domain-specific retrieval tasks; (3) There are additional inadequacies in other feature-based scenarios, such as topic modeling. We introduce FinBERT2, a specialized bidirectional encoder pretrained on a high-quality, financial-specific corpus of 32b tokens. This represents the largest known Chinese financial pretraining corpus for models of this parameter size. As a better backbone, FinBERT2 can bridge the gap in the financial-specific deployment of LLMs through the following achievements: (1) Discriminative fine-tuned models (Fin-Labelers) outperform other (Fin)BERT variants by 0.4%-3.3% and leading LLMs by 9.7%-12.3% on average across five financial classification tasks. (2) Contrastive fine-tuned models (Fin-Retrievers) outperform both open-source (e.g., +6.8\% avg improvement over BGE-base-zh) and proprietary (e.g., +4.2\% avg improvement over OpenAI's text-embedding-3-large) embedders across five financial retrieval tasks; (3) Building on FinBERT2 variants, we construct the Fin-TopicModel, which enables superior clustering and topic representation for financial titles. Our work revisits financial BERT models through comparative analysis with contemporary LLMs and offers practical insights for effectively utilizing FinBERT in the LLMs era.

Figures

Figures reproduced from arXiv: 2506.06335 by the authors.

Figure 1
Figure 1. Illustration of how FinBERT2 bridges the gap in finance-specific deployment. The comparison presents two solution: (1) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A overview of our FinBERT2 work. 2) Company Announcements Corpus (6.4B tokens, 19G): Sourced from web-scraped announcements on the official websites of domestic listed companies, this dataset includes a wide array of corporate disclosures from various industries, such as financial reports, significant event statements, notices of shareholders’ meet￾ings, stock repurchase plans, and executive changes. It spans 20 yea… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 22 canonical work pages

  1. [1]

    Dimo Angelov. 2020. Top2vec: Distributed representations of topics.arXiv preprint arXiv:2008.09470(2020)

    arXiv 2020

  2. [2]

    Dogu Araci. 2019. FinBERT: Financial Sentiment Analysis with Pre-Trained Language Models. https://arxiv.org/abs/1908.10063

    arXiv 2019

  3. [3]

    Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, and Muhammad Abdul-Mageed. 2024. FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models. https://arxiv.org/abs/2402.10986

    arXiv 2024

  4. [4]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258(2021)

    arXiv 2021

  5. [5]

    Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar

    work page

  6. [6]

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre- training with whole word masking for chinese bert.IEEE/ACM Transactions on Audio, Speech, and Language Processing29 (2021), 3504–3514

    work page 2021

  7. [7]

    2019.FinBERT: Pre-Trained Model on SEC Filings for Financial Natural Language Tasks

    Desola. 2019.FinBERT: Pre-Trained Model on SEC Filings for Financial Natural Language Tasks. doi:10.13140/RG.2.2.19153.89442

    work page arXiv 2019

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Comput...

    doi:10.18653/v1/n19-1423 2019

Show all 50 references
  1. [9]

    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794 [cs.CL] https://arxiv.org/abs/2203.05794

    2022 arXiv

  2. [10]

    Panpan Hou, Mengchao Zhang, Zhibing Fu, and Yu Li. 2020. FinBERT. https: //github.com/valuesimplex/FinBERT

    2020

  3. [11]

    Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. 2024. Bad Actor, Good Advisor: Exploring the Role of Large Lan- guage Models in Fake News Detection.Proceedings of the AAAI Conference on Artificial Intelligence38, 20 (2024), 22105–22113. doi:10....

    2024 arXiv

  4. [12]

    Huang. 2023. FinBERT: A Large Language Model for Extracting Information from Financial Text*.Contemporary Accounting Research40, 2 (2023), 806–841. doi:10.1111/1911-3846.12832

    2023

  5. [13]

    Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding- based retrieval in facebook search. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & ...

    2020

  6. [14]

    IDEA-CCNL. 2021. Fengshenbang-LM. https://github.com/IDEA-CCNL/ Fengshenbang-LM

    2021

  7. [15]

    Junkyu Jang, Taehwan Kim, and Sung-Hyuk Park. 2024. Stock Index Forecasting Using an Explainable TAFT Model with Online Data-Driven Social Sentiment Index. InProceedings of the 5th ACM International Conference on AI in Finance (Brooklyn, NY, USA)(ICAIF ’24). Association for Co...

    2024

  8. [16]

    Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering.arXiv preprint arXiv:2004.04906(2020)

    2020 arXiv

  9. [17]

    Jan Koco´n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Koco´n, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, ...

    2023 arXiv

  10. [18]

    Constantinides, and Danilo Mandic

    Thanos Konstantinidis, Giorgos Iacovides, Mingxue Xu, Tony G. Constantinides, and Danilo Mandic. 2024. FinLlama: Financial Sentiment Classification for Algo- rithmic Trading Applications. https://arxiv.org/abs/2403.12285

    2024 arXiv

  11. [19]

    G Lample. 2019. Cross-lingual language model pretraining.arXiv preprint arXiv:1901.07291(2019)

    2019 arXiv

  12. [20]

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent re- trieval for weakly supervised open domain question answering.arXiv preprint arXiv:1906.00300(2019)

    2019 arXiv

  13. [21]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processi...

    2020

  14. [22]

    Lewis, Luke Zettlemoyer, and Veselin Stoyanov

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach.ArXiv(2019)

    2019

  15. [23]

    Ilya Loshchilov, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2017. Fixing weight decay regularization in adam.arXiv preprint arXiv:1711.05101(2017)

    2017 arXiv

  16. [24]

    Guangyuan Ma, Xing Wu, Zijia Lin, and Songlin Hu. 2024. Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval.. InProceed- ings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1818–1827

    2024

  17. [25]

    Vitaly Meursault, Pierre Jinghong Liang, Bryan R Routledge, and Madeline Marco Scanlon. 2023. PEAD. txt: Post-Earnings-Announcement Drift Using Text.Journal of Financial and Quantitative Analysis58, 6 (2023), 2299–2326

    2023

  18. [26]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Dubrovnik, C...

    2023

  19. [27]

    Pandu Nayak. 2019. Understanding searches better than ever before. https: //blog.google/products/search/search-language-understanding-bert/ Accessed: 2025-01-05

    2019

  20. [28]

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al

  21. [29]

    NetEase Youdao

    Inc. NetEase Youdao. 2023. BCEmbedding: Bilingual and Crosslingual Embedding for RAG. https://github.com/netease-youdao/BCEmbedding

    2023

  22. [30]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748(2018)

    2018 arXiv

  23. [31]

    Muhammad Qorib, Geonsik Moon, and Hwee Tou Ng. 2024. Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning?. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (...

    2024 doi

  24. [32]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 5149–5152

    2012

  25. [33]

    Smith, Luke Zettlemoyer, and Tao Yu

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. One Embedder, Any Task: Instruction-Finetuned Text Embeddings. https://arxiv.org/abs/2212. 09741

    2022

  26. [34]

    Yixuan Tang and Yi Yang. 2025. Do We Need Domain-Specific Embedding Models? An Empirical Investigation. arXiv:2409.18511 [cs.CL] https://arxiv.org/ abs/2409.18511

    2025 arXiv

  27. [35]

    Nandan Thakur, Nils Reimers, Andreas R¨uckl´e, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

    2021 arXiv

  28. [36]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text Embeddings by Weakly-Supervised Contrastive Pre-Training. https://arxiv.org/abs/2212.03533

    2022 arXiv

  29. [37]

    Neng Wang, Hongyang Yang, and Christina Dan Wang. 2023. FinGPT: Instruc- tion Tuning Benchmark for Open-Source Large Language Models in Financial Datasets. https://arxiv.org/abs/2310.04793

    2023 arXiv

  30. [38]

    Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization?. InInternational Conference on Machine Learning. PM...

    2022

  31. [39]

    2023.M3E: Moka Massive Mixed Embedding Model

    Yuxin Wang, Qingxuan Sun, and Sicheng He. 2023.M3E: Moka Massive Mixed Embedding Model. Moka Massive Mixed Embedding

    2023

  32. [40]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavi´e, Orion Weller, Oskar Hall- str¨om, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long c...

    2024 arXiv

  33. [41]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A Large Language Model for Finance. https://arxiv.org/abs/ 2303.17564

    2023 arXiv

  34. [42]

    Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre- Training Retrieval-oriented Language Models Via Masked Auto-Encoder. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, A...

    2022

  35. [43]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. https://arxiv.org/ abs/2309.07597

    2023 arXiv

  36. [44]

    Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. 2022. Examining scaling and transfer of lan- guage model architectures for machine translation. InInternational Conference on Machine Learning. PMLR, 26176–26192

    2022

  37. [45]

    Xuanyu Zhang and Qing Yang. 2023. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters. InProceedings of the 32nd ACM international conference on information and knowledge management. 4435–4439. FinBERT2: A Specialized Bidirectional Encoder f...

    2023

  38. [46]

    Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards Lightweight yet Ingenious Pre-Trained Models for Chinese. https://arxiv.org/abs/2110.06696 A Prompts of Downstream Tasks using LLMs(Chinese to English alread...

    2021 arXiv

  39. [49]

    Conciseness Definition: A topic should not contain irrelevant or meaningless words, such as noise words or semantically redun- dant terms

  40. [50]

    Evaluation Instructions:For the provided topic keyword list, rate each criterion on a scale of 1 to 3: 1 point: Poor performance, does not meet the standard

    Informativity Definition: A topic should provide sufficient, specific, meaningful, or valuable information, covering different aspects of the same topic. Evaluation Instructions:For the provided topic keyword list, rate each criterion on a scale of 1 to 3: 1 point: Poor perfor...

  41. [2020]

    Pre-training tasks for embedding-based large-scale retrieval.arXiv preprint arXiv:2002.03932(2020)

    2020 arXiv

  42. [2022]

    Text and code embeddings by contrastive pre-training.arXiv preprint arXiv:2201.10005(2022)

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.