{"id":"aecea6e8-0d92-4a0d-99f8-6a2c5c10e471","arxiv_id":"2501.02237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Generic LLMs score lower than fine-tuned BERT and RoBERTa on financial NER, but few-shot prompting narrows the gap and reveals five recurring error types.","lead":"This paper tests how well six large language models, including GPT-4o, LLaMA-3.1, and Gemini-1.5, recognize person, organization, and location names in financial news articles under three prompting styles. A generalist reader could use the findings to choose a model and prompt format for financial information extraction tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.1(3)'s claimed 'consistent performance ranking' (Gemini > GPT-4o > LLaMA) is contradicted by Table 1: under in-context learning, LLaMA-70B scores 0.8321, above GPT-4o's 0.8203, and the LLaMA family average also exceeds GPT's.","rationale":"After a careful read, the strongest central claim is not just vulnerable to sampling noise; it is contradicted by the paper's own numbers. I therefore do not need to rely on hypothetical reruns to say the 'consistent ranking' is unsupported: Table 1's ICL block puts LLaMA-70B ahead of GPT-4o, and family averages also put LLaMA slightly ahead of GPT for ICL. The paper gives no aggregation rule that would make LLaMA 'lowest' in both few-shot conditions. This is an internal inconsistency, not a matter of external consensus. I still think the broad finding that fine-tuned RoBERTa/BERT beat zero-shot LLMs is supported (largest gap about 0.09), and the failure taxonomy is useful, so conditional acceptance is appropriate. The reader's robustness concern about single runs and fixed shots is real but secondary: it affects tight margins, whereas the table-vs-text contradiction affects the claim as written. The critical next step is to correct or qualify the ranking claim and state the aggregation rule; repeated runs would then validate the corrected version.","tokens_in":5764,"tokens_out":7345,"duration_ms":63441,"concrete_test":"Recompute the rankings from Table 1 separately for In-Context Learning and CoT, using both (a) flagship models only and (b) family averages of the large and mini variants. In (a), ICL order is Gemini (0.8368) > LLaMA-70B (0.8321) > GPT (0.8203), while CoT order is Gemini (0.8369) > GPT (0.8142) > LLaMA-70B (0.8036). In (b), ICL family order is Gemini (0.8214) > LLaMA (0.7988) > GPT (0.7973). If either computation is confirmed, revise Section 4.1(3) to state per-prompt rankings or specify the aggregation rule; optionally, seed/temperature reruns can check whether the mixed ordering is stable, but they are not required to establish the contradiction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's key LLM-ranking claim in Section 4.1(3) fails against its own Table 1. The text says a 'consistent performance ranking' holds after few-shot learning, with Gemini optimal, GPT-4o next, and LLaMA-3.1 lowest. But in the In-Context Learning block, LLaMA-70B's weighted F1 is 0.8321, higher than GPT-4o's 0.8203; family averages (including mini models) give LLaMA 0.7988 versus GPT 0.7973. In the CoT block, GPT-4o (0.8142) is above LLaMA-70B (0.8036), so the ordering flips. No aggregation rule stated in the paper makes LLaMA 'consistently' lowest across the two few-shot conditions. This is an internal inconsistency, independent of sampling noise; at minimum the claim must be reworded to per-prompt rankings, and the ranking conclusion is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a systematic comparison of three LLM families (GPT-4o, LLaMA-3.1, Gemini-1.5, together with smaller variants) under direct prompting, five-shot in-context learning, and chain-of-thought prompting on the FiNER-ORD benchmark, against fine-tuned BERT and RoBERTa. It reports entity-level and weighted F1 scores, claims that fine-tuned models outperform generic LLMs while prompting can narrow the gap, asserts a consistent Gemini > GPT-4o > LLaMA-3.1 ranking after few-shot learning, and presents a manually derived taxonomy of five failure types with percentages.","tokens_in":6006,"tokens_out":5506,"duration_ms":47225,"significance":"If the results are reproducible, the paper provides a useful external benchmark and a practical error taxonomy for LLM-based financial NER. It uses the public FiNER-ORD dataset with ground-truth labels, so the evaluation is not circular, and it releases an artifact link for reproducibility. The failure typology is a concrete contribution to understanding LLM limitations. However, the headline quantitative claims rest on single runs without variance estimates, and one of the ranking conclusions is internally contradicted by Table 1; these issues must be resolved before the evaluation can be fully trusted.","major_comments":[{"comment":"The evaluation is based on a single run per model-prompt configuration: §3.3 states that the five shots are chosen randomly and the same five shots are used in every experiment, §3.2 states that all models use default API configurations, and Table 1 reports only point estimates with no error bars, repeats, or significance tests. Many headline differences are small (e.g., Gemini 0.8368 vs. LLaMA-70B 0.8321 under in-context learning), so sampling variance and the particular choice of shots could alter the model ordering or the claim that few-shot learning helps. Please provide multiple runs with different shots and report means and variances, or explicitly temper the comparative claims.","section":"§3.2, §3.3, Table 1"},{"comment":"The claimed 'consistent performance ranking' (Gemini optimal, GPT-4o next, LLaMA-3.1 lowest) is not supported by Table 1. Under in-context learning, LLaMA-70B achieves weighted F1 0.8321, above GPT-4o's 0.8203, and the family averages also place LLaMA (0.7988) above GPT (0.7973); under chain-of-thought, GPT-4o (0.8142) is above LLaMA-70B (0.8036), so the ordering flips across conditions. Since no aggregation rule stated in the paper yields LLaMA as consistently lowest, this conclusion must be reworded to per-condition, per-size rankings or removed.","section":"§4.1(3), Table 1"},{"comment":"The failure taxonomy is manually annotated without inter-annotator agreement or a described coding protocol, so the reported distributions (e.g., 31.3% contextual misunderstanding) are not shown to be reliable. Additionally, the text says the paper identifies 'five representative failure types,' but Table 2 lists six rows, with 'Boundary errors' having no percentage; clarify whether boundary errors are a sixth category or part of 'Entity omission and boundary errors.'","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"There are typos: 'are yet need a better understanding' in the abstract and 'artifcats' in the conclusion should be corrected.","section":"Abstract and Conclusion"},{"comment":"The sentence beginning 'The Gemini series outperforms the GPT-4o and LLaMA 3.1 series...' appears twice verbatim; one copy should be removed.","section":"§4.1(3)"},{"comment":"The 'Boundary errors' row lacks a numerical ratio, unlike the other rows, which makes the table incomplete; either provide the percentage or explain why it is absent.","section":"Table 2"},{"comment":"The exact prompt templates are only shown in figures; for reproducibility, include the full prompts in an appendix or in the repository.","section":"§3.3, Figures 1–2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: useful benchmark data, but the paper's own Table 1 contradicts the ranking claim in Section 4.1(3), so that conclusion needs rewording or re-analysis. The rest of the empirical picture is plausible and worth having.\n\nThe genuinely new stuff is a clean comparison of six LLM configurations (GPT-4o, LLaMA-3.1, Gemini-1.5, plus small variants) under direct, 5-shot ICL, and CoT prompting on FiNER-ORD, with fine-tuned BERT/RoBERTa baselines. The manual failure taxonomy (five types with ratios) is a nice practical addition. This extends Li et al. (2023a) rather than replaces it; the 'first comprehensive comparison' line is overstated, since Li et al. already covered ChatGPT and GPT-4 on financial text analytics. Still, the wider model/prompt grid is a legitimate contribution.\n\nWhere it gets soft: Section 4.1(3) says 'a consistent performance ranking, with the Gemini series achieving the optimal performance, followed closely by the GPT-4o series. The LLaMA 3.1 series exhibits the lowest performance.' Table 1 doesn't support that as stated. Under In-Context Learning, LLaMA-70B scores 0.8321 vs GPT-4o's 0.8203; family averages are almost tied. Under CoT, GPT-4o (0.8142) beats LLaMA-70B (0.8036). So the ordering flips across conditions. Maybe they meant per-configuration rankings, but the sentence says 'consistent' across the board. That's an internal inconsistency, not just noise. It needs to be fixed.\n\nThe deeper problem is that all numbers come from a single run with one fixed set of five random examples and default API sampling. No error bars, no repeats, no significance tests. The gaps that drive the ranking claims are small (around 0.01), so we can't know if they're real. The failure-type annotation is manual with no inter-annotator agreement; the ratios in Table 2 are a starting point, not a measurement.\n\nOn the positive side, the evaluation uses an external benchmark (FiNER-ORD) and standard fine-tuned baselines, so there's no circularity. The failure examples are concrete and useful. The paper is honest about limitations in the discussion, though it doesn't mention the single-run issue explicitly.\n\nBottom line: this is a solid practitioner-oriented benchmark for financial NER, with one clear internal contradiction to fix and a need for repeated runs or confidence intervals. I'd send it to review if the authors fix the 4.1(3) claim and add stability checks; it's a useful data point, not a breakthrough. For a reading group, it's a maybe—quick read, but no new method to chew on.","headline":"Useful benchmark data undercut by a ranking claim that contradicts the paper's own Table 1.","tokens_in":6499,"tokens_out":2615,"would_cite":true,"duration_ms":22673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned transformer models still hold the accuracy edge over generic LLMs on financial named entity recognition, but few-shot prompting narrows the gap.","keywords":["financial named entity recognition","large language models","prompt engineering","few-shot learning","chain-of-thought prompting","FiNER-ORD","error analysis","fine-tuned transformers"],"falsifier":"Run the same six LLMs and three prompting styles on FiNER-ORD multiple times, varying the five-shot examples and sampling parameters, then check whether the weighted-F1 ordering (Gemini above GPT-4o above LLaMA, and few-shot above direct) survives; a single counterexample where a different shot set flips the leaderboard or makes few-shot hurt would falsify the headline conclusions.","tokens_in":5584,"feed_emoji":"📊","tokens_out":5150,"duration_ms":41955,"temperature":0.7,"pith_summary":"This paper asks how well generic large language models perform on financial named entity recognition compared with fine-tuned transformer models. Using the FiNER-ORD benchmark, it evaluates GPT-4o, LLaMA-3.1, and Gemini-1.5 under direct, few-shot, and chain-of-thought prompting. It finds that fine-tuned BERT and RoBERTa models still achieve the highest weighted F1 scores, while few-shot prompting narrows the gap and makes Gemini the best LLM among those tested. The paper also catalogs five recurring failure types that explain where LLMs lose accuracy. If correct, the results give a practical baseline for deciding when to fine-tune versus prompt a general model for financial NER.","feed_headline":"Fine-tuned models outdo generic LLMs on financial NER","feed_subtitle":"RoBERTa hits 0.8792 F1; the best LLM reaches 0.8369, and few-shot prompting narrows the gap.","key_machinery":"The benchmark is FiNER-ORD, a manually annotated financial news corpus with three entity types (PER, LOC, ORG) balanced at a ratio of 2.29:1.17:1. The main measurement is entity-level precision, recall, and weighted F1 computed by word matching. The architecture of the comparison is three prompting modes—direct instruction, five-shot in-context learning, and chain-of-thought—applied to six LLM variants, with the systematically annotated failure taxonomy (five types with proportions) supplying the qualitative side of the argument.","core_discovery":"On the FiNER-ORD dataset, fine-tuned transformer models outperform every generic LLM configuration tested: RoBERTa reaches a weighted F1 of 0.8792 and BERT 0.8744, versus a best LLM score of 0.8369 for Gemini-1.5 with chain-of-thought prompts and 0.8368 with few-shot in-context learning. The paper claims the gap can be narrowed through few-shot learning, prompt design, and model size, with the Gemini series ranking first among LLMs after few-shot prompting, followed by GPT-4o and LLaMA-3.1. Chain-of-thought prompting gives no consistent benefit and hurts LLaMA models. The error analysis identifies five failure categories: contextual misunderstanding, pronouns and generic terms, citizenship terms, implied entities, and entity omission and boundary errors.","pith_inferences":["Because the evaluation uses one fixed set of five random shots and default API sampling, the model ordering could shift with different shot sets or repeated runs; a multi-run study would show whether the Gemini advantage is stable.","The error taxonomy likely transfers to other financial information-extraction tasks such as relation extraction and event detection, where the same contextual and implied-entity confusions would appear.","A testable extension: using the five failure categories to generate targeted few-shot examples, such as citizenship terms and abbreviations, should improve LLM F1 more than five random examples.","The small-model results suggest a cost-performance sweet spot: lightweight Gemini-8B with few-shot learning outperforms full-size GPT-4o under direct prompting, so model size alone is not the deciding factor."],"forward_implications":["If these results hold, teams should expect fine-tuned task models to remain the accuracy ceiling for financial NER until prompting or training methods change.","Few-shot in-context learning is a reliable way to boost LLM performance on this task, while chain-of-thought prompting is not.","Model size helps: the larger member of each LLM family beats its smaller variant in almost every condition tested.","The five failure categories give a concrete checklist for improving financial NER systems, including citizenship adjectives and implied entities like product names.","The Gemini series is a reasonable choice over GPT-4o and LLaMA-3.1 for few-shot financial NER when fine-tuned models are unavailable."],"supporting_citations":[{"why":"Supplies the FiNER-ORD benchmark dataset and the earlier prompt design that the paper compares against.","marker":"(Shah et al., 2023)"},{"why":"Provides the CRA dataset used to motivate FiNER-ORD's more balanced entity distribution.","marker":"(Alvarado et al., 2015)"},{"why":"Supplies the BERT fine-tuned baseline model.","marker":"(Devlin, 2018)"},{"why":"Supplies the RoBERTa fine-tuned baseline, which achieves the best overall weighted F1.","marker":"(Liu, 2019)"},{"why":"Defines the GPT-4o and GPT-4o-mini models evaluated in the study.","marker":"(OpenAI, 2024)"},{"why":"Defines the LLaMA-3.1-70B-Instruct and LLaMA-3.1-8B-Instruct models evaluated in the study.","marker":"(Dubey et al., 2024)"},{"why":"Defines the Gemini-1.5-flash and Gemini-1.5-flash-8B models evaluated in the study.","marker":"(Google, 2024)"},{"why":"Provides the chain-of-thought prompting method tested as one of the three prompt designs.","marker":"(Wei et al., 2022)"},{"why":"Provides the in-context learning prompting method used for few-shot experiments.","marker":"(Dong et al., 2022)"}],"fun_headline_variants":["LLMs lag fine-tuned models on financial NER","For financial NER, RoBERTa still beats Gemini","Few-shot prompts shrink the LLM NER gap","Generic LLMs trail RoBERTa on finance NER","Chain-of-thought fails to lift LLaMA NER scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on a single run per model and prompt with one fixed set of five randomly chosen examples and default sampling settings, so the reported F1 scores assume these choices are representative.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lag fine-tuned models on financial NER","For financial NER, RoBERTa still beats Gemini","Few-shot prompts shrink the LLM NER gap","Generic LLMs trail RoBERTa on finance NER","Chain-of-thought fails to lift LLaMA NER scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000102,"raw_usage":{"total_tokens":973,"prompt_tokens":839,"completion_tokens":134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":62}},"tokens_in":455,"tokens_out":134,"duration_ms":2339,"temperature":1.0,"reasoning_tokens":62,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:19.835522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six LLMs and three prompting styles on FiNER-ORD multiple times, varying the five-shot examples and sampling parameters, then check whether the weighted-F1 ordering (Gemini above GPT-4o above LLaMA, and few-shot above direct) survives; a single counterexample where a different shot set flips the leaderboard or makes few-shot hurt would falsify the headline conclusions.","supporting_citations":[],"review_version":1}