{"id":"1a944b06-6f34-42b1-971b-da1845541c20","arxiv_id":"2504.14633","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A LoRA-fine-tuned generative LLM that outputs JSON entities and spans achieves F1 0.945 on CCKS 2019, beating the previous best 0.934.","lead":"The paper fine-tunes a large language model with LoRA to output extracted financial event entities as JSON, and reports a state-of-the-art F1 score of 0.945 on the CCKS 2019 financial event extraction benchmark. A generalist might read it to see whether generative LLMs beat traditional sequence labeling for domain-specific structured extraction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unnamed base LLM makes the claimed F1 0.945 unreproducible; Section 4.1 lacks model identity and hyperparameters, so the SOTA claim cannot be verified.","rationale":"The reader's weakest_assumption is exactly the load-bearing gap: the base LLM is never identified. I agree. The SOTA claim relies on a single F1 point; without model identity, tokenizer, and hyperparameters, no one can check whether the result is real, whether it transfers to a different model, or whether the metric computation matches the baselines. The paper also contains a small reporting inconsistency (Recall claimed improved but actually lower), which further reduces trust. No independent artifacts (code, checkpoints) are provided. The appropriate verdict remains REJECT; a revision that discloses the model and provides reproduction details could make it CONDITIONAL/ACCEPT.","tokens_in":13637,"tokens_out":3229,"duration_ms":26966,"concrete_test":"Request the authors to disclose the base model name, tokenizer, LoRA rank/alpha, learning rate, batch size, epochs, and the generation script. Then independently fine-tune the same base model with those hyperparameters on the official CCKS 2019 training split and evaluate on the test split using the exact-match span metric. If the reproduced F1 does not exceed 0.934 (the stated sebertNets baseline), or if the required information is not provided, the claimed SOTA is unverifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Table 1: F1 0.945, new SOTA) depends on a generative LLM whose identity is never disclosed. Section 3.2 describes only 'a large, pre-trained Transformer-based language model'; Section 4.1 repeats 'a publicly available pre-trained Transformer-based language model' with no name, checkpoint, tokenizer, or context length. The method requires the model to emit exact character start/end indices in the original Chinese text; whether this works hinges on the tokenizer and span alignment. An independent researcher cannot select a base model, reproduce the fine-tuning, or determine whether the 0.011 F1 advantage over sebertNets is real or an artifact of a particular model choice. This is not a stylistic omission: the central contribution is a performance number, and the number is unrecoverable without the model. Additionally, Section 4.2 claims 'improvements in both Precision and Recall' while Table 1 shows Recall 0.942 vs 0.943, an internal inconsistency that undercuts confidence in the reported metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a generative approach to financial event entity extraction: a pretrained LLM is fine-tuned with LoRA to map a prompt containing the input text and event type to a JSON object listing each entity with its character start and end indices. The method is evaluated on the CCKS 2019 Financial Event Entity Extraction dataset against sequence-labeling baselines, reporting an F1 score of 0.945 on the test set, which the authors claim is a new state of the art over sebertNets (0.934). The paper also presents per-event-type, per-entity-type, complexity-based, span-error, and human evaluations.","tokens_in":13819,"tokens_out":6891,"duration_ms":57848,"significance":"If the headline result were reproducible, the paper would provide a useful demonstration that generative LLMs with PEFT can perform span-level financial event entity extraction competitively with specialized sequence-labeling architectures. The task formulation is clear, the choice of a standard benchmark is appropriate, and the additional error-type and human evaluations are a positive aspect. However, the contribution is currently not verifiable because the base model is unnamed, no uncertainty estimates are provided, baseline numbers are not re-run, and several tables are internally inconsistent. As it stands, the paper does not substantiate its state-of-the-art claim.","major_comments":[{"comment":"The pre-trained LLM is never identified; the text refers only to \"a large, pre-trained Transformer-based language model.\" The method's central result depends on this model's tokenizer and its ability to emit character offsets for Chinese text, so without the model name, checkpoint, tokenizer, context length, and decoding settings, the reported F1 of 0.945 cannot be reproduced or verified.","section":"Sections 3.2 and 4.1"},{"comment":"The text states the method achieved \"the highest performance across all evaluated metrics\" and \"improvements in both Precision and Recall,\" but Table 1 shows Recall 0.942 for the proposed method versus 0.943 for sebertNets, a decrease. The numerical claim is therefore internally inconsistent; the authors should correct the prose and report uncertainty (e.g., multiple seeds, confidence intervals, or a significance test), especially since the F1 gap is only 0.011.","section":"Section 4.2 and Table 1"},{"comment":"The baseline results are quoted from the published sebertNets paper [6] rather than re-run under the same experimental conditions. To support a state-of-the-art claim, the authors need to either re-run the baselines with the same evaluation script and data split or provide evidence that the published numbers are directly comparable.","section":"Section 4.1, Baselines"},{"comment":"The span-error analysis is numerically inconsistent with Table 1. For sebertNets, Table 5 reports Exact Match (of Gold) as 93.4% and Missing (of Gold) as 4.6%, implying a recall near 93.4%, whereas Table 1 reports Recall 0.943; for the proposed method, Spurious (of Predicted) 5.0% implies a precision of 95.0%, not 94.8%. The definitions and computations behind Table 5 must be clarified and reconciled with the headline metrics.","section":"Section 4.7 and Table 5"}],"minor_comments":[{"comment":"\"Hyperparameters suchables as\" appears to be a typo; it should read \"such as.\"","section":"Section 4.1"},{"comment":"Tables 2-4 present per-event-type, per-entity-type, and complexity-stratified F1 scores without reporting the number of instances in each stratum or any uncertainty, which limits their interpretability.","section":"Tables 2-4"},{"comment":"The human evaluation does not report inter-annotator agreement (e.g., Cohen's kappa) or a significance test for the 4.25 vs. 3.82 difference, so the strength of that conclusion is unclear.","section":"Section 4.8"},{"comment":"References [32] and [34] discuss healthcare and higher-education chatbot applications, not financial LLMs; the related-work section would benefit from either removing these citations or explicitly explaining their relevance to finance.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would be strengthened by releasing the base model name, code, and hyperparameters, as the current absence of these details makes the central performance claim unverifiable. The reference list also contains several citations to unrelated fields (e.g., traffic prediction, medical vision-language models) that do not substantively support the financial LLM discussion; the editor may wish to assess the citation practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The method is the now-standard recipe: take a pretrained LLM, LoRA fine-tune it to emit JSON entity lists with character spans, evaluate on CCKS 2019. The result is a claimed F1 of 0.945 vs 0.934 for sebertNets. The catch: the base LLM is never identified. Section 3.2 says 'a large, pre-trained Transformer-based language model'; Section 4.1 repeats that without a name, checkpoint, tokenizer, or context length. That makes the headline number unreproducible, and the 0.011 gap is small enough to plausibly be model-choice noise.\n\nWhat's good: the paper does a fairly thorough evaluation. It reports precision/recall/F1, breaks performance down by event type, entity type, instance complexity, analyzes span/error types, and adds a human evaluation. The LoRA setup is described in enough detail—learning rate, batch size, epochs, rank are listed as hyperparameters, though not the actual values. The writing is clear about the task reformulation.\n\nSoft spots beyond the missing model: no variance or significance testing—a single test-set run. Baselines are not rerun; they're taken from the sebertNets paper. And the text claims 'improvements in both Precision and Recall,' but Table 1 shows Recall 0.942 vs 0.943, a slight drop. The paper acknowledges 'a slight trade-off' in the same paragraph, so it's internally inconsistent in wording, not necessarily in data. Still, it doesn't inspire confidence.\n\nThe paper also cites no prior generative NER work (e.g., other LLM-based extraction methods), so it doesn't position itself against the closest alternatives.\n\nWho gets value: someone wanting a template for PEFT-based extraction on a Chinese financial benchmark might find the breakdown useful, but as a research claim, the SOTA assertion is not verifiable. I'd recommend a serious editor send it to review only if the authors can identify the base model and share code/hyperparameters; otherwise, desk reject. The empirical core is plausible, but the omission is load-bearing.","headline":"The paper's claimed SOTA F1 on CCKS 2019 rests on an unnamed base LLM, making the result unreproducible; the evaluation is thorough but the central number is not verifiable.","tokens_in":14332,"tokens_out":1957,"would_cite":false,"duration_ms":17368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reframing entity extraction as JSON generation hits F1 0.945 in finance","keywords":["financial event entity extraction","generative LLM","parameter-efficient fine-tuning","LoRA","CCKS 2019","structured output generation","sequence labeling","JSON span extraction"],"falsifier":"Train the same LoRA prompt-to-JSON setup with a publicly named Chinese-capable base LLM and evaluate on the official CCKS 2019 test set; if F1 falls at or below 0.934, the method's edge over sebertNets disappears. Repeating with two different base models would show whether the gain is tied to the specific unnamed model.","tokens_in":13421,"feed_emoji":"📈","tokens_out":4225,"duration_ms":31023,"temperature":0.7,"pith_summary":"The paper claims that financial event entity extraction—identifying companies, people, dates, amounts, and locations tied to events like acquisitions and bankruptcies—can be reformulated as a text-to-JSON generation task. By fine-tuning a large pre-trained language model with LoRA to output entity text and character offsets directly, the method reaches an F1 of 0.945 on the CCKS 2019 test set, above the previous state-of-the-art sequence-labeling model (0.934). The authors argue the generative approach handles long-range dependencies, overlapping entities, and dense instances better than token-level tagging, and support this with per-event, per-entity, complexity, span-accuracy, and human-evaluation analyses. If correct, the result suggests that generative LLMs with parameter-efficient fine-tuning are a strong default for domain-specific structured extraction.","feed_headline":"Reframing entity extraction as JSON generation hits F1 0.945 in finance","feed_subtitle":"Fine-tuning a generative LLM with LoRA beats the best sequence-labeling baseline on the CCKS 2019 benchmark.","key_machinery":"The load-bearing piece of the method is a LoRA-adapted generative LLM trained to maximize the likelihood of a JSON target sequence given a prompt with an instruction, an event type, and the raw text. The target JSON includes each entity's surface text and its start and end character indices in the original input, so the model must learn to map token positions to character offsets. Because LoRA freezes the pre-trained weights and trains only small low-rank adapters, the method adapts a large model with far fewer trainable parameters than full fine-tuning.","core_discovery":"The central discovery is that treating entity extraction as a structured-output generation problem outperforms token-level sequence labeling. On the CCKS 2019 Financial Event Entity Extraction test set, a LoRA-fine-tuned generative LLM that outputs a JSON list of entities with exact character offsets reaches precision 0.948, recall 0.942, and F1 0.945, surpassing the strongest sequence-labeling baseline sebertNets (0.925, 0.943, 0.934). Across all examined event types, entity types, and complexity levels the generative model stays ahead, and its margin widens on instances with five or more entities (0.905 vs 0.875). The error analysis and human evaluation reinforce the same conclusion: more exact span matches, fewer spurious entities, and higher perceived extraction quality.","pith_inferences":["The paper never names its base LLM; if the result transfers across base models, the method is robust, but if it hinges on a specific tokenizer's Chinese character-offset behavior, the headline F1 may not generalize. A natural next experiment is running the identical LoRA setup on several open-weight Chinese-capable base models and comparing F1.","The generative paradigm trades some inference-time latency and decoding cost for tagging simplicity; in real-time trading settings, a distilled or speculative-decoding variant could be needed to make the approach practical.","One testable extension the authors leave implicit is using the same JSON-generation objective to jointly extract entities across multiple event types in a single pass, which could reduce error propagation compared to per-event prompts."],"forward_implications":["The same prompt-to-JSON recipe can be lifted to other information-extraction tasks: relation extraction, event argument extraction, and nested entity recognition.","Parameter-efficient fine-tuning makes the approach feasible with modest compute, so the paradigm is accessible for teams working on domain-specific IE.","Because the output is already structured JSON, downstream pipelines like knowledge-graph construction and alerting systems can consume the extraction directly without span-conversion post-processing.","The improved handling of dense instances (5+ entities) suggests the method is especially valuable for verbose financial documents such as merger filings and regulatory releases.","Reported error patterns—fewer spurious entities but occasional boundary errors—point to span-boundary calibration as the next natural improvement target."],"supporting_citations":[{"why":"Supplies the CCKS 2019 dataset, its official train/test splits, and the evaluation protocol that defines the benchmark.","marker":"[7]"},{"why":"Provides the SEBERTNets and sebertNets baselines whose published scores the generative method must surpass.","marker":"[6]"},{"why":"Frames entity extraction as a core task of deep-learning-based information extraction, giving the task definition the paper rethinks.","marker":"[1]"},{"why":"Surveys large language models in finance, situating the fine-tuning approach within the financial LLM literature.","marker":"[23]"},{"why":"Surveys LLMs for financial applications and grounds the generative structured-output paradigm for information extraction.","marker":"[28]"}],"fun_headline_variants":["Generative LLM with LoRA turns extraction into JSON, top F1 0.945","Finance entity extraction: LLM JSON output beats sequence labeling (F1 0.945)","From labels to JSON: LLMs win financial event extraction","Fine-tune an LLM to output JSON, beat finance NER baselines","LLM-based extraction lifts F1 to 0.945 on CCKS 2019 finance dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed F1 of 0.945 depends on an unnamed pre-trained language model whose tokenizer and context length must correctly align character offsets in Chinese financial text; if that model is replaced, the result may not reproduce.","fun_headline_variants_meta":{"raw":{"variants":["Generative LLM with LoRA turns extraction into JSON, top F1 0.945","Finance entity extraction: LLM JSON output beats sequence labeling (F1 0.945)","From labels to JSON: LLMs win financial event extraction","Fine-tune an LLM to output JSON, beat finance NER baselines","LLM-based extraction lifts F1 to 0.945 on CCKS 2019 finance dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1876,"prompt_tokens":972,"completion_tokens":904,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":588,"tokens_out":904,"duration_ms":7342,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:43:31.936841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LoRA prompt-to-JSON setup with a publicly named Chinese-capable base LLM and evaluate on the official CCKS 2019 test set; if F1 falls at or below 0.934, the method's edge over sebertNets disappears. Repeating with two different base models would show whether the gain is tied to the specific unnamed model.","supporting_citations":[{"cited_title":"SEBERTNets: Sequence Enhanced BERT Networks for Event Entity Extraction Tasks Oriented to the Finance Field","cited_arxiv_id":"2401.11408","evidence_quote":"Provides the SEBERTNets and sebertNets baselines whose published scores the generative method must surpass."},{"cited_title":"Applied Sciences 12(19), 9691 (2022)","cited_arxiv_id":null,"evidence_quote":"Frames entity extraction as a core task of deep-learning-based information extraction, giving the task definition the paper rethinks."}],"review_version":1}