{"id":"8209d5bf-7e41-4ccb-9245-8c109cb5a87b","arxiv_id":"2412.04856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a 500-item trading instruction dataset, five LLMs produced well-formatted JSON most of the time but achieved only 5-10% full accuracy and frequently asked unnecessary follow-up questions.","lead":"This paper tests five large language models on 500 sample stock trading orders written in everyday language, asking each to turn them into machine-readable trade instructions. It finds the models generate the right format often but get the details right only 5 to 10 percent of the time, and they ask too many extra questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 5–10% accuracy claim rests on an unreleased, single-authored gold standard, and Tables 5–6 are arithmetically inconsistent, so the central quantitative result is not currently reproducible or verifiable.","rationale":"The reader's weakest assumption identified the reliability of the manually generated JSON labels and the validity of the metric definitions. My independent reading confirms that this is the most load-bearing point: the paper's headline quantitative claims, especially the 5–10% accuracy and the 'perfect follow-up rates,' depend entirely on these labels and formulas, and the published tables contain internal inconsistencies that prevent the numbers from being reconstructed from the stated definitions. This is not a disagreement with the broad plausibility of the finding that LLMs format well but frequently extract trading content incorrectly; that qualitative direction is supported by the examples and the authors' own discussion. However, a 5–10% accuracy figure is extraordinary and needs a defensible measurement chain. The paper provides no dataset release, no inter-annotator agreement, no error bars, and no code, so the quantitative contribution is currently unverifiable. The proposed pipeline in Section 7 is a plausible engineering contribution, but its effectiveness is not quantitatively benchmarked either. These issues are addressable through release and re-annotation, so the appropriate verdict remains conditional rather than outright rejection. The reader's conditional verdict already captures this, so I do not change it.","tokens_in":10188,"tokens_out":7036,"duration_ms":72037,"concrete_test":"Ask the authors to release the 500-item dataset, every model's raw JSON output, follow-up transcripts, and the 472 manual labels, then perform a two-part audit. First, reconstruct Table 5 and Table 6 from per-item counts using the stated formulas; if any row cannot be reproduced, the reported metrics are unreliable. Second, independently re-annotate a random 100-item subset with two finance-literate annotators using a written rubric (including handling of null vs. None and market vs. limit orders), compute Cohen's kappa between the annotators and against the authors' labels, and recompute Accuracy and Missing Rate using only consensus labels. If kappa is below 0.8 or the recomputed accuracy moves by more than 5 percentage points, the claim that no tested LLM can reliably convert instructions is not supported by the current data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that the authors manually generated JSON labels for 472 of 500 items, but no annotation protocol, inter-annotator agreement, or release of the labels is provided. The central 5–10% accuracy figure is therefore a comparison against an unverifiable gold standard. The problem is amplified by arithmetic inconsistencies in the published tables. For Table 5, applying the Table 4 formulas to any fixed N=500 produces no consistent assignment. Example: Yi-large, 87.50% generation rate gives 437.5 JSON outputs; 14.29% missing rate gives roughly 62.5 missing outputs; 82.86% 'Correctness' gives roughly 362.5 outputs that are apparently not missing or erroneous; yet Accuracy is 10%, i.e., 50 correct outputs, which cannot equal the 362.5 non-error outputs unless the definitions of 'Correctness' and 'Accuracy' are unrelated, which is never stated. For GPT-4o, Missing Rate equals Correctness (63.56%), forcing Error Rate to be 0, while Accuracy is 10%, again irreconcilable. Table 6 is also ambiguous: a 100% follow-up rate with 15–43% missed follow-up is only coherent if 'follow-up' means 'asked at least one question,' so the abstract's 'perfect follow-up rates' overstates the models' ability to ask the right questions. Because the dataset and code are not released, none of the headline percentages can be reproduced or checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an intelligent pipeline that converts natural-language financial trading instructions into a standardized JSON format, describes a 500-item dataset generated with GPT-4o and manual refinement, and evaluates five LLMs (Yi-large, DeepSeek-v2.5, GPT-4o, GPT-4o-mini, Qwen-max-0428) using a hierarchy of metrics: Generation Rate, Missing Rate, Error Rate, Accuracy, Follow-up Rate, Missed Follow-up Rate, and Extra Follow-up Rate. The claimed findings are that models generate JSON at high rates (87.50–98.33%) and always follow up, but accuracy is only 5–10%, missing rates range from 14.29% to 67.29%, and models tend to over-inquire. The paper also describes a trading execution pipeline and a user interaction example.","tokens_in":10476,"tokens_out":5508,"duration_ms":51478,"significance":"The study addresses a practical and under-explored task: translating noisy natural-language trading orders into executable structured commands. If the reported numbers are validated, the result would be a useful caution about using LLMs as-is in financial execution systems, and the over-inquiry finding is a concrete behavioral insight. The paper's strength is its attempt to define an ordered set of metrics that separates generation from correctness and completeness. However, the central quantitative claims are not currently verifiable: the gold-standard labels are not released, no inter-annotator agreement is reported, and Tables 5 and 6 are internally inconsistent with the metric definitions in Table 4. No code, prompts, or raw counts are provided, so the headline 5–10% accuracy figure cannot be reproduced or checked by readers.","major_comments":[{"comment":"The note under Table 4 assigns '#' (higher is better) to Missing Rate, Error Rate, Missed Follow-up Rate, and Extra Follow-up Rate, and '*' (lower is better) to Follow-up Rate. This is reversed for all five of these metrics: higher missing/error rates are worse, missed follow-ups and extra follow-ups are worse, and a higher follow-up rate is better. The inversion contradicts the interpretation in §6.1, where high Missing Rate is explicitly treated as a defect. The symbol key must be corrected before any of the reported rates can be interpreted.","section":"Table 4"},{"comment":"Table 5 cannot be reconciled with the formulas in Table 4. For GPT-4o, Generation Rate 98.33% implies about 492 JSON outputs out of 500; Missing Rate 63.56% would then mean about 313 outputs with missing fields, leaving at most 179 non-missing outputs. Correctness is also 63.56%, which would require Error Rate to be zero, but Accuracy is 10%, i.e., 50 correct outputs out of 500. For Yi-large, Generation Rate 87.50% gives 437.5 outputs; Missing Rate 14.29% gives about 62.5 missing; Correctness 82.86% gives about 362.5 non-error outputs; yet Accuracy 10% gives 50 correct outputs. The stated definitions of 'Correctness' and 'Accuracy' make these numbers mutually inconsistent. Please report raw counts and define each denominator; otherwise the headline 5–10% accuracy figure is not interpretable.","section":"§6.1, Table 5"},{"comment":"Table 6 reports Follow-up Rate 100% for all models while also reporting Missed Follow-up Rate values between 15.25% and 43.22%. Under the Table 4 definitions, Missed Follow-up Rate = Missing Follow-ups / Total Required Follow-ups and Follow-up Rate = Follow-ups / Total Required Follow-ups; if any required follow-up is missed, Follow-ups cannot equal Total Required Follow-ups. A rate of 100% is possible only if 'Follow-up Rate' means 'the model asked at least one question per incomplete case' rather than 'asked all required questions.' This distinction changes the abstract's claim of 'perfect follow-up rates' and must be clarified with explicit numerators and denominators.","section":"§6.2, Table 6"},{"comment":"The gold standard for the accuracy evaluation is described in one sentence: the authors 'manually generated JSON outputs for 472 out of 500' items. No annotation protocol, per-field correctness definition, inter-annotator agreement, or release of the labels is provided. Because accuracy (5–10%) is the central quantitative result, the paper should release the dataset with labels and prompts and report agreement on at least a subset. Without this, the headline accuracy cannot be independently verified.","section":"§3.3"},{"comment":"Section 5.1 states that the inquiry-capability evaluation used 'a dataset of 118 trading strategy descriptions,' which is inconsistent with the paper's description of a 500-item dataset evaluated across all metrics. It is unclear whether the 118-item set is a subset of the 500, a separate set, or the base from which the 500 were derived; no cross-reference is given. The denominators of the Table 6 rates are therefore undefined, and the reader cannot determine what population the follow-up results describe.","section":"§5.1"}],"minor_comments":[{"comment":"The abstract's 'perfect follow-up rates' overstates the results given the 15.25–43.22% Missed Follow-up Rates in Table 6; this should be reworded once the metric definitions are clarified.","section":"Abstract and §6.2"},{"comment":"The organization paragraph lists Section 6 for results but omits Section 5 (Experiment) from the roadmap, even though Section 5 contains the experimental setup.","section":"§1"},{"comment":"There are numerous typographical errors, including 'Developd' (Section 1), 'difficultly' (Section 6.2), 'mis-follow-up' (Table 7), and 'honoring of information' (Section 1); these should be corrected in a careful proofreading pass.","section":"Throughout"},{"comment":"The example labels a price question as a correct follow-up for a market order, but Section 3.3 states that market orders have price 'None' by definition; the criterion for when a price field is missing should be stated explicitly so that the labeling is consistent.","section":"§6.2, Table 7, Case 2"},{"comment":"The statement that 'all results are validated in our system' is not supported by any quantitative validation evidence in the paper; please either provide the validation results or soften the claim.","section":"§7"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., entries missing venue, volume, or page numbers), and some in-text mentions such as the 'Financial NLP Benchmark' are not cited; please harmonize the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is an empirical benchmark paper whose central quantitative claims are not reproducible without data release, and whose published tables are internally inconsistent. The errors are fixable by correcting the metric directions, clarifying denominators, reporting raw counts, and releasing the dataset and labels, so I recommend major revision rather than rejection. The editor should require the dataset, gold-standard labels, prompts, and raw counts as part of the revision, and should ask the authors to re-run or recalculate all tables under the clarified definitions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the LLM trading instruction paper. Short version: the task is worth studying and the benchmark idea is useful, but the numbers as reported don't hold together, and the gold standard is unreleased, so the central 5–10% accuracy claim is currently unverifiable.\n\nWhat's new: a 500-item Chinese-English trading instruction dataset with noise injection and slicing, and an evaluation design that separates format generation from inquiry capability. That split is the best part. It lets you see that models can emit well-formed JSON while often getting fields wrong, and that they tend to over-ask rather than under-ask. The qualitative pattern—high generation, low accuracy, high extra follow-ups—is plausible and relevant for anyone building a human-in-the-loop execution system.\n\nThe soft spots are real. Table 4's footnote has the 'better' arrows backwards for every metric, which makes the table confusing even before you check the arithmetic. Table 5 doesn't add up. For Yi-large, 87.50% generation on 500 inputs gives 437.5 JSON outputs; 14.29% missing gives about 62.5 missing; 82.86% 'correct' gives about 362.5 correct outputs. Yet accuracy is 10%, i.e., 50 correct. Those numbers can't all be true under the stated definitions. GPT-4o's row is worse: missing rate equals correctness, which forces error rate to zero, while accuracy is 10%. Either the definitions of 'Correctness' and 'Accuracy' are unrelated, which is never stated, or the table has errors.\n\nTable 6's uniform 100% follow-up rate with 15–43% missed follow-up is coherent only if 'follow-up' means asked at least one question. The abstract calls that 'perfect follow-up rates,' which overstates the models' ability to ask the right questions. The deeper problem: the manual labels for 472 of 500 items were created without a documented protocol or inter-annotator agreement, and neither the dataset nor the code is released. So every headline percentage is a comparison against a gold standard that no one can inspect. There's also mild circularity in generating the data with GPT-4o and then scoring GPT-4o on it, though the manual pass softens that.\n\nThis paper is for people building LLM-based order execution or a financial NLP evaluation benchmark. The qualitative findings are probably right, but as a quantitative contribution it needs major revision. I'd send it to peer review rather than desk reject—any decent referee will demand corrected metrics and released data, and the paper will be much better for it. As it stands, I wouldn't cite the numbers.","headline":"Useful task, plausible qualitative pattern, but the reported accuracy numbers are internally inconsistent and the unreleased human labels make them unverifiable.","tokens_in":10988,"tokens_out":3728,"would_cite":false,"duration_ms":35398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No tested LLM can reliably turn natural-language trade orders into executable JSON; accuracies run 5% to 10%.","keywords":["large language models","trade order recognition","financial trading instructions","JSON generation","automated trading","LLM evaluation","missing information detection","natural language processing in finance"],"falsifier":"Take the same 500 instructions, have two independent finance annotators label executable JSON for every item, and compare the five models under exact field-level matching; if accuracy climbs well above 10% or the manual labels disagree with each other on more than a small fraction, the paper's central numbers would not survive.","tokens_in":9995,"feed_emoji":"📉","tokens_out":4619,"duration_ms":43120,"temperature":0.7,"pith_summary":"This paper asks whether current large language models can convert a human trader's natural-language instruction into a structured, executable order. It tests five models on a 500-item Chinese-English dataset of trading instructions and finds that the models almost always produce well-formed JSON (87.50% to 98.33%) and always ask follow-up questions, yet their outputs are correct only 5% to 10% of the time. Required fields are omitted at rates from 14.29% to 67.29%, and extra, unnecessary follow-up questions appear in 76.27% to 91.53% of cases. If this holds, LLMs are not yet safe as direct parsers for trade execution without a validation layer or manual checks.","feed_headline":"LLMs get trade orders right only 5 to 10 percent of the time","feed_subtitle":"High generation and follow-up rates hide missing fields and over-inquiry, so direct use in trading systems needs a validation layer.","key_machinery":"The system's core is a trade order recognition pipeline plus a six-metric evaluation stack. The pipeline maps a natural-language instruction onto a fixed JSON schema with fields for strategy, symbol, order type, price, and quantity, then parses, validates, and asks for missing information before execution. The metrics are ordered from shallow to deep: Generation Rate (valid JSON produced), Missing Rate, Error Rate, Accuracy, Follow-up Rate, and Missed/Extra Follow-up Rate. The load-bearing measurement device is the 472-item manual alignment of gold JSON labels used as the reference for accuracy.","core_discovery":"The central discovery is an empirical mismatch: five state-of-the-art LLMs (GPT-4o, GPT-4o-mini, Qwen-max-0428, DeepSeek-v2.5, and Yi-large) can generate JSON in the requested format and reliably initiate follow-up questions, but they rarely produce complete and correct executable orders. On a dataset of 500 trading instructions augmented with noise and slicing, with 472 manually aligned JSON labels as the reference, the best accuracy is 10% (Yi-large and GPT-4o) and the worst is 5% (GPT-4o-mini). Missing rates range from 14.29% to 67.29%, and the models' perfect 100% follow-up rates occur alongside missed follow-ups of 15.25% to 43.22% and extra-field rates of 76.27% to 91.53%. The paper also documents specific error modes: confusion between market and limit orders, confusion between the string 'None' or 'NULL' and true null in JSON, and unnecessary follow-up questions when the instruction is already complete.","pith_inferences":["A natural next step, not in the paper, would be to measure how much the pipeline's validation layer actually repairs the 90% to 95% error rate; the paper demonstrates the pipeline but does not report an end-to-end accuracy after its corrections.","The results suggest a testable extension: fine-tune a general-purpose LLM on a few thousand noisy trade instructions and re-run the same six metrics to see whether missing and extra-field rates drop.","The manual gold labels are the pivot point; re-labelling with two independent annotators and reporting agreement would tell whether the 5% to 10% accuracy range is a property of the models or partly of the labels."],"forward_implications":["No tested model can be dropped directly into a live trading system as a natural-language order parser; the paper argues that manual correction or a validation pipeline is required.","A 100% follow-up rate is not evidence of good inquiry: every model combined it with 15.25% to 43.22% missed follow-ups and 76.27% to 91.53% extra questions, so follow-up behavior should be judged by precision, not just presence.","Because over-interrogation extracts additional personal and financial information, the models' tendency to over-ask carries information-security consequences for voice or chat trading interfaces.","The specific failure modes, such as market-versus-limit-order confusion and JSON string-versus-null confusion, identify concrete targets for prompt engineering or fine-tuning."],"supporting_citations":[{"why":"Supplies GPT-4o and GPT-4o-mini, two of the five models whose generation and inquiry behavior is measured.","marker":"[36]"},{"why":"Supplies Qwen-max-0428, the model with the highest missing rate in the evaluation.","marker":"[37]"},{"why":"Supplies DeepSeek-v2.5, one of the three Chinese-oriented models compared.","marker":"[38]"},{"why":"Supplies Yi-large, the model with the lowest missing rate in the evaluation.","marker":"[39]"},{"why":"Provides the template for generating the 500-item instruction dataset with GPT-4o.","marker":"[34]"},{"why":"Grounds the self-instruction approach that motivated generating evaluation data from a strong LLM.","marker":"[35]"}],"fun_headline_variants":["AI traders nail orders 5–10% of time","LLM trade order accuracy: single digits","Trade bots generate fine, execute poorly","LLMs over-ask, under-execute trade orders","LLM trade execution: high talk, low precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the 472 manually created JSON labels being a correct gold standard and on the Table 4 metric definitions accurately measuring what they name; Section 3.3 does not report inter-annotator agreement, so if the labels or metric interpretations are loose, the accuracy and missing-rate numbers shift.","fun_headline_variants_meta":{"raw":{"variants":["AI traders nail orders 5–10% of time","LLM trade order accuracy: single digits","Trade bots generate fine, execute poorly","LLMs over-ask, under-execute trade orders","LLM trade execution: high talk, low precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1371,"prompt_tokens":988,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":310}},"tokens_in":604,"tokens_out":383,"duration_ms":4301,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:11:58.987009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 500 instructions, have two independent finance annotators label executable JSON for every item, and compare the five models under exact field-level matching; if accuracy climbs well above 10% or the manual labels disagree with each other on more than a small fraction, the paper's central numbers would not survive.","supporting_citations":[{"cited_title":"GPT-4o mini: advancing cost-efficient intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4o and GPT-4o-mini, two of the five models whose generation and inquiry behavior is measured."},{"cited_title":"Notes on Qwen-Max-0428","cited_arxiv_id":null,"evidence_quote":"Supplies Qwen-max-0428, the model with the highest missing rate in the evaluation."}],"review_version":1}