{"id":"0ba2549f-3b5e-4f36-9c3a-0dc4bccf6c5e","arxiv_id":"2505.07871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding annotators' instructions to the prompt raises LLM financial sentiment accuracy on a new WallStreetBets dataset by an average of 5.90 percent, and a confidence-based sentiment score helps stock prediction on some tickers.","lead":"The authors show that adding the exact instructions given to human annotators into the prompt improves LLM accuracy on a new WallStreetBets financial sentiment dataset by up to 9.08 percent. The finding is relevant because it suggests current financial sentiment benchmarks may be unfair to models, and because the resulting sentiment scores are used as features for stock price prediction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WSBS gold labels are the load-bearing target: with no inter-annotator agreement reported and 48% of samples resolved by majority vote, AIAP's gains may reflect reproducing the annotation rubric rather than robust sentiment understanding; a reliability check is needed.","rationale":"The paper's strongest claim is a quantitative accuracy gain on WSBS. For that claim to be meaningful, the accuracy target—the WSBS labels—must be reliable. The annotation process (Section 2.1) provides no kappa or alpha, describes first-pass annotation with only two annotators per sample, and leaves 48.3% of samples to an underspecified majority-vote stage. The AIAP prompt (Section 2.2) then contains the same instruction used to produce those labels, so the model is given the annotation rubric at inference time. This makes the reported gains a measure of how well the model follows the rubric, not necessarily of improved general sentiment understanding. The paper's own few-shot and incremental-component controls show that the gain is not just from adding examples, which is evidence for the mechanism, but they do not establish that the rubric-based labels are stable across annotator teams or that the method would improve accuracy against independently produced labels. All 18 reported comparisons are positive, so the lack of statistical testing is a weaker concern than label reliability. A direct re-annotation check would settle the issue: high agreement would validate WSBS as a benchmark and leave the conditional accept intact; low agreement would require reinterpreting the headline as rubric-following on a self-consistent dataset, which would reduce the claim's external significance. The reader's conditional verdict already anticipates this, so no change is needed.","tokens_in":13910,"tokens_out":15354,"duration_ms":157563,"concrete_test":"Have an independent team of at least three annotators, blind to the original WSBS labels, re-annotate a random 300-sample subset of the Full dataset using only the Figure 1 instruction. Then compute pairwise Cohen's kappa between the original majority labels and the new majority labels. If kappa is below 0.6, or if label agreement is below 80%, the WSBS gold labels are not stable enough to support the +5.90% headline as a measure of sentiment accuracy, and results on the All-Agree subset alone would not rescue the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim (Section 1: +5.90% average, +9.08% best case) is an accuracy comparison against the WSBS labels described in Section 2.1. The comparison is internally consistent and all 18 reported cells are positive, so the absence of significance tests is not the primary risk. The load-bearing assumption is that the WSBS majority labels are a stable, meaningful ground truth. The paper reports no inter-annotator agreement statistic: only 1,509 of 2,920 samples (51.7%) were unanimous in the first pass, and the remaining 48.3% were assigned through an incompletely specified conflict-resolution stage. Since AIAP (Section 2.2) inserts the exact instruction from Figure 1 into the prompt, the model is handed the annotation rubric at test time. The observed gain is therefore compatible with the model reproducing the rubric on a self-consistent label set, rather than demonstrating a general improvement in financial sentiment understanding. The All-Agree subset showing larger gains than Full is consistent with this rubric-matching story, but does not separate it from the explanation that cleaner labels make the measured gain more reliable. The claim is not internally contradictory—aligning with annotators is the paper's stated objective—but the external significance of the accuracy gain rests entirely on WSBS label reliability, which is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Annotators' Instruction Assisted Prompting (AIAP), which inserts the exact annotation instruction given to human annotators (definition, grounding, and example) into the base classification prompt for financial sentiment analysis. The authors introduce a new WallStreetBets sentiment dataset (WSBS) with two versions (Full and All-Agree) and evaluate FinGPT-SA, Llama-3, and GPT-4 in inference mode across three identifier terms ('news', 'tweet', 'input'). They report an average accuracy gain of 5.90% and a best-case gain of 9.08% over a base prompt, with a component ablation showing incremental improvement. They also compare against few-shot prompting, show a near-null result on Financial PhraseBank, and apply the sentiment predictions to stock price prediction using a quantitative score (QuantSS) and a proposed confidence-score-based method (CSBS). The main quantitative claims are that AIAP consistently improves LLM sentiment accuracy on WSBS and that CSBS improves stock prediction over baselines in some settings.","tokens_in":14173,"tokens_out":4931,"duration_ms":49587,"significance":"If the accuracy gains are robust, the paper offers a lightweight and reproducible intervention (adding annotator instructions to prompts) that could improve financial sentiment evaluation, along with a new public dataset. The consistency of gains across 18 model-dataset-identifier cells, the monotonic component ablation, and the negative control on Financial PhraseBank are strengths that go beyond a single anecdotal result. However, the external significance of the central claim rests entirely on the reliability of the WSBS gold labels, and the paper provides no inter-annotator agreement statistic. Because the AIAP prompt contains the very instruction used to create those labels, the measured gain could largely reflect rubric-matching on a self-consistent label set rather than improved sentiment understanding. The application section also shows mixed stock-prediction results that are more suggestive than conclusive. The central idea is plausible and worth publishing after the label-reliability issue is addressed.","major_comments":[{"comment":"The load-bearing assumption of the paper is that the WSBS majority-vote labels are a reliable ground truth for financial sentiment, but no inter-annotator agreement statistic (e.g., Fleiss' kappa, Krippendorff's alpha) is reported. The subsection title is 'Annotators' Agreement,' yet only the proportion of unanimous first-pass labels (1509/2920, 51.7%) is given, and the conflict-resolution stage is described incompletely. Since AIAP inserts the exact annotation instruction from Figure 1 into the prompt, the model is handed the labeling rubric at test time. Without a measure of label reliability, the reported gains of +5.90% average and +9.08% best case may reflect the model reproducing that specific rubric on a label set whose consistency is unknown. A concrete test would be to compute agreement on the first-pass double annotations and, if it is low, to re-evaluate on a held-out set of independently produced labels.","section":"§2.1, §2.2, Table 2"},{"comment":"There is a substantive inconsistency in the model used: §3.1 lists 'Meta-Llama-3-70B-Instruct' as the Llama-3 model, while Table 2's header and the surrounding text refer to 'Meta-Llama3-8B-Instruct.' This matters because the claims about 'Llama-3' gains (e.g., +7.56% on All-Agree with 'news') are attributed to a model family without specifying which variant was actually run. If the experiments used the 8B model, the statements about Llama-3 should say so explicitly, and if the 70B model was used, the table header must be corrected. The paper should clarify this before the results can be interpreted.","section":"§3.1 vs. §4, Table 2"},{"comment":"No significance tests, confidence intervals, or repeated runs are provided for any of the 18 accuracy comparisons. The headline average (+5.90%) and best case (+9.08%) are point estimates, and the best case is the maximum over three identifier terms, which should be flagged as a selection over multiple comparisons. Although the consistency of all positive gains is encouraging, a paired test (e.g., McNemar's test) or bootstrap confidence intervals for at least the main model–dataset comparisons would establish whether the gains exceed sampling noise, especially for the smaller All-Agree subset.","section":"§4, Table 2"},{"comment":"The stock-prediction claims are only partially supported by the reported numbers. The paper states that CSBS 'can extract more value out of models that were not trained on WSB's texts compared to the quantitative approach,' but Table 6 shows that for BERT-FPB the average RMSE improvement is nearly identical between CSBS (2.55) and QuantSS (2.39), and for several individual stocks the differences are negative. The conclusion that BERT-WSBS has 'the most advantage' over other fine-tuning datasets is also based on small and variable margins. The application section should be framed more cautiously, with the per-stock variability acknowledged as limiting the strength of the CSBS claim.","section":"§5.4, §5.5, Table 6"}],"minor_comments":[{"comment":"The abstract states 'improvements up to 9.08' without a percent sign; also, §1 says the dataset and code 'will be released upon acceptance' while the abstract says the dataset is 'available at' a GitHub URL. These statements should be reconciled.","section":"Abstract and §1"},{"comment":"The annotation process is described as 'two different annotations from different annotators' in the initial stage, but the reader is not told how the 2,920 samples were split between the two stages or how 'majority voting' operated when only two annotators examined a sample. Please clarify the mechanics of the conflict-resolution step.","section":"§2.1"},{"comment":"The few-shot comparison is performed only on FinGPT with one identifier ('news') and up to three shots; the conclusion that 'few-shot prompting cannot improve performance' is too strong given this limited coverage. It would be more precise to say that these few-shot configurations did not improve FinGPT's performance.","section":"§4.2, Table 3"},{"comment":"The claim that AIAP is not a 'lottery prompt' is supported by the FPB null result, but the FPB test uses only FinGPT and the best-case improvement from 86.0% to 86.4% is within typical run-to-run variation. The sentence should note that this is a single-model best case, not a rigorous falsification test.","section":"§4.3"},{"comment":"There are several typos and formatting issues, including 'Pubished' in the affiliation footer, inconsistent capitalization of 'Tweet' in Figure 2, 'FInancial' in reference [8], and the unspecified claim that GPT-4 has '1.76 trillion parameters' without a citation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the dataset could be a useful resource, but the referee needs to be convinced that the WSBS labels are reliable before the headline accuracy numbers are taken at face value. The Llama-3 model inconsistency is easy to fix but currently undermines interpretability. The paper might be better positioned at a workshop focused on LLM evaluation or financial NLP; the stock-prediction application, while interesting, is not yet strong enough to be a central contribution. No concerns about novelty disclosure beyond the dataset release statement being inconsistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the AIAP result is probably real as a prompt-engineering effect, but the headline number is only as good as the WSBS labels, and the paper doesn't quantify label reliability. The stress-test note lands.\n\nWhat's new and good: the WSBS dataset (clearly defined annotation instruction, public release) is a genuine asset, and the evaluation design is better than most. All 18 model-dataset-identifier cells improve, the component ablation is monotonic in most cases, few-shot prompting doesn't explain the gain, and the Financial PhraseBank null result is a useful control. The CSBS scoring idea for turning classifier confidence into a sentiment index is a small but reasonable addition to the stock prediction toolkit.\n\nSoft spots, in order of importance. First, the circularity burden is real: the AIAP prompt is literally the annotation instruction from Figure 1, and the labels are the product of that instruction. The gains on WSBS are compatible with the model learning to reproduce the annotation rubric on a self-consistent label set. The All-Agree subset showing larger gains only sharpens the point. The paper reports no inter-annotator agreement; 48% of labels came from majority vote with an underspecified conflict-resolution stage. The paper itself attributes some prompt-ablation drops to 'ambiguity of the annotations' in Section 4.1, an implicit admission that label noise is live. A kappa or similar reliability measure, plus an analysis on a held-out set labeled independently, would go a long way. Second, there are no significance tests or error bars anywhere; with 18 positive cells the sign pattern is suggestive, but the magnitudes (2.47 to 9.08) vary a lot. Third, the stock prediction section is honestly mixed: GME/AMC improve, AAPL/SPY don't, and the paper says so, but the negative results are tucked into Table 6 and the narrative spends more time on the wins. Also minor: Section 3.1 says Llama3 70B, Table 2 caption says 8B; abstract says 'up to 9.08' without percent.\n\nWho this is for: people working on financial sentiment evaluation or WSB-specific text; it's a solid workshop-level contribution with a usable dataset. The core claim needs a reliability check before I'd trust the numbers as strong evidence.\n\nRecommendation: send to peer review if the venue has a revision path; the dataset and control experiments merit referee time. The author should be asked for inter-annotator agreement, significance testing, and a clear statement that gains are relative to the annotation instruction, not absolute sentiment understanding.","headline":"A real prompt-engineering effect, but the WSBS labels are the load-bearing wall and they're unmeasured.","tokens_in":14665,"tokens_out":3150,"would_cite":false,"duration_ms":27545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that inserting the human annotators' own instruction sheet into an LLM prompt improves financial sentiment classification accuracy by 5.90 percentage points on average and 9.08 points at best.","keywords":["financial sentiment analysis","large language models","annotators' instruction assisted prompt","prompt design","WallStreetBets","stock price prediction","confidence-score sentiment index","evaluation benchmarking"],"falsifier":"Compare AIAP's accuracy gain on the 1,509 WSBS samples with unanimous first-pass labels against its gain on the samples resolved by majority vote; if the gain is concentrated in the conflict-resolved samples, the prompt is teaching the label-resolution rule rather than improving sentiment judgment. A second check: rewrite the instruction with the same definitions but different examples; if the gain disappears, the examples, not the task definition, are doing the work.","tokens_in":13736,"feed_emoji":"📈","tokens_out":13300,"duration_ms":122163,"temperature":0.7,"pith_summary":"The paper's central claim is that much of the apparent weakness of large language models on financial sentiment tasks is an evaluation artifact: models are asked to guess positive, negative, or neutral labels without being told how the human annotators defined those classes. To fix this, the paper introduces the Annotators' Instruction Assisted Prompt (AIAP), which inserts the full annotation instruction—definition, market grounding, and one example per class—into the prompt before the model classifies text. On a new WallStreetBets-derived dataset called WSBS, this one prompt change raises accuracy by 5.90 percentage points on average and 9.08 points in the best configuration across three instruction-tuned LLMs. The paper further shows that sentiment features built from these predictions, including a confidence-score-based index, improve next-day stock price prediction for several heavily discussed tickers. If the result holds, standard financial sentiment benchmarks have been understating LLM ability, and a cheap, model-agnostic prompt alignment can recover much of the gap.","feed_headline":"Same annotator instruction sheet lifts LLM sentiment accuracy 9.08%","feed_subtitle":"Across three LLMs the prompt change adds 5.90 percentage points on average; the best gain is 9.08.","key_machinery":"The central object is the Annotators' Instruction Assisted Prompt (AIAP): a template that takes a minimal classification request ('What is the sentiment of this input? Please choose an answer from negative/neutral/positive') and appends the three-part instruction sheet given to WSBS annotators—a definition for each sentiment class, a grounding that aligns positive/negative/neutral with bullish/bearish/stable market language, and one unambiguous example per class. It works as an in-context task re-definition, turning an under-specified sentiment question into one whose label semantics match the gold annotations. The supporting machinery is the WSBS dataset, whose gold labels were produced under that same instruction, plus a confidence-score-based sentiment scoring method (CSBS) that converts classifier probability outputs into daily sentiment features for stock prediction.","core_discovery":"The central claim is that a frozen LLM's financial sentiment judgments converge with human annotators' judgments when the model is handed the same written task definition the annotators used. The demonstration rests on WSBS, a dataset of 2,920 WallStreetBets posts and comments labeled by five annotators under a three-part instruction—definition, market grounding, and one example per class—with 1,509 samples receiving unanimous first-pass labels. Across FinGPT, Llama-3, and GPT-4, the Annotators' Instruction Assisted Prompt (AIAP) beats a minimal base prompt on both the unanimous and full subsets; the average gain is 5.90 percentage points, the largest single gain is 9.08 percentage points, and the best overall accuracy rises from 74.95% to 80.91%. Stepwise additions of the three instruction components raise accuracy in most configurations, while ordinary few-shot prompting with the same examples does not match AIAP, and applying the dataset-specific AIAP to a different financial news benchmark yields little gain. The paper takes this as evidence that the improvement transfers the annotators' label semantics rather than coming from a lucky prompt phrasing.","pith_inferences":["A natural boundary the paper does not fully close: because the same instruction produced the gold labels and the AIAP prompt, part of the measured gain could be protocol mimicry rather than deeper sentiment understanding; a cleaner control would compare AIAP with a differently worded instruction that preserves the same label meanings.","The grounding component ties each label to expected market direction, so the prompt may reward models for detecting directional cues rather than affective tone; a testable extension would replace 'bullish/bearish/stable' with neutral wording and measure the drop.","The confidence-score sentiment index (CSBS) is a general recipe—neutral predictions get a signed weight from the positive-minus-negative probability gap—so it could be applied to any classifier on other text streams to see whether the stock-prediction gains replicate outside WallStreetBets.","If the same annotator-prompt alignment pattern appears on other subjective tasks such as stance or emotion, AIAP would generalize from a financial benchmark trick into a general evaluation principle: give the model the labeler's written definition, not a generic paraphrase."],"forward_implications":["Evaluation practice for financial sentiment benchmarks should change: when annotator instructions exist, they belong in the prompt, or benchmark scores will keep understating LLM ability.","The effect is not tied to one model: every tested instruction-tuned LLM improves with AIAP, with the largest single gain reaching 9.08 percentage points on the All-Agree WSBS subset.","The component ablation shows the instruction is doing real work—definition alone helps, grounding adds more, and the example completes the effect—while plain few-shot prompting with the same examples does not reproduce the gain.","Frozen models can be improved without retraining: AIAP is purely a prompt change, so any instruction-following model can adopt it in evaluation or production.","Stock prediction gains are concentrated where WallStreetBets attention is concentrated: sentiment features reduce next-day price prediction error most for GME and AMC, and far less for a broad index or a thinly discussed stock."],"supporting_citations":[{"why":"Establishes the benchmark subjectivity that motivates AIAP; also serves as the control dataset where AIAP gives little gain.","marker":"[26]"},{"why":"Supplies the FinGPT instruction-tuned model used across the main accuracy experiments and in stock-prediction sentiment scoring.","marker":"[43]"},{"why":"Supplies the Llama-3 instruction-tuned model used as the open-source generic LLM in the accuracy comparison.","marker":"[7]"},{"why":"Supplies the GPT-4 model used as the closed-source generic LLM in the accuracy comparison.","marker":"[28]"},{"why":"Provides the few-shot prompting baseline against which AIAP is compared to show examples alone do not explain the gain.","marker":"[3]"},{"why":"Defines the 'lottery prompt' phenomenon that the cross-model and cross-dataset checks are designed to rule out.","marker":"[5]"},{"why":"Provides the quantitative sentiment index baseline (QuantSS) used for stock prediction comparison.","marker":"[12]"}],"fun_headline_variants":["Same annotator instruction sheet lifts LLM sentiment accuracy 9.08 points","LLM financial sentiment jumps 9.08 points when given annotators' instructions","AIAP prompt: +5.90 avg points on LLM financial sentiment, best 9.08","Annotator instruction prompt: 9.08-pt LLM sentiment gain, better stock predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the WSBS gold labels are a reliable measure of financial sentiment; if those labels are noisy or already encode the instruction, the measured gains may reflect the model replaying the annotation rule instead of understanding sentiment better.","fun_headline_variants_meta":{"raw":{"variants":["Same annotator instruction sheet lifts LLM sentiment accuracy 9.08 points","LLM financial sentiment jumps 9.08 points when given annotators' instructions","AIAP prompt: +5.90 avg points on LLM financial sentiment, best 9.08","Annotator instruction prompt: 9.08-pt LLM sentiment gain, better stock predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00135,"raw_usage":{"total_tokens":5553,"prompt_tokens":1087,"completion_tokens":4466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":4371}},"tokens_in":703,"tokens_out":4466,"duration_ms":28543,"temperature":1.0,"reasoning_tokens":4371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:43:29.647362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare AIAP's accuracy gain on the 1,509 WSBS samples with unanimous first-pass labels against its gain on the samples resolved by majority vote; if the gain is concentrated in the conflict-resolved samples, the prompt is teaching the label-resolution rule rather than improving sentiment judgment. A second check: rewrite the instruction with the same definitions but different examples; if the gain disappears, the examples, not the task definition, are doing the work.","supporting_citations":[{"cited_title":"Exploring Lottery Prompts for Pre-trained Language Models","cited_arxiv_id":"2305.19500","evidence_quote":"Defines the 'lottery prompt' phenomenon that the cross-model and cross-dataset checks are designed to rule out."}],"review_version":1}