{"id":"3b4db1c4-0e16-49c4-a55f-03c40c2d2de5","arxiv_id":"2506.23767","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"TinyXRA, a TinyBERT-based hierarchical attention model trained with triplet loss, claims state-of-the-art risk ranking for 10-K reports on volatility, skewness, kurtosis, and the Sortino ratio.","lead":"The paper introduces TinyXRA, a lightweight transformer model that reads the narrative section of annual U.S. company reports (10-Ks) and ranks companies by four stock-return risk statistics. The authors say it beats prior explainable risk-ranking models while also producing word clouds that show which terms drive the ranking.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Triplet loss and evaluation orient skewness and Sortino backward: higher bins are favorable per §7, so positive Spearman/Kendall for these measures may reflect raw measure order, not risk.","rationale":"The reader's weakest assumption identifies the most load-bearing flaw: the triplet loss and evaluation metrics treat label 2 as high risk for all four measures, while Section 7 explicitly reverses this for skewness and Sortino. This directly undermines the paper's central claim of 'comprehensive risk assessment,' because for two of the four measures the model is trained to rank safer companies as riskier and the reported correlation metrics are computed against the wrong ordering. If this concern is correct, the favorable results for skewness and Sortino do not demonstrate risk-assessment ability at all; they merely show correlation with the raw measure's percentile. The TF-IDF competitiveness and the code-availability inconsistency are additional issues, but they are secondary: TF-IDF's strong performance could be discussed as a baseline that challenges SOTA, whereas the label inversion makes the evaluation itself conceptually invalid for a large portion of the study. The paper does provide substantial engineering, detailed experiments, and a clear architecture, but this orientation error is fundamental and cannot be patched by a footnote. The proposed sign-flip test is a cheap and decisive check that would confirm the concern. Given the reader's rejection is well-supported, my assessment does not change the verdict.","tokens_in":40193,"tokens_out":5057,"duration_ms":55164,"concrete_test":"Recompute Spearman's Rho and Kendall's Tau for the skewness and Sortino test sets with ground-truth labels flipped to risk = 2 - bin (so bin 0 becomes high risk, bin 2 becomes low risk) while keeping TinyXRA's logits unchanged. If the correlations become negative or drop to near zero, the reported positive values are an artifact of the raw label orientation, confirming that the model ranks high skewness and high Sortino as high risk, contrary to §7.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of comprehensive risk assessment depends on the model ranking companies by actual financial risk, not merely by the raw value of each risk measure. For skewness and the Sortino ratio, however, the paper's own Section 7 states that a higher bin is generally more favorable to investors, meaning bin 2 is the low-risk group and bin 0 is the high-risk group. The triplet loss in Eq. 19 treats label 2 as the positive (High Risk) sample and label 0 as the negative (Low Risk) sample, forcing the model to assign higher logits to high skewness and high Sortino values. Consequently, for these two measures the model is trained to rank safer firms as riskier. The reported Spearman's Rho and Kendall's Tau are computed against ground-truth labels 0, 1, 2 without reversing the order for skewness and Sortino, so positive correlations only indicate agreement with the raw measure's percentile, not with risk. This is not a mere labeling preference: it inverts the risk semantics for half of the four tasks that constitute the 'comprehensive' assessment, and it undermines the validity of the favorable results in Tables 4 and 6. If the labels are reversed, the model may be actively anti-correlated with true risk on these measures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TinyXRA, a lightweight hierarchical transformer for financial risk assessment from the MD&A sections of 10-K reports. Risk labels are derived from four stock-return measures (standard deviation, skewness, kurtosis, Sortino ratio), discretized into three percentile bins. The model uses TinyBERT sentence embeddings, a bidirectional GRU with sentence-level attention, and a triplet ranking loss, and it provides explanations through attention-based word clouds and heatmaps. The authors report experiments on a rolling-origin design over 2018–2024 and claim state-of-the-art predictive accuracy across all risk measures and years, together with faithfulness evaluations of the attention explanations.","tokens_in":40392,"tokens_out":5237,"duration_ms":50993,"significance":"If the claims were supported, TinyXRA would be a practically useful contribution: a lightweight, explainable model for financial risk ranking with competitive accuracy and transparent word- and sentence-level explanations. The paper includes a broad experimental setup, seven test years, multiple baselines, and public code. However, the central SOTA claim is contradicted by the paper's own tables, where TF-IDF and sometimes XRR outperform TinyXRA on several metrics and years. In addition, the triplet loss and evaluation metrics are oriented inconsistently with the risk semantics for skewness and the Sortino ratio, as acknowledged in Section 7. These issues undermine the two main contributions claimed in the abstract and introduction.","major_comments":[{"comment":"The claim that TinyXRA achieves state-of-the-art predictive accuracy across seven test years is contradicted by the paper's own results. For instance, TF-IDF beats TinyXRA on F1 for standard deviation in 2021 (75.4 vs 74.1) and 2022 (75.8 vs 75.1) in Table 3; on skewness F1 in 2024 (46.3 vs 44.4) and 2018 (49.1 vs 46.0) in Table 4; and on kurtosis Spearman in 2023 (38.0 vs 35.2) and 2018 (39.4 vs 39.0) in Table 5. Table 6 shows TF-IDF winning several Sortino cells as well. The paper itself acknowledges in §5.3.2 that TF-IDF 'performs remarkably well, particularly under the skewness and kurtosis measures, consistently ranking close to or better than more advanced neural models.' The consolidated Cohen's d values in Table 9 aggregate over all risk measures and years, which masks these per-cell losses and cannot support the unqualified statement that TinyXRA 'consistently demonstrates superior predictive performance' (§5.3.2) or the abstract's 'state-of-the-art' claim. This is a load-bearing discrepancy between the stated contribution and the reported evidence.","section":"§5.3.2, Tables 3–6, Table 9"},{"comment":"The triplet loss in Eq. (19) treats label 2 as the 'High Risk' positive sample and label 0 as the 'Low Risk' negative sample for all four risk measures. Section 7, however, states that for skewness and the Sortino ratio, a higher bin is generally more favorable to investors. Because the bins are defined as percentile ranges of the raw measure (§3.3), label 2 corresponds to the lowest-risk firms for these two measures. Training with the triplet loss as written therefore forces the model to assign higher logits to safer firms on two of the four tasks. The Spearman and Kendall correlations in Tables 4 and 6 are computed against the numeric labels 0, 1, 2 without reversing the order for skewness and Sortino, so a positive correlation indicates agreement with the raw measure's percentile, not with financial risk. This inverts the risk semantics for half of the 'comprehensive' assessment and invalidates the favorable results reported for skewness and Sortino. The authors need to either reverse the label order for these measures in the loss and evaluation, or provide a clear justification for treating higher skewness and higher Sortino as higher risk, which would contradict the paper's own Section 7.","section":"§4.4.2, Eq. (19); §3.3; §7"}],"minor_comments":[{"comment":"The faithfulness test removes the top-k% attended words/sentences and measures performance drop. Because the same attention scores are used both to select the removed elements and to compute the document representation, the test is partially self-referential. Adding a control condition that removes the lowest-attended elements (or random elements) would strengthen the claim that the observed degradation is due to the importance of the attended content rather than simply the reduction of input length.","section":"§6.3, Appendix C"},{"comment":"The paper describes the task as 'risk quartile classification' in the abstract and as 'quartiles' in §4.4, but the actual binning is three percentile groups (0–30%, 30–70%, 70–100%). Please use consistent terminology, such as 'terciles' or 'percentile bins', throughout.","section":"§3.3, Table 1"},{"comment":"The description of the F1 metric in §5.1.1 says Micro F1 is equivalent to accuracy, but the paper only reports Macro F1. Please clarify how the predicted logits are converted to discrete bins for F1 computation, including the exact thresholds used, since this affects the interpretation of the reported F1 values.","section":"§5.1.1, §5.3.2"},{"comment":"The hyperparameter table lists 'Qwen2.5-0.5B' while the main text and experiment descriptions use 'Qwen2-0.5B'. Please reconcile the naming to avoid confusion about which model was actually evaluated.","section":"Appendix E, Table 10"},{"comment":"The limitations section candidly notes that earlier years are affected by missing data for delisted or merged companies and by the use of the latest CIK-to-ticker mapping. These issues likely bias the earlier test years, yet the experimental analysis in §5.3.2 does not address or quantify this potential bias. A sensitivity analysis or explicit discussion of how the data gaps affect the 2018–2020 results would be helpful.","section":"§8.3"}],"recommendation":"reject","confidential_remarks":"The paper's own tables contradict the headline SOTA claim, and the triplet-loss orientation in Eq. (19) is internally inconsistent with the risk semantics stated in Section 7 for skewness and the Sortino ratio. These are not minor presentation issues: they affect the validity of the main empirical results and the central contribution. The authors would need to substantially revise the experimental setup and re-evaluate the claims, which goes beyond a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent engineering paper with a load-bearing flaw. The authors build TinyXRA, a lightweight hierarchical transformer for ranking firms by four risk measures from 10-K MD&A text, and add a nice attention-based word cloud. The empirical apparatus is substantial: seven test years, five seeds, ablations, faithfulness checks. If the task were well-posed, this would be a useful contribution to financial text analytics.\n\nThe problem is that the task is not well-posed for two of the four measures. The paper states in Section 7 that for skewness and the Sortino ratio, a higher bin is more favorable to investors. Yet Eq. 19 defines the triplet loss with the positive sample drawn from label 2 ('High Risk') and the negative from label 0 ('Low Risk') for all risk measures. So on skewness and Sortino, the model is trained to assign higher logits to safer firms. The reported Spearman and Kendall correlations are computed against the original 0/1/2 labels, so a positive correlation only shows agreement with the raw measure's percentile, not with risk. Unless the authors flipped the labels at training time (the text gives no indication they did), the favorable numbers in Tables 4 and 6 for these measures do not mean what the abstract claims.\n\nThere is a second, independent issue: the SOTA claim is contradicted by the paper's own tables. TF-IDF beats TinyXRA on skewness F1 in five of seven years, and on kurtosis Spearman in four of seven years. The Cohen's d analysis is pooled across measures and years, which hides these reversals. Claiming 'state-of-the-art predictive accuracy across seven test years' while TF-IDF wins several cells is not defensible.\n\nTwo smaller issues worth noting. The rolling-origin split may not fully prevent leakage: training labels for, say, 2017 filings use returns that extend into 2018, which overlaps with the 2018 test window. And the abstract promises code at a GitHub URL while the body says code and data will be released upon acceptance; that inconsistency should be fixed.\n\nWhat is genuinely new is the specific combination—TinyBERT with a GRU and hierarchical attention for multi-measure risk ranking, plus the attention-weighted word clouds with group-discriminative filtering. The word cloud idea is a reasonable novelty, and the faithfulness experiments (removing high-attention words/sentences) are good practice. But none of that rescues the central claim.\n\nBottom line: the paper should not be accepted as is. The label-orientation error is likely fixable—reverse the ordering for skewness and Sortino, rerun the experiments, and re-examine the temporal split. If the authors do that, the result might be a solid contribution. As it stands, I would not cite it without a major caveat, but I would send it to review: the issue is substantive and the empirical base is broad enough that a referee can give actionable guidance.","headline":"Solid engineering, but the triplet loss trains the model to rank favorable firms as risky on two of four measures, so the headline claims do not survive contact with the paper's own definitions.","tokens_in":40973,"tokens_out":3086,"would_cite":false,"duration_ms":33101,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TinyXRA, a lightweight hierarchical transformer, claims to rank U.S.","keywords":["TinyXRA","Explainable AI","financial risk assessment","10-K reports","hierarchical transformer","triplet ranking loss","attention word cloud","risk measures"],"falsifier":"Rerun the training and evaluation for skewness and Sortino with the risk order reversed in the triplet loss—treating low skewness and low Sortino as the high-risk end—and compare Spearman's rho and Kendall's tau with the paper's reported values; if the positive correlations collapse or change sign, the central accuracy claim for those two measures is an artifact of the label mapping.","tokens_in":39894,"feed_emoji":"📊","tokens_out":8711,"duration_ms":94593,"temperature":0.7,"pith_summary":"This paper sets out to show that a small, efficient transformer can take the Management's Discussion and Analysis (MD&A) section of a company's 10-K filing and rank the company's financial risk on four measures—standard deviation, skewness, kurtosis, and the Sortino ratio—rather than relying on volatility alone. The motivation is practical: financial institutions often lack the hardware for large language models and are required to explain their risk decisions, so a lightweight model with transparent attention is valuable if it can match or beat heavier approaches. The authors report that TinyXRA achieves top-two performance on F1, Spearman's rho, and Kendall's tau across seven test years (2018–2024) against XRR, TF-IDF, and two small LLMs, and that removing the words and sentences it attends most to degrades accuracy, which they read as evidence that the explanations are faithful. A central design choice is triplet ranking loss, which enforces that high-, medium-, and low-risk documents are separated with a margin, improving over pairwise ranking.","feed_headline":"Lightweight transformer ranks 10-K risk ahead of larger models","feed_subtitle":"It runs on a consumer GPU, reads annual-report prose, ranks risk four ways, and shows the telling words.","key_machinery":"The machinery is a hierarchical attention network: TinyBERT produces sentence embeddings and word-level attention from the [CLS] token, a bidirectional GRU adds sentence context, a sentence-level attention layer weights sentences, and the weighted sum becomes a document vector scored by a linear layer. A triplet ranking loss with margin 0.1 pulls high-risk scores above medium-risk scores and medium above low, using anchors sampled from the medium group. For explanations, a dynamic word cloud multiplies word-level and sentence-level attention, applies a per-document quantile threshold, and subtracts a token's maximum frequency in other risk groups so only group-distinctive terms remain.","core_discovery":"The central claim is that TinyXRA can replace volatility-only risk assessment with a single architecture that reads prose and predicts a richer risk profile, while staying small enough to run on an 11GB-VRAM GPU and producing explanations that survive quantitative scrutiny. The paper argues that interpretability does not cost accuracy: the attention weights that explain predictions are also the mechanism that lets the model outperform the prior state-of-the-art XRR, and the contextual embeddings from TinyBERT outperform XRR's static embeddings. It further claims that the triplet loss captures both direction and magnitude of risk differences, giving a real but modest gain over pairwise loss, and that the attention word clouds, built from sentence-level and word-level attention with group-specific filtering, isolate risk vocabulary that aligns with economic conditions, such as 'deposits' and 'dividends' appearing in low-risk word clouds during a rising-rate year.","pith_inferences":["The label-ordering choice for skewness and Sortino is a testable extension: re-running with the risk order reversed would show whether the positive correlations reported for those two measures survive.","The word clouds could be turned into an early-warning monitor: year-over-year shifts in group-distinctive terms might flag emerging systemic risks before balance-sheet data reflect them.","Since the architecture is not finance-specific beyond its labels, the same lightweight hierarchical design could be applied to other long regulatory filings, such as 10-Q reports, with minimal adaptation."],"forward_implications":["Financial firms with a consumer-grade GPU could screen thousands of 10-K filings in real time, because the encoder is TinyBERT and the design runs within 11GB VRAM.","Risk assessment can move beyond volatility: the same model outputs rankings on downside-focused measures such as the Sortino ratio and distribution-shape measures such as skewness and kurtosis.","Attention-based explanations can be treated as faithful: removal experiments show that deleting the most-attended words or sentences degrades F1, Spearman, and Kendall consistently across years.","Triplet ranking loss is preferable to pairwise and cross-entropy losses for ordinal risk bins, improving F1 with small but non-negligible effect sizes.","The seven-year rolling-origin results indicate the model generalizes across different market regimes, including the COVID-19 shock in 2020 and the post-COVID recovery."],"supporting_citations":[{"why":"Supplies the XRR baseline, the prior state-of-the-art that TinyXRA compares against, and the pairwise-ranking design that the triplet loss replaces.","marker":"Lin et al. 2021"},{"why":"Defines the factor model whose residuals are used to compute all four risk labels.","marker":"Fama and French 1993"},{"why":"Provides the TinyBERT encoder that makes the model lightweight and context-aware.","marker":"Jiao et al. 2020"},{"why":"Introduces the hierarchical attention architecture that TinyXRA extends with transformer multi-head attention and contextual embeddings.","marker":"Yang et al. 2016"},{"why":"Introduces the deep ranking / triplet formulation that TinyXRA adapts for ordinal risk bins.","marker":"Wang et al. 2014"},{"why":"Establishes the transformer attention mechanism that underlies both the sentence-level and word-level explanation scores.","marker":"Vaswani et al. 2017"}],"fun_headline_variants":["TinyXRA: tiny transformer for comprehensive 10-K risk","Lightweight AI reads 10-Ks to rank risk beyond volatility","Small model beats larger ones on 10-K risk prediction","Explainable transformer uses triplet loss for risk quartiles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that a higher bin always means higher risk when building triplets, but for skewness and the Sortino ratio a higher value is better for investors, so the model may be trained to rank safer firms as riskier on two of the four tasks.","fun_headline_variants_meta":{"raw":{"variants":["TinyXRA: tiny transformer for comprehensive 10-K risk","Lightweight AI reads 10-Ks to rank risk beyond volatility","Small model beats larger ones on 10-K risk prediction","Explainable transformer uses triplet loss for risk quartiles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1741,"prompt_tokens":1006,"completion_tokens":735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":622,"tokens_out":735,"duration_ms":8511,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:32:29.422622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the training and evaluation for skewness and Sortino with the risk order reversed in the triplet loss—treating low skewness and low Sortino as the high-risk end—and compare Spearman's rho and Kendall's tau with the paper's reported values; if the positive correlations collapse or change sign, the central accuracy claim for those two measures is an artifact of the label mapping.","supporting_citations":[{"cited_title":"Findings of the Association for Computational Linguistics: EMNLP 2020, 4163--4174","cited_arxiv_id":null,"evidence_quote":"Provides the TinyBERT encoder that makes the model lightweight and context-aware."},{"cited_title":"Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, 1480--1489","cited_arxiv_id":null,"evidence_quote":"Introduces the hierarchical attention architecture that TinyXRA extends with transformer multi-head attention and contextual embeddings."},{"cited_title":"Proceedings of the IEEE conference on computer vision and pattern recognition, 1386--1393","cited_arxiv_id":null,"evidence_quote":"Introduces the deep ranking / triplet formulation that TinyXRA adapts for ordinal risk bins."},{"cited_title":"Advances in neural information processing systems 30","cited_arxiv_id":null,"evidence_quote":"Establishes the transformer attention mechanism that underlies both the sentence-level and word-level explanation scores."}],"review_version":1}