{"id":"592633e9-f74a-4832-bfaf-d705c8a98373","arxiv_id":"2411.09249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Composing a general Japanese instruction model with a finance-specialized model via CALM cross-attention improves Japanese financial benchmark scores beyond LoRA, even when trained on a different finance dataset.","lead":"Researchers connected a general Japanese language model to a finance-specialized model with a small trainable cross-attention layer and report higher scores on a Japanese financial benchmark than either model alone or LoRA fine-tuning. The result suggests a cheap way to give large language models financial knowledge, but the evaluation benchmark was built by the same team that made the finance model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative claim stands on a self-authored benchmark (pfmt-bench-fin-ja) sharing authors with the augmenting model; without independent evaluation, CALM's margin over LoRA is not established.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the only benchmark used for the headline result was authored by the same Preferred Networks researchers who built the augmenting model and are co-authors of this paper. Because that benchmark and the proprietary training data of the augmenting model cannot be independently inspected, the quantitative comparison is vulnerable to unrecognized alignment between benchmark design and the augmenting model's capabilities. The architecture itself is clearly described and follows the published CALM formulation, so I do not see an internal mathematical flaw; the issue is empirical validity. The lack of error bars and the presence of many zero scores compound the uncertainty, since a 0.19 average margin over LoRA may be driven by a small number of tasks and a non-blind GPT-4 judge. I therefore agree with the conditional verdict: the paper should not be fully accepted until an independent evaluation settles whether the advantage is real. If the external test confirms the margin, the concern is resolved; if not, the claims should be narrowed.","tokens_in":10682,"tokens_out":4957,"duration_ms":69034,"concrete_test":"Re-run the full comparison (anchor-only, augmenting-only, LoRA, CALM) on an externally maintained Japanese financial benchmark not authored by any co-author, ideally public financial QA, extraction, or reasoning tasks from an independent institution, with 5 training seeds for LoRA and CALM and report mean ± standard error. The central claim is supported only if CALM's mean exceeds LoRA's mean by a margin that survives a paired significance test at p < 0.05; otherwise, the advantage over LoRA should be treated as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical conclusion (Section V-A, Table I) is that CALM scores 0.711 on pfmt-bench-fin-ja versus 0.521 for LoRA and 0.043 for the anchor. The evaluation is described in Section IV-D and uses benchmark [26], which shares two authors (Hirano and Imajo) with augmenting model [3] and with this paper. The augmenting model's pre-training corpus is proprietary, so an external user cannot audit whether the benchmark's 12 categories, dialogue format, or GPT-4 scoring protocol are aligned with that model's strengths. The reported averages also lack error bars and significance tests, and the paper itself notes zero or near-zero scores on many tasks due to early end-of-text generation. If GPT-4 as judge rewards fluency or response length, CALM's longer and more structured outputs could inflate its margin regardless of financial knowledge. This is a benchmark-validity risk, not an allegation of misconduct, but it is load-bearing because the paper's only quantitative evidence for the headline improvement depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies Composition to Augment Language Models (CALM) to adapt a Japanese general-purpose instruction-tuned model to the financial domain. The anchor model is nekomata-14b-instruction and the augmenting model is nekomata-14b-pfn-qfin, a financial-specialized model; cross-attention connections between the two are trained on the JAFIN instruction dataset, which is distinct from the proprietary corpus used to build the augmenting model. On the pfmt-bench-fin-ja benchmark, the CALM model achieves an average score of 0.711, compared with 0.521 for a LoRA-tuned anchor, 0.188 for the augmenting model alone, and 0.043 for the anchor alone. A single cross-attention connection at the middle layer (layer 20) outperforms connections at the head (layer 0) and tail (layer 38), leading the authors to conclude that CALM is an effective method for financial domain adaptation and that middle-layer connections are most effective.","tokens_in":10916,"tokens_out":9199,"duration_ms":79394,"significance":"If the quantitative results are reliable, the paper provides evidence that model composition via cross-attention can transfer financial knowledge without modifying the base model's weights, including when the training data differ from the augmenting model's pretraining corpus. The use of a distinct training dataset and a parameter-matched LoRA baseline are good experimental design choices, and the connection-point ablation is informative. However, the strength of the evidence is limited by the benchmark's overlap with the augmenting model's authors, the absence of repeated runs or significance tests, and the paper's own admission of near-zero scores on several task categories. These issues must be addressed before the headline claims can be accepted.","major_comments":[{"comment":"The benchmark pfmt-bench-fin-ja [26] is released by the same two researchers (Hirano and Imajo) who built the augmenting model [3] and are co-authors of this paper, and the augmenting model's training corpus is proprietary. This overlap means the benchmark may be implicitly aligned with knowledge contained in the augmenting model, so the reported advantage of CALM over LoRA (0.711 vs 0.521) may be inflated. The paper neither discloses this overlap nor provides any independent evaluation. Please add results on a public, independently authored benchmark (or a human evaluation with external raters) and discuss the potential for benchmark alignment.","section":"Section IV-D and V-A, Table I"},{"comment":"All quantitative results are point estimates from a single run. With 360 two-turn dialogues, the 0.19-point gap between CALM and LoRA could be within run-to-run noise, especially given the large variability implied by near-zero scores in several categories. Please provide multiple training runs, bootstrap confidence intervals over dialogue-level scores, and a paired significance test for the CALM versus LoRA comparison.","section":"Section V-A, Table I"},{"comment":"The paper reports that scores are zero or near zero in half of the remaining tasks because the models emit an early <endoftext> token, yet the abstract and conclusion state a general result of 'superior responses with higher scores.' The average score aggregates tasks on which CALM succeeds and tasks on which it produces no output, and the paper does not provide a per-task numerical table or an explicit list of the failed categories. Please include per-task scores and qualify the headline claim accordingly; if CALM fails completely on several categories, the contribution should be described as selective rather than general financial-domain adaptation.","section":"Section V-A, Fig. 1"},{"comment":"The LoRA baseline is described only as having a rank set to match the parameter count of CALM, with no statement of which linear modules were adapted, the LoRA alpha, dropout, or training details. This makes the LoRA result (0.521) difficult to reproduce and leaves open the possibility that the comparison is not parameter-for-parameter fair. Please provide the full LoRA configuration and, if possible, an additional LoRA baseline with a larger rank or different target modules to show that the conclusion is robust.","section":"Section IV-C"},{"comment":"The conclusion that middle-layer connections are most effective rests on a single comparison of one connection at layer 0, 20, or 38, with no variance estimates. Moreover, the paper does not explain how this single-connection result relates to the main CALM model that inserts connections every four layers, which reaches 0.711 while the best single connection reaches only 0.244. Please provide confidence intervals for the connection-point comparison and discuss the relationship between the single-connection finding and the multi-connection setting.","section":"Section V-A, Table II"}],"minor_comments":[{"comment":"The citation for DARE is incorrect: reference [21] is a paper on safety alignment, not the DARE method for model merging. Please cite the original DARE paper (e.g., Yu et al., 'Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch').","section":"Section II, Reference [21]"},{"comment":"The qualitative examples are presented as representative, but the augmenting model's zero <endoftext> responses in the examples contradict its nonzero average score of 0.1875 in Table I; please acknowledge that these examples are selected and provide a more systematic summary of response lengths and success rates across the benchmark.","section":"Section V-B"},{"comment":"The description of the JAFIN dataset says it was primarily collected from government websites and Wikipedia, but the paper does not discuss whether these sources overlap with the proprietary corpus used to build the augmenting model; a brief analysis of potential overlap would strengthen the claim that the training data are genuinely different.","section":"Section IV-B"},{"comment":"Please state the exact number of additional parameters introduced by CALM and by the LoRA baseline; the claim that the ranks are matched is not verifiable without these numbers.","section":"Section IV-C"},{"comment":"The GPT-4 evaluation protocol is underspecified: please report the exact prompt, decoding parameters, and whether the judge was run more than once per dialogue to assess stability.","section":"Section IV-D"},{"comment":"The phrase 'half of the remaining tasks' is ambiguous; please specify exactly which task categories have near-zero scores and which tasks contribute to the improvement over LoRA.","section":"Section V-A, Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark overlap is the most serious issue in this submission. If the authors cannot provide an independent evaluation, I would not be comfortable accepting the headline comparison. Please consider requiring explicit disclosure of the overlap and an independent benchmark or human evaluation in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, clearly written application paper. It doesn't invent a new mechanism—CALM is Bansal et al.'s composition method—but it shows CALM trained on a public financial instruction set (JAFIN) can compose a Japanese generalist anchor with a finance-specialized augmenting model when the training data is intentionally different from the augmenting model's proprietary corpus. The mid-layer connection result is a nice empirical datapoint. The authors also keep LoRA parameter-matched and are upfront that many tasks fail with endoftext.\n\nThe soft spot is exactly where the reader's report puts it. The quantitative claim, CALM 0.711 vs LoRA 0.521, comes from one run with no confidence interval or significance test, and the benchmark is pfmt-bench-fin-ja, authored by two of this paper's co-authors (also the augmenting model's creators), with no external audit. The augmenting model's training data is proprietary, so you can't rule out that the benchmark is implicitly aligned with its strengths. On top of that, GPT-4 is the judge and CALM's outputs are visibly longer and more structured, so part of the gap could be style. The absolute scores are also tiny—0.711 out of 10—so even the headline number is not practically impressive. Half the tasks are at or near zero, which the paper acknowledges.\n\nI don't think this is misconduct or sloppiness; it's a common evaluation-limitation pattern in applied LLM work. But because the entire advantage claim rests on this benchmark, the limitation is load-bearing, not cosmetic. An external benchmark or at least a second judge and repeated runs with error bars would be needed before I'd put weight on the LoRA comparison.\n\nWho is this for? Anyone working on Japanese financial dialogue models or on compositional adaptation of LLMs. It's a legitimate extension that deserves a serious referee—the method is reproducible in principle, the writing is clear, and the authors disclose the main failure mode. I'd send it to peer review, with the requirement that the evaluation be strengthened. I would not cite it in my own work until the result is independently confirmed.","headline":"A useful CALM application study whose headline result is undercut by a single-run, self-authored benchmark with near-zero scores in half the tasks.","tokens_in":11437,"tokens_out":2707,"would_cite":false,"duration_ms":26708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By connecting a general Japanese chat model to a finance-specialized model with one trainable cross-attention layer, this paper reports financial dialogue scores of 0.711, beating LoRA fine-tuning (0.521) and both original models, with a…","keywords":["CALM","cross-attention","financial domain adaptation","Japanese large language models","model composition","LoRA","financial dialogue benchmark"],"falsifier":"Run the same CALM setup on an independent financial benchmark not written by the augmenting model's creators, and include a control with a randomly re-initialized or unrelated augmenting model; if the gain over LoRA disappears or the random control scores equally well, the claimed knowledge transfer from the financial specialist is not the cause.","tokens_in":10514,"feed_emoji":"📈","tokens_out":9041,"duration_ms":86489,"temperature":0.7,"pith_summary":"The paper claims that composing two frozen Japanese LLMs with CALM—a trainable cross-attention layer that lets a general instruction-tuned model selectively pull knowledge from a finance-specialized model—adapts the general model to financial dialogue better than LoRA fine-tuning. This matters because domain adaptation normally requires expensive continued pre-training or fine-tuning, whereas CALM trains only a small set of connection parameters and works even when the connection-training data are not the same as the finance model's proprietary corpus. On a 360-dialogue Japanese financial benchmark, the connected model averaged 0.711, compared with 0.521 for LoRA, 0.188 for the finance model alone, and 0.043 for the general model alone. A single connection at the middle layer (layer 20) beat connections at the head (layer 0) and tail (layer 38), which the paper interprets as the sweet spot for integrating abstract knowledge.","feed_headline":"Cross-attention hookup beats fine-tuning on finance language test","feed_subtitle":"One cross-attention layer between two frozen models outscores LoRA on Japanese financial dialogue.","key_machinery":"The mechanism is Composition to Augment Language Models (CALM), which inserts a cross-attention layer between a chosen layer of the augmenting model and a chosen layer of the anchor model. Query comes from the anchor's representation, keys and values come from the augmenting model; the cross-attention output is added residually to the anchor's representation and passed to the next anchor layer, so the anchor decides at each step how much financial detail to draw in. Because the two same-architecture models share the same hidden dimension, no projection layer is needed; the learned parameters are the 128-head cross-attention weight matrices. The connection was trained on the Japanese Financial Instruction Dataset with the base models frozen, and the paper also varies the connection depth (every four layers, plus single head/middle/tail connections) to ask where the knowledge transfer works best.","core_discovery":"The central discovery is that one cross-attention bridge between two same-architecture models transfers financial capability to the anchor without changing either model's weights, and the transfer does not depend on the connector being trained on the same corpus that created the financial specialist. The paper demonstrates this with a general instruction-tuned Japanese model as anchor and a finance-specialized Japanese model as augmenting, training only the cross-attention on the Japanese Financial Instruction Dataset. Evaluated by an external judge on a 360-dialogue Japanese financial benchmark, the composed model reaches an average score of 0.711, with stronger second-turn (0.794) than first-turn (0.628) responses; LoRA, the strongest baseline, reaches 0.521. The paper's interpretation is that CALM performs contextual, token-by-token retrieval of financial knowledge from the augmenting model rather than storing limited facts in new adapter weights, which is why it answers more comprehensively in the qualitative examples.","pith_inferences":["Because the benchmark was authored by the same team that built the augmenting model on a non-public corpus, the reported gap over LoRA should be re-checked on independently written financial questions; this is a testable prediction, not a paper claim.","A control with a randomly initialized or unrelated augmenting model would isolate whether the gain comes from financial knowledge in that model's weights or from the cross-attention structure itself.","The middle-layer result hints at a general design rule for model composition—connect near the middle where representations are abstract but not yet committed to surface form—that could be probed across model families and tasks beyond finance.","The frequent end-of-text generation failure in the base models means a portion of CALM's advantage is repairing unusable outputs rather than purely adding knowledge; separating these two effects would sharpen the interpretation."],"forward_implications":["CALM offers a parameter-efficient route to financial domain adaptation: only cross-attention weights are trained, and both base models keep their original behavior, so the general model's broad capabilities are not overwritten.","Using a training dataset that differs from the augmenting model's proprietary corpus means the method is usable by external groups who cannot access the specialist model's training data.","A single middle-layer connection beats head and tail connections and does so consistently across task categories, suggesting no per-task search over connection depth is needed for financial dialogue.","The composed model's higher second-turn scores suggest the benefit is especially strong in multi-turn dialogue, where the model must build on earlier context."],"supporting_citations":[{"why":"Introduces CALM, the composition method with cross-attention between frozen LLMs that this paper applies to the financial domain.","marker":"[4]"},{"why":"Provides the finance-specialized augmenting model and the proprietary financial corpus that CALM is shown to work without.","marker":"[3]"},{"why":"Supplies the Japanese financial instruction dataset used to train the CALM connection and the LoRA baseline.","marker":"[24]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning baseline that CALM is compared against.","marker":"[25]"},{"why":"Defines the Japanese financial dialogue benchmark used for the quantitative evaluation.","marker":"[26]"}],"fun_headline_variants":["One cross-attention layer beats LoRA on finance","Frozen LLMs pick up finance via cross-attention","Cross-attention transfer outperforms fine-tuning","Financial domain adaptation without changing weights","Single cross-attention outdoes LoRA in Japanese finance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmark is an unbiased measure of financial dialogue quality; since it was built by the same group that created the augmenting model on a proprietary dataset, the reported advantage could be inflated if the benchmark happens to reward that model's strengths.","fun_headline_variants_meta":{"raw":{"variants":["One cross-attention layer beats LoRA on finance","Frozen LLMs pick up finance via cross-attention","Cross-attention transfer outperforms fine-tuning","Financial domain adaptation without changing weights","Single cross-attention outdoes LoRA in Japanese finance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1347,"prompt_tokens":927,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":543,"tokens_out":420,"duration_ms":4820,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:50:20.105634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same CALM setup on an independent financial benchmark not written by the augmenting model's creators, and include a control with a randomly re-initialized or unrelated augmenting model; if the gain over LoRA disappears or the random control scores equally well, the claimed knowledge transfer from the financial specialist is not the cause.","supporting_citations":[{"cited_title":"Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training","cited_arxiv_id":"2404.10555","evidence_quote":"Provides the finance-specialized augmenting model and the proprietary financial corpus that CALM is shown to work without."},{"cited_title":"pfmt-bench-fin-ja: Preferred Multi-turn Benchmark for Finance in Japanese,","cited_arxiv_id":null,"evidence_quote":"Defines the Japanese financial dialogue benchmark used for the quantitative evaluation."}],"review_version":1}