{"id":"ee30bdd6-0a7f-42b6-921b-2847cb1fecc2","arxiv_id":"2506.20093","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ITFormer aligns time-series encoder outputs with a frozen large language model via lightweight instruction tokens, and EngineMT-QA provides a four-task aero-engine benchmark for temporal-textual QA.","lead":"This paper introduces EngineMT-QA, a question-answering dataset built from aircraft engine sensor signals, and ITFormer, a method that connects a time-series encoder to a frozen large language model using a small trainable alignment bridge. The authors report strong gains over several baselines, but the comparison is uneven because some baselines were not fine-tuned.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EngineMT-QA's validity is the load-bearing premise; no text-only or signal-ablated control exists to rule out language-prior shortcuts, and the 110k vs 11k dataset-size discrepancy is unresolved.","rationale":"Agree with the reader that dataset validity is the weakest assumption. The strongest claim—a working recipe for attaching time-series encoders to frozen LLMs—requires the benchmark to measure temporal-textual reasoning. The paper's internal inconsistency between 110k and 11k QA pairs (Abstract/Introduction vs. Section 5.1 and Figure 3) compounds the concern, as does the lack of error bars and the unclear adaptation of baselines. I still credit the TimeSeriesExam transfer experiment as independent evidence that ITFormer transfers across domains, so I do not recommend rejection. The appropriate outcome is the same conditional verdict: accept only after the authors release data/code, provide the text-only/signal-shuffled control, and resolve the dataset-size discrepancy. No ad hominem intended; the concern is about missing controls, not author conduct.","tokens_in":22800,"tokens_out":6612,"duration_ms":71453,"concrete_test":"Run a controlled evaluation on the Perception and Reasoning test splits using a frozen strong LLM (e.g., Qwen2.5-7B) under three input conditions: (1) question text only, with the time-series placeholder removed; (2) question text plus a time-series whose sensor values are randomly permuted across samples (breaking the signal-label link); and (3) the same text plus the true time series. If condition (1) or (2) yields accuracy far above chance—especially near Table 1's ChatGPT-4o scores—the benchmark admits language-prior shortcuts and the gains do not demonstrate temporal reasoning. If both collapse to chance while ITFormer retains its reported accuracy, the temporal signal is necessary. Separately, verify the released dataset's QA-pair count against the 'over 110k' versus 11k claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that ITFormer's lightweight alignment module enables cross-modal temporal-textual QA. This rests on EngineMT-QA being a valid measure of that skill. Appendix A states that Q&A pairs were produced by LLM-based description plus expert validation, but the paper reports no inter-annotator agreement, no human baseline, and no control for answerability from text alone. The Perception and Reasoning tasks are multiple-choice with answers derived from N-CMAPSS health labels (Section 5.1, Figure 3). If option wording, template phrasing, or sensor-channel statistics correlate with labels, a model could score highly without temporal reasoning. Section 5.2 provides no text-only or signal-shuffled baseline, so the large margins (e.g., Reasoning Accuracy 88.69 vs. best baseline 54.78) cannot be attributed to temporal understanding. Furthermore, the Abstract and Introduction claim 'over 110k' QA pairs while Section 5.1 and Figure 3 state 11k; this tenfold discrepancy is unexplained and undermines the benchmark's reported scale. The TimeSeriesExam transfer result offers partial external support, but it does not validate EngineMT-QA's labels or rule out shortcuts on the main benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Time-Series Question Answering (Time-Series QA) task, a new benchmark dataset called EngineMT-QA built from the N-CMAPSS aero-engine dataset, and an alignment method called ITFormer that connects a frozen time-series encoder with a frozen LLM using only a small number of trainable parameters. The claimed contributions are threefold: a large-scale multi-task temporal-textual QA dataset, a lightweight cross-modal alignment architecture (with Time Token Position Encoding, Learnable Instruct Tokens, Instruct Time Attention, and Time Token as Language), and experimental results showing that ITFormer outperforms a range of adapted baselines on EngineMT-QA as well as on the external TimeSeriesExam benchmark.","tokens_in":23095,"tokens_out":7606,"duration_ms":71817,"significance":"If the results hold, the paper would provide a useful recipe for attaching time-series encoders to frozen LLMs for question answering and a new benchmark for temporal-textual reasoning. The lightweight design (claimed <1% trainable parameters), the public release of code and dataset, and the transfer experiment to TimeSeriesExam are positive aspects that support reproducibility and external validation. However, the significance is currently undercut by internal inconsistencies in the dataset scale, by an unfair comparison setup against API baselines, by missing control experiments that would validate the benchmark, and by dimensional errors in the core attention equations. These issues must be resolved before the claimed state-of-the-art results and the benchmark's validity can be accepted.","major_comments":[{"comment":"The paper states in the Abstract and Section 1 that EngineMT-QA contains 'over 110k question-answer pairs,' while Section 5.1 and Figure 3 both state that the dataset 'comprises 11k QA pairs.' This tenfold discrepancy is unresolved and directly affects the central 'large-scale multi-task dataset' contribution. The authors must correct the number and ensure consistency across the abstract, introduction, main text, and figures.","section":"Abstract / Section 1 vs. Section 5.1 / Figure 3"},{"comment":"The comparison in Table 1 is not apples-to-apples: ITFormer is fine-tuned on the EngineMT-QA training split, while ChatGPT-4o and Gemini are evaluated zero-shot on plotted signal images (as described in Appendix D). The 'fair comparison' paragraph claims that all baselines were adapted and trained with the same steps and epochs, but API models cannot be fine-tuned in this setup. The large margins in Table 1 may therefore reflect the fine-tuning versus zero-shot gap rather than architectural superiority. Please either fine-tune or otherwise adapt the API baselines under the same training protocol, or add a zero-shot ITFormer baseline and explicitly discuss the difference.","section":"Section 5.2 / Table 1 / Appendix D"},{"comment":"The validity of EngineMT-QA as a measure of temporal-textual reasoning is unverified. For the multiple-choice Perception and Reasoning tasks, answers are derived from N-CMAPSS health labels, but the paper provides no text-only baseline (the question and options without the time series), no sensor-shuffled or time-permuted control, no human performance estimate, and no inter-annotator agreement for the LLM-generated and expert-validated questions. Without these controls, high accuracy could result from template artifacts or language priors rather than genuine temporal reasoning. The TimeSeriesExam transfer in Section 5.6 offers partial external support but does not validate the EngineMT-QA labels. Please add these controls and report them.","section":"Section 5.1 / Appendix A / Figure 3"},{"comment":"The Channel Instruct Fusing equations are dimensionally inconsistent. With Q_channel in R^{n x d_k} and K_channel derived from H_T in R^{L' x V x d}, the product Q_channel K_channel^T in Eq. (8) gives an attention matrix indexed by (n, L'*V), but Eq. (9) indexes A_channel[q, v] with v ranging only over V and then sums over q and v to produce H_channel[l, k]. No reshaping or summation over L' is specified, so the equations do not define a valid channel aggregation. Please rewrite this step with explicit tensor shapes and operations.","section":"Section 4.1 / Eqs. (7)-(9)"},{"comment":"All performance claims in Table 1 appear to be based on a single train/test split with no error bars, no multiple seeds, and no statistical significance tests. Given that the headline result is that 'ITFormer-7B consistently achieves the best performance on every metric,' the authors should report means and standard deviations over at least 3-5 seeds (or bootstrap confidence intervals) to support the claimed differences, especially where margins are narrow (e.g., Perception accuracy 65.07 for ITFormer-7B versus 64.07 for ITFormer-3B).","section":"Section 5.2 / Table 1"}],"minor_comments":[{"comment":"The column headers use 'BLUE' in both tables; this should be 'BLEU'.","section":"Tables 1 and 2"},{"comment":"There are minor language errors, including 'a adaptable paradigm' in the Abstract and 'an innovate framework' in the contributions list; these should be corrected.","section":"Abstract and Section 1"},{"comment":"The dataset is referred to as 'N-MAPSS' in Appendix A but as 'N-CMAPSS' in the main text; please use the official dataset name consistently.","section":"Appendix A"},{"comment":"Figure 8 and its surrounding text contain unreadable encoding artifacts (e.g., '/uni00000038/uni00000051/...'), so the reported statistical indicators cannot be inspected. Please replace with a clean figure and caption.","section":"Appendix B / Figure 8"},{"comment":"The reference for MCAN-VQA, listed as 'Zhou Yu, J. Y., Yuhao Cui, D. T., and Tian, Q.', appears malformed; the author names should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially useful dataset and a lightweight adapter, but the strong performance claims should not be accepted without fixing the dataset-size inconsistency, the unfair baseline comparison, and the missing validity controls. The claim of being 'the first' large-scale Time-Series QA benchmark is also too strong in light of existing resources such as Time-MMD and TimeSeriesExam; the authors should position EngineMT-QA more carefully. The garbled Appendix B suggests a LaTeX/rendering issue that must be corrected. I recommend major revision rather than rejection because the core idea is defensible and the identified issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: the real contribution here is the EngineMT-QA dataset, not ITFormer. The architecture is a reasonable combination of existing pieces—PatchTST encoder, LLaVA-style projector, learnable instruction tokens—and it works on its own benchmark, but the evaluation currently doesn't support the headline claims.\n\nWhat is genuinely new: EngineMT-QA, a multi-task QA dataset built from N-CMAPSS, covering understanding, perception, reasoning, and decision-making, with expert validation. That is a useful resource for the time-series-plus-LLM community. The transfer experiment to TimeSeriesExam gives some external support that pretraining on EngineMT-QA helps, and the adaptability study shows the framework works across encoders and LLMs. The lightweight design (under 1% trainable parameters) is also a practical plus.\n\nThe soft spots, in order of severity. First, the baselines are not on the same footing. ChatGPT-4o and Gemini run zero-shot with plotted images, while ITFormer is fine-tuned on the training split. That alone explains part of the gap. The vision-text baselines are adapted by swapping in PatchTST, which helps, but they remain weak comparisons. Second, and this is the load-bearing issue: there is no text-only or signal-shuffled control. Without that, we cannot tell whether the large margins on Perception and Reasoning come from temporal understanding or from language priors and template artifacts. The stress-test note is right to flag this. Third, the dataset-scale discrepancy: the abstract and introduction say over 110k QA pairs; Section 5.1 and Figure 3 say 11k. A tenfold difference is not a typo you can wave away. Fourth, minor but real: no error bars or multiple seeds, and no inter-annotator agreement or human baseline for the dataset itself.\n\nWho this is for: anyone working on time-series QA or on attaching encoders to frozen LLMs. The dataset is worth knowing about even if the empirical claims need work.\n\nMy recommendation: this deserves a serious referee, not a desk reject. The revision bar should be high—fix the size discrepancy, add matched baselines and text-only/signal-shuffled controls, report seeds, and release the data and code with a stable hash. Until then, treat the state-of-the-art claim as unverified.","headline":"Useful new dataset, unproven architecture claims; needs fair baselines, text-only controls, and a resolution of the 110k vs 11k discrepancy before the results can be taken at face value.","tokens_in":23606,"tokens_out":2536,"would_cite":true,"duration_ms":26844,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small trained bridge can attach any time-series encoder to a frozen large language model for question answering.","keywords":["time-series question answering","multimodal large language models","temporal-textual alignment","EngineMT-QA","ITFormer","frozen language model","cross-modal fusion","aero-engine health monitoring"],"falsifier":"Run ITFormer on EngineMT-QA with the signal degraded while the question text stays identical — shuffle the sensor channels, permute the ten cycles, or replace the window with noise matched to its statistics. If accuracy remains high, the answers come from language priors rather than the signal. Two supporting checks settle the benchmark itself: an inter-annotator agreement study on a sample of pairs, and a direct count of released pairs to resolve the abstract's 110k versus the main text's 11k.","tokens_in":22623,"feed_emoji":"📈","tokens_out":12375,"duration_ms":112406,"temperature":0.7,"pith_summary":"The paper tries to establish that natural-language question answering over raw time-series signals is solvable by a small trainable bridge between two frozen models. It defines the Time-Series QA task, releases EngineMT-QA, a multi-task benchmark of aero-engine sensor question-answer pairs, and proposes ITFormer, which keeps both the time-series encoder and the large language model frozen and trains only an alignment module (about 0.07% of the total parameters). ITFormer-7B is reported to win every metric on all four tasks — understanding, perception, reasoning, decision-making — beating multimodal APIs such as GPT-4o and Gemini as well as time-series-text and vision-text baselines. The paper also reports that pretraining on EngineMT-QA transfers to the unrelated TimeSeriesExam benchmark. The contribution, if it holds, is a recipe: freeze the big models, train the bridge, and any sensor stream gains a natural-language interface.","feed_headline":"A tiny adapter makes frozen LLMs answer time-series questions","feed_subtitle":"ITFormer trains only an alignment module and tops GPT-4o, Gemini, and other baselines on a new aero-engine QA benchmark.","key_machinery":"The central object is the ITFormer alignment module, a lightweight trainable bridge composed of four named components: Time Token Position Encoding (hierarchical position information at temporal, channel, and segment levels), Learnable Instruct Tokens (query-derived tokens that carry task instructions into the fusion), Instruct Time Attention (a two-stage channel-then-time aggregation guided by those instructions), and Time Token as Language (injecting the fused tokens into the LLM's input sequence by replacing placeholder tokens). The module's job is to make the frozen LLM read time-series semantics as if they were words, so the only parameters updated during supervised fine-tuning are the bridge's own.","core_discovery":"On its own terms, the paper's claim is that temporal semantics can be written into the embedding space of a frozen LLM. ITFormer encodes multivariate sensor windows with a frozen PatchTST encoder, adds position encodings at three levels (time steps, channels, segments), and derives task-specific guidance from learnable instruct tokens that are prepended to the question and refined by self-attention. A two-stage Instruct Time Attention then aggregates channels for each time step and then aggregates time, both under that guidance, producing fused tokens that replace placeholder tokens in the question's embedding sequence so the frozen decoder treats them as language. With only the alignment module trained, ITFormer-7B achieves the best reported scores on every EngineMT-QA metric, including 88.69 Accuracy and F1 on reasoning and 58.04 Rouge-L on understanding, and the ablation study attributes the largest gains to the position encoding and the instruct attention.","pith_inferences":["If the recipe generalizes, a single small adapter could add interactive natural-language analysis to medical, climate, and industrial sensing systems; the paper only demonstrates the pattern on engine data plus one domain-agnostic exam, so the breadth is an extrapolation.","The paper does not run any control that removes the signal while keeping the question fixed, so how much of the reported accuracy reflects genuine signal reading versus answerable patterns in the LLM-generated questions remains my open concern, not a claim the paper addresses.","The abstract states over 110k question-answer pairs while the dataset section states 11k; the actual released count needs verification before downstream work can rely on the benchmark's scale.","A natural testable extension is whether the same instruct-token, channel-then-time attention pattern transfers to other structured modalities, such as multi-sensor arrays, graphs, or time-frequency images."],"forward_implications":["Any existing time-series encoder can gain a natural-language question-answering interface by training the small bridge, without retraining either the encoder or the LLM.","EngineMT-QA offers a reusable four-task benchmark, allowing future methods to be compared on temporal understanding, perception, reasoning, and decision-making over the same sensor data.","Results scale with the frozen LLM's size, so upgrading from a 0.5B to a 7B model improves temporal-textual accuracy at essentially no change to the trainable parameter budget.","Pretraining on EngineMT-QA raises accuracy on the domain-agnostic TimeSeriesExam, indicating the dataset is a transferable resource beyond aero engines.","The same framework handles open-ended generation and fixed-option classification, so one deployment covers both answer styles."],"supporting_citations":[{"why":"Source data: the N-CMAPSS aero-engine run-to-failure records, with 32 sensor channels and health labels, from which EngineMT-QA is built.","marker":"(Arias Chao et al., 2021)"},{"why":"Provides PatchTST, the frozen time-series encoder whose patch-transformed tokens ITFormer aligns with language.","marker":"(Nie et al., 2023)"},{"why":"Provides the frozen Qwen2.5 LLM family (0.5B, 3B, 7B) that serves as ITFormer's decoder.","marker":"(Yang et al., 2024)"},{"why":"Supplies TimeSeriesExam, the domain-agnostic benchmark used to test ITFormer's transfer and EngineMT-QA's reusability.","marker":"(Cai et al., 2024)"},{"why":"Time-LLM, a time-series-text baseline adapted to the same PatchTST encoder for a fair comparison.","marker":"(Jin et al., 2024b)"},{"why":"Defines BLEU, used to evaluate the open-ended Understanding and Decision-making answers.","marker":"(Papineni et al., 2002)"},{"why":"Defines Rouge-L, the other metric for the open-ended tasks.","marker":"(Lin, 2004)"},{"why":"GPT-4o, a multimodal API baseline that ITFormer claims to outperform on all four tasks.","marker":"(OpenAI, 2024)"},{"why":"Gemini, the second multimodal API baseline, with the same role.","marker":"(Team et al., 2023)"}],"fun_headline_variants":["Tiny adapter lets frozen LLMs answer time-series questions","ITFormer: small trained module, frozen LLM, SOTA on time-series QA","Time-series QA: train only the adapter, freeze the LLM","New multi-modal QA dataset and method bridges time and text","Frozen LLM plus small adapter: efficient time-series QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that EngineMT-QA's ground-truth answers, drafted by language models and checked by experts, genuinely require reading the time-series signal; the paper reports no human baseline or agreement measure, so if the questions are answerable from templates or language priors, the reported gains would not show true cross-modal understanding.","fun_headline_variants_meta":{"raw":{"variants":["Tiny adapter lets frozen LLMs answer time-series questions","ITFormer: small trained module, frozen LLM, SOTA on time-series QA","Time-series QA: train only the adapter, freeze the LLM","New multi-modal QA dataset and method bridges time and text","Frozen LLM plus small adapter: efficient time-series QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1798,"prompt_tokens":946,"completion_tokens":852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":762}},"tokens_in":562,"tokens_out":852,"duration_ms":8338,"temperature":1.0,"reasoning_tokens":762,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:58:00.917110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ITFormer on EngineMT-QA with the signal degraded while the question text stays identical — shuffle the sensor channels, permute the ten cycles, or replace the window with noise matched to its statistics. If accuracy remains high, the answers come from language priors rather than the signal. Two supporting checks settle the benchmark itself: an inter-annotator agreement study on a sample of pairs, and a direct count of released pairs to resolve the abstract's 110k versus the main text's 11k.","supporting_citations":[{"cited_title":"Aircraft Engine Run-to-Failure Dataset under Real Flight Conditions for Prognostics and Diagnostics","cited_arxiv_id":null,"evidence_quote":"Source data: the N-CMAPSS aero-engine run-to-failure records, with 32 sensor channels and health labels, from which EngineMT-QA is built."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"Defines BLEU, used to evaluate the open-ended Understanding and Decision-making answers."},{"cited_title":"Rouge: A package for automatic evaluation of summaries","cited_arxiv_id":null,"evidence_quote":"Defines Rouge-L, the other metric for the open-ended tasks."},{"cited_title":"Gpt-4 technical report, 2024","cited_arxiv_id":null,"evidence_quote":"GPT-4o, a multimodal API baseline that ITFormer claims to outperform on all four tasks."}],"review_version":1}