Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CAMB introduces a seven-task benchmark for civil aviation maintenance and claims it can tell apart how well embedding models and LLMs handle domain knowledge and multi-step reasoning.

desk verdict A domain benchmark with a genuinely novel fault-tree reasoning task, but self-referential label construction undermines the headline rankings until fixed. read the letter →

arxiv 2508.20420 v1 pith:4R7UYTDP submitted 2025-08-28 cs.CL

Feng Zhang , Chengjie Pang , Yuehan Zhang , Chenyu Luo This is my paper
classification cs.CL
keywords civilaviationmaintenanceLLMbenchmarkembeddingmodelsretrieval-augmentedgenerationfaulttreeQAmultiple-choiceevaluationtest-timescalingdomain-specific
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CAMB, a benchmark built from maintenance textbooks, bilingual corpora, FIM/TSM manuals, fault cases, and exam questions, covering seven tasks from terminology alignment to fault-tree reasoning. It claims the benchmark is effective at distinguishing embedding models and LLMs in civil aviation maintenance, a vertical currently underserved by math- and code-focused evaluations. Results across eight embedding models show semantic-similarity strength but weak factual-knowledge retrieval, and 13 LLM configurations score 60-70% on multiple-choice maintenance questions. The paper also reports that thinking mode, while exhibiting test-time scaling behavior, does not significantly beat non-thinking mode on this domain. A reader should care because maintenance work is knowledge-intensive and safety-critical, and RAG is the dominant deployment pattern; a reliable way to compare models in this domain would guide fine-tuning, retrieval, and prompt choices.

What carries the argument

CAMB's carrying mechanism is the seven-task/eight-dataset mapping of a single regulated domain into formats usable by both embeddings and LLMs. For embeddings, the same source material becomes bitext mining, classification, clustering, pair classification, retrieval, reranking, and fault-tree retrieval tasks; for LLMs it becomes translation, classification, multiple-choice QA, manual matching, open QA, and tree-structured reasoning. The most distinctive component is the fault-tree structured QA, built from real B737/A320 troubleshooting cases: each case is converted into a tree with the observed fault at the root, causes at intermediate nodes, and the root cause at leaves, and models are ask

What would settle it

Rebuild the sentence-pair negatives and the curated retrieval corpus using only human experts or a held-out model that is not among the eight ranked embedding systems, then re-run the benchmark; if the resulting model ranking changes materially, the claimed discriminative power depends on construction bias.

Watch

Extended reading notes

Core claim

The authors' central claim is that CAMB, their civil aviation maintenance benchmark, is effective to distinguish the performance of different embedding models and LLMs. The benchmark contains eight datasets mapped to seven tasks: alignment bilingual terminology, aircraft fault system localization, aircraft text chapter localization, civil aviation maintenance multiple choice, fault description and FIM manual match, civil aviation maintenance QA, and fault-tree structured QA. On these tasks, Qwen3-Embedding-8B ranks first among embeddings with a mean task score of 66.27, while Qwen3-235B-A22B-Instruct leads LLM multiple-choice accuracy at 68.87%. Across models, the authors find that embedding

Load-bearing premise

The hardest evaluation data is built with the same embedding models that are later ranked, so the gold labels may encode those models' similarity preferences; if so, the benchmark's ability to distinguish models is partly an artifact of its own construction.

Editorial extensions

If this is right

  • If CAMB works as claimed, model selection for civil aviation maintenance can be based on task-level scores instead of parameter count or general benchmarks.
  • The reported 60-70% multiple-choice ceiling gives a concrete target: domain fine-tuning, RAG optimization, or specialized prompts have clear headroom before models reach competence.
  • The finding that thinking mode stalls without domain knowledge implies that increasing inference-time compute alone will not fix maintenance reasoning; the bottleneck is knowledge, not deliberation.
  • Embedding-model users in regulated industries should treat semantic-similarity rankings as insufficient and benchmark factual retrieval separately, as CAMB does.
  • The open-sourced benchmark lets future models be compared against the same corpus and tasks, making progress in this vertical measurable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the five embedding models used to build the hardest negatives and curated corpus are the same five later ranked against smaller models, I infer that part of the measured ranking could reflect each model's own similarity preferences in the gold labels; an independent construction would test this.
  • The fault-tree format, with its acceptance of multiple valid diagnostic paths, seems transferable to other safety-critical maintenance domains such as power plants or marine engines, where troubleshooting knowledge is similarly hierarchical.
  • The contrast between CMMLU scaling gains and CAMB thinking-mode stagnation suggests a testable hypothesis: test-time scaling helps when the model already holds the relevant knowledge, and stalls when the knowledge is missing entirely.
  • The 60-70% accuracy band across very differently sized models hints that the binding constraint is domain coverage in training data rather than model capacity, which would predict that retrieval augmentation should outperform model scaling in this domain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces CAMB, a Chinese/English benchmark for evaluating embedding models and LLMs in civil aviation maintenance. It contains seven tasks spanning bilingual terminology, fault-system and ATA-chapter localization, multiple-choice QA, FIM-manual sentence matching, open-domain maintenance QA, and fault-tree reasoning, built from textbooks, FIM/TSM manuals, failure cases, and exam questions. The authors evaluate eight embedding models and thirteen LLMs (with thinking/non-thinking prompts) and report that: (1) embedding models perform well on semantic tasks but poorly on factual-knowledge retrieval; (2) LLMs reach 60–70% multiple-choice accuracy; (3) thinking modes show test-time scaling properties but do not significantly beat non-thinking modes; and (4) the benchmark is effective at distinguishing model performance. The main body is accompanied by appendices on prompts, fault-tree construction, curated-corpus construction, and consistency metrics.

Significance. If the evaluation were valid, CAMB would address a genuine gap: there are few public, domain-specific benchmarks for civil aviation maintenance, and the authors provide a large multiple-choice set (7,969 items) plus retrieval and reasoning tasks that are directly relevant to industrial RAG deployment. The open-source release and the attempt to benchmark both embeddings and LLMs are also useful contributions. However, the current manuscript contains several load-bearing evaluation choices—most notably the near-vacuous BLEU correctness threshold, the use of the evaluated models to construct the hardest labels, and the inconsistency between the stated human evaluation and the actual GPT-4o judging prompts—that prevent the reported rankings and the central 'effectiveness to distinguish models' claim from being accepted as reliable. The contributions are real but the evidence presented does not yet support the precision of the reported numbers.

major comments (5)
  1. [§3.2, Alignment bilingual terminology] The translation task counts a response as correct if its BLEU score is greater than 0.05. On single-sentence translations, BLEU>0.05 is almost always satisfied by any output that shares a word or a short n-gram, so this threshold is effectively vacuous. This inflates all Translation scores in Table 6 and makes the column nearly uninformative for model ranking (all models lie in 65–71). Please replace this with a meaningful criterion (e.g., chrF, TER, or a validated LLM judge with human calibration), or at least report the full BLEU distribution and exact-match accuracy.
  2. [§3.1 and Appendix E] The hardest labels are constructed with the same models that are later ranked. Negative pairs for the sentence-pair task are selected by BGE-large-zh-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3-Embedding-4B, and Qwen3-Embedding-8B, all of which appear in Table 3. The curated retrieval corpus is built from TOP10 retrievals of the same five embeddings and reranked by QWEN-235B-A22B, which is itself evaluated in Table 6. This creates a circularity risk: models whose similarity geometry matches the construction models will look better on Pair Class, Retrieval, and Reranker-text. To support the central claim, please demonstrate label independence—for example, by constructing negatives from human annotations or from models held out of the evaluation—and report whether the Table 3/Table 6 rankings survive.
  3. [§3.2 vs. Appendix B.1.6 and Appendix D] The main text states that fault-tree QA uses human evaluation with a three-tier scoring system, but Appendix B.1.6 gives a GPT-4o prompt for scoring fault-tree answers, and Appendix D is titled 'Human vs. GPT-4o Consistency Metrics.' This discrepancy matters because Table 6 uses Reasoning on Tree scores to rank models. Please clarify the actual protocol: Was GPT-4o the sole judge? Were all 50 fault trees double-scored by humans? The consistency evidence in Figures 9 and 10 is currently unreadable (rendered as '/uni...' strings), so the reported agreement cannot be checked. Provide readable confusion matrices and inter-annotator agreement numbers.
  4. [§5.2, Tables 4 and 5] The conclusion that thinking mode provides only marginal improvement for the 235B model is not traceable to the tables. The text states 'thinking mode (62.22%) shows only marginal improvement over non-thinking mode (62.04%),' but Table 4 lists Qwen3-235B-A22B at 62.24% and Table 5 lists non-thinking Qwen3-235B-A22B at 63.84%; moreover, the comparison mixes Qwen3-235B-A22B with Qwen3-235B-A22B-Thinking-2507 and with Qwen3-235B-A22B-Instruct. Since the thinking-vs-non-thinking finding is a headline contribution, please report paired comparisons on identical model instances with exact numbers and specify which rows are being compared.
  5. [Table 6 and §5.2] Several rankings are based on very small sample sizes without uncertainty estimates. The fault-tree dataset has 50 items, so a 4–8 point difference in Reasoning on Tree corresponds to only 2–4 items; the 202-item QA set gives similarly coarse resolution. The claim that the benchmark 'is effective to distinguish the performance of different embedding models and LLMs' needs confidence intervals, bootstrap resampling, or significance tests (e.g., McNemar for paired LLM outputs) before such fine-grained ordering can be asserted.
minor comments (6)
  1. [Appendix B.1.2] Typo: 'Engilsh' should be 'English'.
  2. [Appendix B.1.4] The prompt starts with 'DPlease' instead of 'Please'.
  3. [Table 4] The header and the Overall row appear to have mismatched column counts; the last model column is truncated. Please reformat so each model has a full row of scores.
  4. [Figure 3 and Figure 5] Axis labels and captions are garbled (e.g., 's um', 'qw e n3-4', 'Retrie v a l'). Please replace with clean text.
  5. [Appendix D, Figures 9 and 10] The consistency-metric figures are unreadable due to '/uni...' encoding. Please replace with actual confusion matrices or tables.
  6. [§3.1, Multiple-choice dataset] The 995 open-access online multiple-choice questions may overlap with public training corpora. A contamination/leakage analysis (e.g., n-gram overlap with known training sets) would strengthen the benchmark's validity.

Circularity Check

3 steps flagged · score 6.0 of 10

CAMB's hardest labels (sentence-pair negatives, curated retrieval corpus, and 100 translation references) are generated by the same embedding models and LLMs that the benchmark later ranks, so the headline 'effective to distinguish models' is partially a self-selection artifact.

  1. self definitional [Section 3.1, Sentence pairing dataset; Table 3]
    "For negative sample text pairs, we used BGE-large-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3-Embedding-4B, and Qwen3-Embedding-8B to evenly divide the positive sample set. For each fault issue, we used the models to select three descriptions or entry names that were most similar to the corresponding description or entry name but different."

    The five models listed are exactly the models evaluated in Table 3. The negative examples are defined as the 'most similar but different' entries under those models' own similarity functions. For any one of these generator models, the Pair Class task asks it to separate positives from near-misses that it itself selected; for models with a different similarity geometry, the negatives were selected under another model's metric. Thus the Pair Class scores in Table 3 rank models on a criterion partially constructed from the outputs of the very same models, so the benchmark's claimed ability to 'distinguish the performance of different embedding models' is not independent of those models.

  2. self definitional [Appendix E, Curated Corpus Set Construction; Table 3 (Retrieval, Reranker-text)]
    "For each question-answer pair, we use BGE-large-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3-Embedding-4B, and Qwen3-Embedding-8B to retrieve the TOP10 most similar texts from the corpus based on the question stem in the question-answer dataset. ... These parts are sorted using QWEN-235B-A22B, respectively."

    The curated corpus and curated re-ranked set used for Retrieval and Reranker-text are built by taking each of the five embedding models' TOP10 retrievals and assigning scores based on that ordering, with additional reranking by QWEN-235B-A22B (a model also evaluated elsewhere in the paper). Consequently, a model that shares the retrieval preferences of those five generators will retrieve the labeled 'relevant' texts and receive high ndcg@10 by construction, while a model with a different but equally valid relevance geometry is systematically penalized. The retrieval and reranker columns of Table 3 are therefore partly determined by the same models whose performance they are used to distinguish.

1 more flagged steps
  1. self definitional [Section 3.1, Chinese-English aligned dataset; Table 6 (Translation)]
    "Additionally, we translated 100 high-quality fault issues from the FIM manual into Chinese using QWEN3-32B, forming aligned Chinese and English sentences."

    Qwen3-32B is one of the LLMs evaluated in Table 6 (Qwen3-32B(N)). For the 100 FIM-derived items, the Chinese reference used to compute BLEU is Qwen3-32B's own previously generated output. When Qwen3-32B is later asked to translate the same English sentences, its score is measured against its own text, mechanically inflating its Translation result relative to other models. The Translation column is thus not a fair cross-model comparison on that subset; the gold reference for an evaluated model is literally that model's own output.

full rationale

The paper's central claim (Contributions and Conclusion) is that CAMB is effective to distinguish the performance of different embedding models and LLMs. Three construction choices make parts of that evidence circular: (1) negative pairs for the sentence-pair task are selected by five embedding models that are then scored on that task, so their Pair Class performance measures discrimination against their own near-miss choices; (2) the curated retrieval/reranking corpus and score labels are assembled from the TOP10 outputs of the same five embedding models plus reranking by QWEN-235B-A22B, so Retrieval and Reranker-text ndcg@10 reward models that share those generators' similarity geometry; (3) 100 FIM translation references are generated by QWEN3-32B, which is then evaluated on Translation in Table 6. These are not cases of a fitted parameter merely renamed as a prediction, but they are benchmark labels defined by the models under test, so the benchmark's discriminative power is partly self-fulfilling. Mitigating factors: human review corrected the fault trees, several tasks (multiple choice, system localization, QA) come from external exams/manuals, and Appendix G reproduces MTEB results that broadly agree with the embedding ranking, providing some external support. Hence the circularity is partial, not total: score 6 rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark itself is a constructed artifact, not an invented physical entity. The main ledger items are the hand-chosen evaluation thresholds and the domain assumptions about gold label correctness and the reliability of the GPT-4o judge.

free parameters (3)
  • BLEU correctness threshold = 0.05
    Hand-chosen threshold in Section 3.2: any translation with BLEU > 0.05 is counted as correct, making the translation metric nearly trivially satisfied and likely inflating all Translation column scores.
  • Text chunk size for corpus segmentation = 300
    Hand-chosen chunk size used for cluster and retrieval corpus construction (Section 3.1); no sensitivity analysis is reported.
  • Retrieval depth for Reranker-text = 10
    Top-10 retrieved texts are used for ndcg@10 and reranking (Section 3.2). This is a standard but hand-chosen parameter.
assumptions (4)
  • domain assumption Gold answers from textbooks, exams, and historical fault cases are correct and representative.
    The benchmark's scoring assumes the collected reference answers and fault-tree gold nodes are authoritative; no independent expert adjudication is described.
  • ad hoc to paper GPT-4o can serve as a reliable judge for maintenance QA and fault-tree reasoning.
    Appendix B.1.6 and D show GPT-4o is used to score model outputs, with human consistency checks, while the main text claims human evaluation. The validity of the scores depends on this unvalidated judge.
  • ad hoc to paper Negative samples generated by five embedding models are valid hard negatives.
    The sentence-pair negative set is created by selecting 'most similar but different' texts with the same models that are later evaluated (Section 3.1), assuming this does not bias the task.
  • domain assumption The 50 fault trees cover the plausible cause space for each case.
    The fault-tree evaluation assumes the constructed trees are complete and correctly annotated, yet only 50 historical cases are used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance." pith.science (2026). https://pith.science/paper/4R7UYTDP

@misc{pith2026250820420,
  author       = {Pith},
  title        = {Pith review of: CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4R7UYTDP}},
  note         = {Machine review of arXiv:2508.20420}
}
read the original abstract

Civil aviation maintenance is a domain characterized by stringent industry standards. Within this field, maintenance procedures and troubleshooting represent critical, knowledge-intensive tasks that require sophisticated reasoning. To address the lack of specialized evaluation tools for large language models (LLMs) in this vertical, we propose and develop an industrial-grade benchmark specifically designed for civil aviation maintenance. This benchmark serves a dual purpose: It provides a standardized tool to measure LLM capabilities within civil aviation maintenance, identifying specific gaps in domain knowledge and complex reasoning. By pinpointing these deficiencies, the benchmark establishes a foundation for targeted improvement efforts (e.g., domain-specific fine-tuning, RAG optimization, or specialized prompt engineering), ultimately facilitating progress toward more intelligent solutions within civil aviation maintenance. Our work addresses a significant gap in the current LLM evaluation, which primarily focuses on mathematical and coding reasoning tasks. In addition, given that Retrieval-Augmented Generation (RAG) systems are currently the dominant solutions in practical applications , we leverage this benchmark to evaluate existing well-known vector embedding models and LLMs for civil aviation maintenance scenarios. Through experimental exploration and analysis, we demonstrate the effectiveness of our benchmark in assessing model performance within this domain, and we open-source this evaluation benchmark and code to foster further research and development:https://github.com/CamBenchmark/cambenchmark

Figures

Figures reproduced from arXiv: 2508.20420 by the authors.

Figure 1
Figure 1. An overview of tasks and datasets in CAMB. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of TC-Net structure. aligned data with the aviation maintenance bilingual corpus’s words and sentences to create an aligned Chinese and English dataset for the Alignment bilingual terminology task. Cluster dataset:We selected chapters 21 to 30 from ATA, semantically segmented each chapter’s content, defined chunksize as 300, organized the segmented text along with its chapter titles into a data format of (t… view at source ↗
Figure 3
Figure 3. Retrieval tasks analysis. Figure (a) shows that in Retrieval, Qwen3-8B is 4% lower than Qwen3-4B, but in Reranker-Text, Qwen3-8B is 8% higher than Qwen3-4B, indicating that Qwen3-8B has stronger ranking capabilities but may retrieve more diverse content. Figure (b) displays the number of low-relevance texts retrieved by both models, revealing that Qwen-8B retrieves more low-relevance content, which might be the reas… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: analysis of the Fault tree Given that the fault tree dataset only has 50 entries, to more robustly evaluate the model’s results, we assess the models from three dimensions, prioritized from (1) to (3).For the definition of question difficulty, see the appendix: (1) The…
Figure 5
Figure 5. Figure 5: Model-efficiency 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance Trends on 9 CMMLU Datasets [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Performance Trends on 12 Aircraft Models [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Model-Temperature Sensitivity Curve on Fault Tree [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Human vs. GPT-4o Consistency in Fault Trees: LLM-0/LLM-1 Discrepancy Counts [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Consistency Metrics: Human vs. GPT-4o Evaluation of Fault Trees via Confusion Matrices [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Curated corpus set construction For each question-answer pair, we use BGE-large-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3- Embedding-4B, and Qwen3-Embedding-8B to retrieve the TOP10 most similar texts from the corpus based on the question stem in the…
Figure 12
Figure 12. Figure 12: Correlation between models performances and tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Fault-tree evaluation Due to the issue of complete overlap between the Query and the Gold list corresponding to other Queries in the fault-tree dataset, we adopted a two-step deduplication evaluation method, as shown in [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    DiagnosticIQ benchmark shows frontier LLMs perform similarly on standard rule-to-action tasks but lose substantial accuracy under distractor expansion and condition inversion, pointing to calibration as the key deploy...

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Scheduling challenges within maintenance repair and overhaul operations in the civil aviation sector

    Salah Albakkoush, Emanuele Pagone, and Konstantinos Salonitis. Scheduling challenges within maintenance repair and overhaul operations in the civil aviation sector. 2020

    work page 2020

  2. [2]

    Dave, Tat Nghia Nguyen, and Richard B

    Akshay J. Dave, Tat Nghia Nguyen, and Richard B. Vilim. Integrating llms for explainable fault diagnosis in complex systems, 2024. URL https://arxiv.org/abs/2402.06695

    arXiv 2024

  3. [3]

    A quantitative and qualitative evaluation of llm-based explainable fault localization

    Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. A quantitative and qualitative evaluation of llm-based explainable fault localization. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1851–1861, New York, NY , USA, 2019. ACM. doi: 10.1145/ 3292500.3330758

    arXiv 2019

  4. [4]

    Cost-effectiveness of structural health monitoring in fuselage maintenance of the civil aviation industry

    Ting Dong and Nam H Kim. Cost-effectiveness of structural health monitoring in fuselage maintenance of the civil aviation industry. Aerospace, 5(3):87, 2018

    work page 2018

  5. [5]

    Development of an aerospace engineering evaluation set for large language model benchmarking

    Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, and Alon Jacovi. Development of an aerospace engineering evaluation set for large language model benchmarking. arXiv preprint arXiv:1804.09028, 2018

    arXiv 2018

  6. [6]

    Evaluating large language models for real-world engineering tasks

    René Heesch, Sebastian Eilermann, Alexander Windmann, Alexander Diedrich, Philipp Rosenthal, and Oliver Niggemann. Evaluating large language models for real-world engineering tasks. In 2014 14th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, 2024

    work page 2014

  7. [7]

    Applied research on an aircraft maintenance assistant based on a large language model

    Lei Hou, Beixi Jia, Chenguang Xing, Zhaojiang Chen, and Ziliang Du. Applied research on an aircraft maintenance assistant based on a large language model. In Proceedings of the 2025 4th International Conference on Intelligent Systems, Communications and Computer Networks, ISCCN ’25, page 1–7, New York, NY , USA, 2025. Association for Computing Machinery. ...

    work page arXiv 2025

  8. [8]

    A multi-agent approach to fault localization via graph-based retrieval and reflexion

    George Kour and Raid Saabne. A multi-agent approach to fault localization via graph-based retrieval and reflexion. In Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of, pages 312–318. IEEE, 2014

    work page 2014

Show all 29 references
  1. [9]

    A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations

    Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. A systematic survey and critical review on evaluating la...

    2024

  2. [10]

    FD-LLM: Large language model for fault diagnosis of machines

    Lin Lin, Sihao Zhang, Song Fu, and Yikun Liu. FD-LLM: Large language model for fault diagnosis of machines. Advanced Engineering Informatics, 65, Part A, 2025. doi: https://doi.org/10.1016/j.aei.2025.103208

    2025

  3. [11]

    Llm evaluation based on aerospace manufacturing expertise: Automated generation and multi-model question answering, 2025

    Beiming Liu, Zhizhuo Cui, Siteng Hu, Xiaohua Li, Haifeng Lin, and Zhengxin Zhang. Llm evaluation based on aerospace manufacturing expertise: Automated generation and multi-model question answering, 2025. URL https://arxiv.org/abs/2501.17183

    2025 arXiv

  4. [12]

    Joint knowledge graph and large language model for fault diagnosis and its application in aviation assembly

    Peifeng LIU, Lu Qian, Xingwei Zhao, and Bo Tao. Joint knowledge graph and large language model for fault diagnosis and its application in aviation assembly. IEEE Transactions on Industrial Informatics, 20(6):8160–8169,

  5. [13]

    Aviation language understanding evaluation (alue)–large language model benchmark with aviation datasets

    Eugene Mangortey, Satyen Singh, Shuo Chen, and Kunal Sarkhel. Aviation language understanding evaluation (alue)–large language model benchmark with aviation datasets. In AIAA AVIATION FORUM AND ASCEND 2025, page 3247, 2025

    2025

  6. [14]

    Cellmea:a collaboratively enhanced large language model-based entity alignment for aircraft fault maintenance

    Xiangzhen Meng, Xiaoxuan Jiao, Jiahui Li, Shenglong Wang, Jinxin Pan, Bo Jing, and Xilang Tang. Cellmea:a collaboratively enhanced large language model-based entity alignment for aircraft fault maintenance. Expert Systems with Applications, 282:127630, 2025. ISSN 0957-4174. do...

    2025

  7. [15]

    Aviation safety qa dataset for extracting knowledge from incident reports

    Timilehin P Oderinde, Chetan Chandra, Leslie Albertoli, Jirat Bhanpato, Mayank V Bendarkar, and Dimitri Mavris. Aviation safety qa dataset for extracting knowledge from incident reports. In AIAA AVIATION FORUM AND ASCEND 2025, page 3248, 2025

    2025

  8. [16]

    Cast-eval: A domain-specific benchmark for large language models in civil aviation safety

    Dongxiao Qiao, Xianhui Tian, Yubin Xu, Jie Yang, Le Yang, and Xuhui Wang. Cast-eval: A domain-specific benchmark for large language models in civil aviation safety. In 2024 IEEE 2nd International Conference on Electrical, Automation and Computer Engineering (ICEACE), pages 341...

    2024

  9. [17]

    Implementation of a large language model for the interpretation of the colombian aeronautical regulations

    Alec Mauricio Rosales Cabezas, Danny Stevens Traslavina Navarrete, et al. Implementation of a large language model for the interpretation of the colombian aeronautical regulations. 2024

    2024

  10. [18]

    Towards enhancing aviation safety through advanced incident analysis using large language models

    Vaishali Siddeshwar, Akramul Azim, Sanaa Alwidian, and Masoud Makrehchi. Towards enhancing aviation safety through advanced incident analysis using large language models. In 2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON), pages ...

    2024

  11. [19]

    Retrieval-augmented generation and in-context prompted large language models in aircraft engineering

    A Sobester, S Middleton, H Yong, R Marsh, and E A Da Cruz Silva. Retrieval-augmented generation and in-context prompted large language models in aircraft engineering. In AIAA SCITECH 2025 Forum. American Institute of Aeronautics and Astronautics, January 2025. doi: https://doi...

    2025 doi

  12. [20]

    Benchmarking the capabilities of large language models in transportation system engineering: Accuracy, consistency, and reasoning behaviors

    Usman Syed, Ethan Light, Xingang Guo, Huan Zhang, Lianhui Qin, Yanfeng Ouyang, and Bin Hu. Benchmarking the capabilities of large language models in transportation system engineering: Accuracy, consistency, and reasoning behaviors. 08 2024. doi: 10.48550/arXiv.2408.08302

    2024 doi arXiv

  13. [21]

    Assessing large language models in mechanical engineering education: A study on mechanics-focused conceptual understanding, 2024

    Jie Tian, Jixin Hou, Zihao Wu, Peng Shu, Zhengliang Liu, Yujie Xiang, Beikang Gu, Nicholas Filla, Yiwei Li, Ning Liu, Xianyan Chen, Keke Tang, Tianming Liu, and Xianqiao Wang. Assessing large language models in mechanical engineering education: A study on mechanics-focused con...

    2024 arXiv

  14. [22]

    Generating troubleshooting trees for industrial equipment using large language models (llm)

    Lasitha Vidyaratne, Xian Yeow Lee, Aman Kumar, Tsubasa Watanabe, Ahmed Farahat, and Chetan Gupta. Generating troubleshooting trees for industrial equipment using large language models (llm). In 2024 IEEE International Conference on Prognostics and Health Management (ICPHM), pa...

    2024

  15. [23]

    Aviationllm: An llm-based knowledge system for aviation training

    Jia’ang Wan, Feng Shen, Fujuan Li, Yanjin Sun, Yan Li, and Shiwen Zhang. Aviationllm: An llm-based knowledge system for aviation training. arXiv preprint arXiv:2506.14336, 2025

    2025 arXiv

  16. [24]

    Excytin-bench: Evaluating llm agents on cyber threat investigation

    Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, and Anand Mudgerikar. Excytin-bench: Evaluating llm agents on cyber threat investigation. arXiv prepri...

    2024 arXiv

  17. [25]

    An intelligent guided troubleshooting method for aircraft based on HybirdRAG

    Xiaoyue Xie, Xilang Tang, Siwei Gu, and Lijie Cui. An intelligent guided troubleshooting method for aircraft based on HybirdRAG. Nature, 15(17752), 2025

    2025

  18. [26]

    A retrieval-augmented generation-based method for aviation ac- cident data analysis

    Jianzhong Yang, Xinyu Xiang, and Xiyuan Chen. A retrieval-augmented generation-based method for aviation ac- cident data analysis. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 868–874, 2024. doi: 10.1109/ICAIRC641...

    2024

  19. [27]

    Empirical study on fine-tuning pre-trained large language models for fault diagnosis of complex systems

    Shuwen Zheng, Kai Pan, Jie Liu, and Yunxia Chen. Empirical study on fine-tuning pre-trained large language models for fault diagnosis of complex systems. Reliability Engineering & System Safety , 252:110382, 2024. ISSN 0951-8320. doi: https://doi.org/10.1016/j.ress.2024.110382...

    2024

  20. [28]

    Application of reliability technologies in civil aviation: Lessons learnt and perspectives

    Enrico ZIO, Mengfei FAN, Zhiguo ZENG, and Rui KANG. Application of reliability technologies in civil aviation: Lessons learnt and perspectives. Chinese Journal of Aeronautics, 32(1):143–158, 2019. ISSN 1000-9361. doi: https://doi.org/10.1016/j.cja.2018.05.014. URL https://www....

    2019 doi

  21. [2024]

    doi: 10.1109/TII.2024.3366977

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.