LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering

Beiming Liu; Haifeng Lin; Siteng Hu; Xiaohua Li; Zhengxin Zhang; Zhizhuo Cui

REVIEW 5 major objections 5 minor 2 cited by

LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a 2,480-question aerospace manufacturing exam built from textbooks, the best LLM scores 50.75% and most models fall below half.

desk verdict Useful first benchmark for LLMs in aerospace manufacturing, but the answer-key validation is too thin to trust the headline numbers. read the letter →

arxiv 2501.17183 v2 pith:6CEXI57V submitted 2025-01-25 cs.CL cs.AI

Beiming Liu , Zhizhuo Cui , Siteng Hu , Xiaohua Li , Haifeng Lin , Zhengxin Zhang This is my paper · ORCID

classification cs.CLcs.AI

keywords LLMevaluationaerospacemanufacturingautomatedquestiongenerationmultiple-choiceansweringhallucinationdomain-specificbenchmarksafety-criticalAI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to measure whether large language models actually know aerospace manufacturing, and it concludes they mostly do not. The authors build a multiple-choice exam from aerospace manufacturing textbooks and guidelines, generate questions automatically with an LLM, and have eleven models sit the test. The top model, Gemini-2.0-flash-exp, reaches 50.75% overall accuracy, while most models score below half and every model drops in the aerospace materials category. If the numbers are right, LLM output in this safety-critical domain cannot be trusted without external verification.

What carries the argument

The machinery is a pipeline that turns authoritative aerospace textbooks and guidelines into a benchmark: OCR to Markdown, text-snippet selection, automated multiple-choice question generation with six options and one or more correct answers, difficulty control through prompting, and a penalty-aware scoring scheme (+1 per correct option, -3 per incorrect option, 0 for unselected options). The average score under this asymmetric penalty is what differentiates models, rewarding caution as well as raw accuracy.

What would settle it

Have aerospace engineers independently answer a random sample of the roughly 2,180 questions that were not expert-reviewed, without seeing Gemini's keys, and compare their answers. If agreement with the keys is substantially below what it would be if the keys were authoritative, the reported model accuracies do not measure aerospace knowledge.

Watch

Extended reading notes

Core claim

The central discovery is that current LLMs' aerospace manufacturing expertise is far short of what deployment would require. On the constructed 2,480-question set, the best-performing model, Gemini-2.0-flash-exp, achieves 50.75% overall accuracy and Claude-3-5-sonnet-20241022 achieves 49.37%, while GPT-4o, DeepSeek-Chat-V3, and others fall below 46%. In the Aerospace Materials category the best model scores only 42.92%, and several models obtain negative average scores under the asymmetric penalty scheme. The paper also presents a concrete hallucination case where a leading model recommended nickel plating for titanium fasteners on Boeing 787 fuselage assemblies, contradicting HB 8752-2023, AS9100D, and NASA-STD-6012B. The authors read this as evidence that models lack version-aware standards knowledge, process integration, and the ability to quantify airworthiness risk.

Load-bearing premise

The evaluation's ground truth is a set of answer keys that were mostly written by an LLM, with only about 300 of roughly 7,500 questions expert-reviewed, so the reported accuracy numbers depend on those keys being correct.

Editorial extensions

If this is right

Aerospace manufacturers cannot yet rely on an off-the-shelf LLM for process design, material selection, or tool information retrieval without human review.
The Aerospace Materials category is a consistent failure point across all models, indicating a gap in training data depth for material science.
The asymmetric scoring system supplies a practical way to rank models by reliability, not just accuracy, in safety-critical domains.
The generated question bank, with 7,500 questions produced and 2,480 used in the final evaluation, can serve as a reusable benchmark for future aerospace LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because only about 300 of the roughly 7,500 generated questions were expert-reviewed, the reported accuracies should be read as agreement with an LLM-generated answer key until the remaining keys are independently checked.
The withheld textbook list and non-public question bank make external replication currently impossible; releasing an expert-validated subset would let other labs verify the numbers.
A natural testable extension is to measure whether retrieval-augmented generation or fine-tuning on the source documents lifts accuracy above the roughly 50% ceiling reported here, which the paper's own future-work list points toward.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is the first aerospace-manufacturing QA benchmark and a clear demonstration that current LLMs perform poorly on it. That's useful for a niche safety-critical domain, and the authors deserve credit for building a domain-specific evaluation from authoritative sources, running eleven models, and reporting a custom scoring system that penalizes wrong selections rather than just rewarding right ones. The expert review of a sample, the difficulty control, and the category breakdowns are sensible design choices.

The load-bearing problem is exactly what the stress-test note flags: the answer key is generated by Gemini Pro Vision, and only about 300 of the ~7,500 generated questions were expert-validated. The 2,480 questions used in Table I are not explicitly all expert-checked, and the source list is withheld. So the accuracy figures likely measure agreement with an LLM-produced key more than aerospace expertise. The -3 penalty makes this worse: if a key omits a genuinely correct option, a model with correct knowledge gets penalized, biasing both the accuracy and the average-score rankings. Without the released benchmark or a human reference score, an independent reader cannot verify the central numbers. That is a serious flaw.

The introduction's risk case study also raises eyebrows. Equations (1) and (2) present numbers like P_deviation=0.32 and C_critical=10^6 with no derivation or citation, and the ΔP_cab formula looks like an illustrative estimate rather than a validated calculation. This section reads more like motivation than evidence, and it should be clearly labeled as illustrative or removed.

That said, the qualitative conclusion that LLMs underperform in aerospace manufacturing is plausible and consistent with findings in other specialized domains. The paper is honest about its limitations, such as the JSON-parsing failures that hurt Gemini-2.0-flash-thinking's results, and the authors do not oversell their work. The benchmark itself, if properly released and validated, could be a genuine asset to the community.

I would send this to peer review, but major revisions are required: release the benchmark and the full answer keys, provide expert agreement rates, report a random-chance baseline and human reference scores, and either remove the risk model or back it with real data. With those changes, the paper would be a solid contribution; as it stands, the numbers should be read as indicative, not definitive.

Referee Report

5 major / 5 minor

Summary. The paper proposes an evaluation methodology for LLMs in aerospace manufacturing: it extracts text from aerospace textbooks and guidelines, uses the Gemini Pro Vision API to automatically generate multiple-choice questions with multiple correct answers, and evaluates eleven LLMs on 2,480 such questions using a custom scoring system with a strong penalty for incorrect option selections. The headline result is that the best model, Gemini-2.0-flash-exp, achieves 50.75% overall accuracy, leading the authors to conclude that current LLM capabilities in aerospace professional knowledge are in urgent need of improvement. The paper also reports category-specific results for aerospace assembly, aerospace materials, and structural steel, and interprets the results as evidence against relying on LLMs without human oversight in safety-critical manufacturing applications.

Significance. If the ground-truth answer keys are valid, the paper would fill a genuine gap by providing a domain-specific, professionally oriented evaluation benchmark for LLMs in aerospace manufacturing. The category-specific analysis and the discussion of the custom penalty-based scoring are useful contributions, and the paper explicitly identifies an important application risk. However, the central quantitative claims rest on answer keys produced by the same LLM family used for evaluation, with only a small expert-reviewed sample, a withheld source list, no human reference score, and no random-chance baseline. These issues mean the reported accuracy and ranking figures should be treated as provisional; a strengthened validation would make the contribution significantly more valuable.

major comments (5)

[Section III.B.3 / Table I] The evaluation's ground truth is produced by Gemini Pro Vision (Section III.B.2), and expert review covers only approximately 300 of roughly 7,500 generated questions (Section III.B.3); the paper does not state that all 2,480 questions used in Table I passed expert review, nor does it report per-question agreement between the automated answer keys and the expert reviewers. If un-reviewed keys contain systematic errors, the overall accuracy and average-score figures in Tables I–IV measure agreement with an LLM-generated key rather than aerospace expertise.
[Section III.A] The detailed list of source materials is 'temporarily withheld' (Section III.A). Because the central claim rests entirely on the accuracy numbers, and because individual answer keys cannot be checked without knowing the sources, this withholding blocks independent verification and replication of the benchmark; the source list, or at least the question set with keys, must be released or made available to reviewers.
[Section III.C / Tables I–IV] No human reference score or random-chance baseline is reported. With six options, multiple correct answers, and a −3 penalty per incorrect selection, even a model that selected all options would obtain a specific non-trivial expected score; without such a baseline, the reported accuracies and average scores cannot be interpreted as evidence of domain competence beyond chance, and the claim that models 'in urgent need of improvement' requires a comparison point.
[Section III.D.5 / Tables I–IV] Gemini-2.0-flash-thinking-exp-1219 is scored on only 717 of 2,480 questions (Table I) because of JSON parsing issues, yet its 12.82% accuracy is listed and ranked alongside models evaluated on nearly the full set; the reported figures for this model are not comparable to the others and should either be recomputed on a common subset of questions, reported with a clear confidence interval, or omitted from the ranking tables.
[Section I.B / Eqs. (1)–(2)] The risk case study's numerical parameters (P_deviation=0.32, P_failure|deviation=0.18, C_critical=10^6, the adiabatic expansion coefficient alpha=0.78, and the 500 FH−1 threshold) are presented without derivations, source footnotes, or query logs, so they cannot be independently checked; this illustrative calculation does not substitute for validation of the question-answer keys.

minor comments (5)

[Section III.B.3] The sampling criteria for the approximately 300 expert-reviewed questions are not described; state whether sampling was random and stratified by question category and difficulty, and report the outcome of the review (e.g., number of questions whose keys or options were corrected).
[Section III.C] The definition of ‘Attempt Rate’ is given in words only; provide a formula and clarify whether the denominator for accuracy is ‘Attempted Questions’ or ‘Total Questions’ when these counts differ.
[Section IV.B / Table III] The text says DeepSeek-Chat-V3 achieved the highest accuracy in Aerospace Materials at 35.40%, but Table III reports glm-4-plus at 35.89% and DeepSeek-Chat-V3 at 35.40%; reconcile the text and table.
[Section II.B] The reference placeholder ‘[?]’ for SHAP explanation frameworks is unresolved; add the intended citation or remove the placeholder.
[Section I.B] Figure 1 is referenced in the text but not included in the provided manuscript; add the figure or remove the reference.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported accuracies reduce to agreement with a Gemini-generated answer key; expert review covers only a ~300-question sample, so the central benchmark is partially self-referential.

other [Section III.B.2 (Generation Process), III.B.3 (Expert Review and Validation), III.C (Evaluation Metrics), Table I in Section III.E]
"The generation of questions was autonomously driven by the Gemini Pro Vision API, leveraging the extracted text snippets as contextual input. A detailed prompt, provided to the API, instructed the model to execute the following steps: ... 3) Explicitly indicate one or more correct answers for each question, providing definitive answer keys. ... a sampling of approximately 300 questions was meticulously selected from the generated question pool. ..."

The ground truth is not an independent expert standard but the answer key produced by the Gemini Pro Vision API. Overall Accuracy is defined as exact match with that pre-defined key, and the 2,480-question set scored in Table I is drawn from the ~7,500 generated questions of which only approximately 300 are reported to have undergone expert review. For the un-reviewed majority, the reported accuracies therefore reduce to agreement with a Gemini-generated key rather than with verified aerospace expertise. The central conclusion that LLM capabilities in aerospace knowledge are in urgent need of improvement is thus derived from a metric whose correct answers were set by the same model family under evaluation, making the benchmark self-referential rather than independently grounded.

full rationale

No self-citation chains, uniqueness imports, or fitted parameters are present; the paper's derivation chain is solely benchmark construction plus automated scoring. The circularity is partial and specific: the Overall Accuracy and Average Score values in Tables I-IV are computed by comparing model outputs with answer keys generated by Gemini Pro Vision, and the paper reports expert review for only a sample of about 300 questions from a pool of ~7,500. It never states that all 2,480 questions used in Table I passed expert review, so the quantitative result rests on unvalidated LLM-generated ground truth. The qualitative conclusion that LLMs are unreliable for specialized aerospace knowledge is plausible and consistent with external evidence, which is why the score is 6 rather than higher; however, the exact accuracy numbers and model ordering are not yet independently grounded.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the assumptions that the textbook-derived question bank is accurate, that a small expert review generalizes to the full set, and that QA accuracy is a valid proxy for domain capability. No free parameters are fitted to data; the listed parameters are hand-chosen values in the motivating case study and scoring design.

free parameters (3)

Risk model probabilities (P_deviation=0.32, P_failure_given_deviation=0.18, C_critical=10^6) = 0.32, 0.18, 10^6
Introduced without derivation in Eq. (1) to quantify the fastener coating hallucination case study; not connected to data or cited sources.
Adiabatic expansion coefficient (alpha=0.78) = 0.78
Used in Eq. (2) for cabin pressure loss rate; no derivation or source is given for this value for N2-O2 mixtures.
Custom scoring weights (+1 correct, -3 incorrect) = +1 / -3
Chosen by hand in Section III.C to penalize speculative answers; affects the average score metric but not the overall accuracy metric.

assumptions (4)

domain assumption Selected textbooks and industry guidelines are authoritative and representative of aerospace manufacturing knowledge.
Section III.A treats these sources as ground truth for question generation and for the answer keys.
domain assumption OCR and LLM-based conversion preserves the technical content and structure of source documents.
Section III.A relies on OCR and Markdown conversion to construct the knowledge dataset without independent validation of the conversion step.
ad hoc to paper Expert review of a 300-question sample validates the remaining automatically generated questions and answer keys.
Section III.B.3 reviews approximately 12% of the evaluated set; the remaining ~2,180 question keys are not independently checked.
domain assumption Performance on this multiple-choice QA set is a valid proxy for aerospace manufacturing expertise.
The paper equates accuracy on the generated questions with capability in the domain, without a human expert baseline or a link to real manufacturing tasks.

invented entities (1)

Lambda_risk composite risk index
purpose: Quantifies the risk of the fastener coating hallucination as 57,600 FH^-1 in Eq. (1).
No derivation of the multiplicative components, no calibration to real failure data, and the FH^-1 units are inconsistent with the product of probabilities and a criticality factor.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering." pith.science (2026). https://pith.science/paper/6CEXI57V

@misc{pith2026250117183,
  author       = {Pith},
  title        = {Pith review of: LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CEXI57V}},
  note         = {Machine review of arXiv:2501.17183}
}

read the original abstract

Aerospace manufacturing demands exceptionally high precision in technical parameters. The remarkable performance of Large Language Models (LLMs), such as GPT-4 and QWen, in Natural Language Processing has sparked industry interest in their application to tasks including process design, material selection, and tool information retrieval. However, LLMs are prone to generating "hallucinations" in specialized domains, producing inaccurate or false information that poses significant risks to the quality of aerospace products and flight safety. This paper introduces a set of evaluation metrics tailored for LLMs in aerospace manufacturing, aiming to assess their accuracy by analyzing their performance in answering questions grounded in professional knowledge. Firstly, key information is extracted through in-depth textual analysis of classic aerospace manufacturing textbooks and guidelines. Subsequently, utilizing LLM generation techniques, we meticulously construct multiple-choice questions with multiple correct answers of varying difficulty. Following this, different LLM models are employed to answer these questions, and their accuracy is recorded. Experimental results demonstrate that the capabilities of LLMs in aerospace professional knowledge are in urgent need of improvement. This study provides a theoretical foundation and practical guidance for the application of LLMs in aerospace manufacturing, addressing a critical gap in the field.

Figures

Figures reproduced from arXiv: 2501.17183 by the authors.

**Figure 2.** Evaluation Methodology aircraft manufacturing. To maintain evaluation fairness and mitigate potential biases associated with specific knowledge origins, the detailed list of source materials is temporarily withheld. The knowledge extraction process encompassed the following key steps: 1) Document Acquisition and Digitization: Initially, selected textbooks and guidelines were acquired in PDF format. Subsequently, th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance
cs.CL 2025-08 conditional novelty 5.0 of 10

CAMB provides a seven-task, eight-dataset benchmark for assessing LLM and embedding model performance in civil aviation maintenance, with initial results showing large models top out near 69% on domain multiple-choice...
Machine Assistant with Reliable Knowledge: Enhancing Student Learning via RAG-based Retrieval
cs.IR 2025-06 reject novelty 4.0 of 10

A RAG-based tutoring and support chatbot with hybrid search and an instructor feedback loop is described, but no quantitative evaluation of its accuracy is provided.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages · cited by 2 Pith papers

[1]

China Aviation Industry Standards Committee, ``HB 8751-2023/HB 8752-2023: Technical specifications for aerospace fastener surface treatment,'' Beijing, China: AISC Press, 2023

work page 2023
[2]

International Aerospace Quality Group, ``AS9100D: Quality management systems - Requirements for aviation, space and defense organizations,'' Paris, France: IAQG Publications, 2016

work page 2016
[3]

NASA Technical Standards Program Office, ``NASA-STD-6012 Rev.B: Corrosion protection for spaceflight hardware,'' Washington, DC: NASA Press, 2021

work page 2021
[4]

Hendrycks, T

D. Hendrycks, T. Brown, B. Neelakantan, et al., ``Measuring massive multitask language understanding,'' in Proc. Int. Conf. Learn. Represent., 2020

work page 2020
[5]

Zellers, Y

R. Zellers, Y. Bisk, R. Schwartz, et al., ``HellaSwag: Can a machine really finish your sentence?'' in Proc. 57th Annu. Meet. Assoc. Comput. Linguist., 2019, pp. 4791--4800

work page 2019
[6]

Papineni, S

K. Papineni, S. Roukos, T. Ward, et al., ``Bleu: a method for automatic evaluation of machine translation,'' in Proc. 40th Annu. Meet. Assoc. Comput. Linguist., 2002, pp. 311--318

work page 2002
[7]

Lee and M

J. Lee and M. Mitici, ``Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics,'' Reliab. Eng. Syst. Saf., vol. 228, p. 108908, Dec. 2022

work page 2022
[8]

Yasuda, F.A.M

Y.D.V. Yasuda, F.A.M. Cappabianco, L.E.G. Martins, and J.A.B. Gripp, ``Aircraft visual inspection: A systematic literature review,'' Comput. Ind., vol. 141, p. 103695, Oct. 2022

work page 2022

Show all 12 references

[9]

Z. Zeng, Q. Cheng, X. Hu, et al., ``KoSEL: Knowledge subgraph enhanced large language model for medical question answering,'' Knowledge-Based Systems, vol. 309, p. 112837, Jan. 2025

2025
[10]

Z. Qi, H. Luo, X. Huang, Z. Zhao, Y. Jiang, X. Fan, H. Lakkaraju, and J. Glass, ``Quantifying generalization complexity for large language models,'' arXiv preprint arXiv:2410.01769v2, Oct. 2024

2024 arXiv
[11]

Alonso, M

I. Alonso, M. Oronoz, R. Agerri, ``MedExpQA: Multilingual benchmarking of large language models for medical question answering,'' Artif. Intell. Med., vol. 155, p. 102938, 2024

2024
[12]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

China Aviation Industry Standards Committee, ``HB 8751-2023/HB 8752-2023: Technical specifications for aerospace fastener surface treatment,'' Beijing, China: AISC Press, 2023

work page 2023

[2] [2]

International Aerospace Quality Group, ``AS9100D: Quality management systems - Requirements for aviation, space and defense organizations,'' Paris, France: IAQG Publications, 2016

work page 2016

[3] [3]

NASA Technical Standards Program Office, ``NASA-STD-6012 Rev.B: Corrosion protection for spaceflight hardware,'' Washington, DC: NASA Press, 2021

work page 2021

[4] [4]

Hendrycks, T

D. Hendrycks, T. Brown, B. Neelakantan, et al., ``Measuring massive multitask language understanding,'' in Proc. Int. Conf. Learn. Represent., 2020

work page 2020

[5] [5]

Zellers, Y

R. Zellers, Y. Bisk, R. Schwartz, et al., ``HellaSwag: Can a machine really finish your sentence?'' in Proc. 57th Annu. Meet. Assoc. Comput. Linguist., 2019, pp. 4791--4800

work page 2019

[6] [6]

Papineni, S

K. Papineni, S. Roukos, T. Ward, et al., ``Bleu: a method for automatic evaluation of machine translation,'' in Proc. 40th Annu. Meet. Assoc. Comput. Linguist., 2002, pp. 311--318

work page 2002

[7] [7]

Lee and M

J. Lee and M. Mitici, ``Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics,'' Reliab. Eng. Syst. Saf., vol. 228, p. 108908, Dec. 2022

work page 2022

[8] [8]

Yasuda, F.A.M

Y.D.V. Yasuda, F.A.M. Cappabianco, L.E.G. Martins, and J.A.B. Gripp, ``Aircraft visual inspection: A systematic literature review,'' Comput. Ind., vol. 141, p. 103695, Oct. 2022

work page 2022