{"id":"ee1b84e9-2753-4bcc-89a4-69c42df40802","arxiv_id":"2508.20420","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAMB provides a seven-task, eight-dataset benchmark for assessing LLM and embedding model performance in civil aviation maintenance, with initial results showing large models top out near 69% on domain multiple-choice questions.","lead":"A new benchmark, CAMB, evaluates LLMs and embedding models on civil aviation maintenance tasks, from bilingual terminology to fault-tree reasoning. Early results show current LLMs score only 60-70% on multiple-choice maintenance questions and that thinking modes do not clearly beat direct answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CAMB's hardest task labels are generated by the same embedding/LLM models it later ranks; the claimed ability to distinguish models may be a self-selection artifact.","rationale":"The reader's weakest assumption points to exactly the structural circularity I find load-bearing: the hardest evaluation data are built with the same models that are later ranked. This is not a matter of calibration or scoring thresholds; it affects whether any observed ranking can support the paper's central claim. The MTEB replication in Appendix G is useful but does not resolve this concern, because it validates off-the-shelf rankings on existing MTEB tasks rather than the independence of CAMB's self-generated labels. I also note secondary issues (the BLEU>0.05 translation threshold, and QA/fault-tree evaluation that appears to use GPT-4o despite wording suggesting human evaluation), but these affect score precision rather than the structural validity of the rankings. The concern is serious enough that the reported rankings should not be taken at face value until an independent-label control is run; however, the benchmark is open-sourced and the paper is transparent about its construction, so the appropriate disposition remains conditional rather than outright rejection. Thus I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":20966,"tokens_out":4026,"duration_ms":46707,"concrete_test":"Rebuild the two self-referential sets independently: (a) for the sentence-pair task, generate negatives by sampling uniformly from other FIM entries or by a held-out embedder not in Table 3 (e.g., OpenAI text-embedding-3-large); (b) for retrieval QA, construct relevance labels by human expert judgment alone, without pooling the five embeddings or Qwen reranking. Recompute Table 3 pairwise scores and Spearman rank correlation against the original results. If the top-3 embedding ranking changes, or Spearman < 0.6, the central claim is an artifact of label construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CAMB 'is effective to distinguish the performance of different embedding models and LLMs' (Section 1, Contribution 3). The most discriminative evidence, Tables 3 and 6, rests on two evaluation sets whose gold labels were produced by the very models being ranked. Section 3.1 (Sentence pairing dataset) builds negative pairs by asking BGE-large-zh-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3-Embedding-4B, and Qwen3-Embedding-8B to select the three most similar but wrong entries. All five are later evaluated on the resulting 'Pair Class' task (Table 3). The negative set is therefore a set of near-misses under exactly those models' similarity functions; a model sharing their inductive bias separates positives from these negatives more easily than a model with an equally valid but different geometry. Similarly, Appendix E constructs the retrieval/re-ranking corpus by pooling TOP10 retrievals from the same five embeddings and re-ranking with QWEN-235B-A22B, a model that is itself scored in Table 6. The 202-pair QA set and 50-tree fault-tree set are small enough that self-generated labels can dominate rankings. Unless independence of labels from evaluated models is shown, the headline conclusion that the benchmark is effective to distinguish models is not established; at minimum, the reported ordering is confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAMB, a Chinese/English benchmark for evaluating embedding models and LLMs in civil aviation maintenance. It contains seven tasks spanning bilingual terminology, fault-system and ATA-chapter localization, multiple-choice QA, FIM-manual sentence matching, open-domain maintenance QA, and fault-tree reasoning, built from textbooks, FIM/TSM manuals, failure cases, and exam questions. The authors evaluate eight embedding models and thirteen LLMs (with thinking/non-thinking prompts) and report that: (1) embedding models perform well on semantic tasks but poorly on factual-knowledge retrieval; (2) LLMs reach 60–70% multiple-choice accuracy; (3) thinking modes show test-time scaling properties but do not significantly beat non-thinking modes; and (4) the benchmark is effective at distinguishing model performance. The main body is accompanied by appendices on prompts, fault-tree construction, curated-corpus construction, and consistency metrics.","tokens_in":21357,"tokens_out":5435,"duration_ms":58391,"significance":"If the evaluation were valid, CAMB would address a genuine gap: there are few public, domain-specific benchmarks for civil aviation maintenance, and the authors provide a large multiple-choice set (7,969 items) plus retrieval and reasoning tasks that are directly relevant to industrial RAG deployment. The open-source release and the attempt to benchmark both embeddings and LLMs are also useful contributions. However, the current manuscript contains several load-bearing evaluation choices—most notably the near-vacuous BLEU correctness threshold, the use of the evaluated models to construct the hardest labels, and the inconsistency between the stated human evaluation and the actual GPT-4o judging prompts—that prevent the reported rankings and the central 'effectiveness to distinguish models' claim from being accepted as reliable. The contributions are real but the evidence presented does not yet support the precision of the reported numbers.","major_comments":[{"comment":"The translation task counts a response as correct if its BLEU score is greater than 0.05. On single-sentence translations, BLEU>0.05 is almost always satisfied by any output that shares a word or a short n-gram, so this threshold is effectively vacuous. This inflates all Translation scores in Table 6 and makes the column nearly uninformative for model ranking (all models lie in 65–71). Please replace this with a meaningful criterion (e.g., chrF, TER, or a validated LLM judge with human calibration), or at least report the full BLEU distribution and exact-match accuracy.","section":"§3.2, Alignment bilingual terminology"},{"comment":"The hardest labels are constructed with the same models that are later ranked. Negative pairs for the sentence-pair task are selected by BGE-large-zh-v1.5, gte-Qwen2-1.5B-instruct, gte-Qwen2-7B-instruct, Qwen3-Embedding-4B, and Qwen3-Embedding-8B, all of which appear in Table 3. The curated retrieval corpus is built from TOP10 retrievals of the same five embeddings and reranked by QWEN-235B-A22B, which is itself evaluated in Table 6. This creates a circularity risk: models whose similarity geometry matches the construction models will look better on Pair Class, Retrieval, and Reranker-text. To support the central claim, please demonstrate label independence—for example, by constructing negatives from human annotations or from models held out of the evaluation—and report whether the Table 3/Table 6 rankings survive.","section":"§3.1 and Appendix E"},{"comment":"The main text states that fault-tree QA uses human evaluation with a three-tier scoring system, but Appendix B.1.6 gives a GPT-4o prompt for scoring fault-tree answers, and Appendix D is titled 'Human vs. GPT-4o Consistency Metrics.' This discrepancy matters because Table 6 uses Reasoning on Tree scores to rank models. Please clarify the actual protocol: Was GPT-4o the sole judge? Were all 50 fault trees double-scored by humans? The consistency evidence in Figures 9 and 10 is currently unreadable (rendered as '/uni...' strings), so the reported agreement cannot be checked. Provide readable confusion matrices and inter-annotator agreement numbers.","section":"§3.2 vs. Appendix B.1.6 and Appendix D"},{"comment":"The conclusion that thinking mode provides only marginal improvement for the 235B model is not traceable to the tables. The text states 'thinking mode (62.22%) shows only marginal improvement over non-thinking mode (62.04%),' but Table 4 lists Qwen3-235B-A22B at 62.24% and Table 5 lists non-thinking Qwen3-235B-A22B at 63.84%; moreover, the comparison mixes Qwen3-235B-A22B with Qwen3-235B-A22B-Thinking-2507 and with Qwen3-235B-A22B-Instruct. Since the thinking-vs-non-thinking finding is a headline contribution, please report paired comparisons on identical model instances with exact numbers and specify which rows are being compared.","section":"§5.2, Tables 4 and 5"},{"comment":"Several rankings are based on very small sample sizes without uncertainty estimates. The fault-tree dataset has 50 items, so a 4–8 point difference in Reasoning on Tree corresponds to only 2–4 items; the 202-item QA set gives similarly coarse resolution. The claim that the benchmark 'is effective to distinguish the performance of different embedding models and LLMs' needs confidence intervals, bootstrap resampling, or significance tests (e.g., McNemar for paired LLM outputs) before such fine-grained ordering can be asserted.","section":"Table 6 and §5.2"}],"minor_comments":[{"comment":"Typo: 'Engilsh' should be 'English'.","section":"Appendix B.1.2"},{"comment":"The prompt starts with 'DPlease' instead of 'Please'.","section":"Appendix B.1.4"},{"comment":"The header and the Overall row appear to have mismatched column counts; the last model column is truncated. Please reformat so each model has a full row of scores.","section":"Table 4"},{"comment":"Axis labels and captions are garbled (e.g., 's um', 'qw e n3-4', 'Retrie v a l'). Please replace with clean text.","section":"Figure 3 and Figure 5"},{"comment":"The consistency-metric figures are unreadable due to '/uni...' encoding. Please replace with actual confusion matrices or tables.","section":"Appendix D, Figures 9 and 10"},{"comment":"The 995 open-access online multiple-choice questions may overlap with public training corpora. A contamination/leakage analysis (e.g., n-gram overlap with known training sets) would strengthen the benchmark's validity.","section":"§3.1, Multiple-choice dataset"}],"recommendation":"major_revision","confidential_remarks":"The paper has several useful components—especially the 7,969-item multiple-choice set and the domain-specific retrieval/reasoning tasks—but the current version is not ready for publication in its present form. The circularity in label construction and the BLEU threshold are not mere presentation issues; they directly affect the headline rankings. The inconsistent description of the fault-tree evaluation (human vs. GPT-4o) and the unreadable consistency figures in Appendix D also need to be resolved before external reviewers can trust the results. I would encourage a revision that either removes the problematic metrics or provides independent-label validation, and that adds statistical rigor to all small-sample comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead CAMB, the civil aviation maintenance LLM benchmark. The most useful thing here is the fault-tree reasoning task: 50 historical B737/A320 troubleshooting cases turned into hierarchical trees with stepwise inference, evaluated by partial/complete matching. That is a real niche contribution and the most interesting part of the paper. The rest is a standard MTEB-style embedding suite plus a 7,969-question multiple-choice set for LLMs, applied to maintenance domain text. The writing is clear and the work is open-sourced, which helps.\n\nThe soft spots are real, and they hit the paper's central claim. The benchmark is supposed to distinguish embedding models and LLMs, but some of the hardest tasks are built using the exact models that are then ranked. Negative pairs for the sentence-pair task are chosen by BGE-large-zh, gte-Qwen2-1.5B/7B, Qwen3-Embedding-4B/8B—five of the eight embedding models in Table 3. The retrieval corpus is pooled from the same five and reranked by Qwen3-235B-A22B, which is itself scored in Table 6. If the gold labels embed those models' similarity preferences, the rankings are partly self-selection. To fix this, add an independent label construction path (human selection or a held-out model family) and show the rankings survive.\n\nMinor but worth fixing: the translation task counts BLEU > 0.05 as correct—that's nearly vacuous and likely compresses all translation scores. The paper says fault-tree evaluation uses human evaluation, but Appendix B.1.6 contains a GPT-4o judge prompt and Appendix D reports human-vs-GPT-4o consistency, which reads as the actual judge being GPT-4o. That should be stated plainly. The fault-tree set (50 cases) and QA set (202 pairs) are small; the paper's own sub-analysis on fault trees is honest about this, but the headline numbers in Tables 3 and 6 should carry confidence intervals.\n\nWho's this for? Anyone building RAG or maintenance-assistant systems in aviation, and benchmark designers who want a domain template. The fault-tree task deserves to be reused and extended. The paper needs a revision before the rankings are taken at face value, but it deserves referee time—the domain gap is real, the material is there, and the flaws are fixable.\n\nRecommendation: send to peer review with a request for the independence fix and evaluation-methodology clarifications. Conditional accept, not reject.","headline":"A domain benchmark with a genuinely novel fault-tree reasoning task, but self-referential label construction undermines the headline rankings until fixed.","tokens_in":21773,"tokens_out":2156,"would_cite":false,"duration_ms":24377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAMB introduces a seven-task benchmark for civil aviation maintenance and claims it can tell apart how well embedding models and LLMs handle domain knowledge and multi-step reasoning.","keywords":["civil aviation maintenance","LLM benchmark","embedding models","retrieval-augmented generation","fault tree QA","multiple-choice evaluation","test-time scaling","domain-specific evaluation"],"falsifier":"Rebuild the sentence-pair negatives and the curated retrieval corpus using only human experts or a held-out model that is not among the eight ranked embedding systems, then re-run the benchmark; if the resulting model ranking changes materially, the claimed discriminative power depends on construction bias.","tokens_in":20896,"feed_emoji":"✈️","tokens_out":5911,"duration_ms":62017,"temperature":0.7,"pith_summary":"The paper proposes CAMB, a benchmark built from maintenance textbooks, bilingual corpora, FIM/TSM manuals, fault cases, and exam questions, covering seven tasks from terminology alignment to fault-tree reasoning. It claims the benchmark is effective at distinguishing embedding models and LLMs in civil aviation maintenance, a vertical currently underserved by math- and code-focused evaluations. Results across eight embedding models show semantic-similarity strength but weak factual-knowledge retrieval, and 13 LLM configurations score 60-70% on multiple-choice maintenance questions. The paper also reports that thinking mode, while exhibiting test-time scaling behavior, does not significantly beat non-thinking mode on this domain. A reader should care because maintenance work is knowledge-intensive and safety-critical, and RAG is the dominant deployment pattern; a reliable way to compare models in this domain would guide fine-tuning, retrieval, and prompt choices.","feed_headline":"Aircraft-maintenance benchmark puts LLMs at 60-70 percent","feed_subtitle":"Seven tasks, from manual matching to fault-tree reasoning, reveal where AI models hold and lose domain knowledge.","key_machinery":"CAMB's carrying mechanism is the seven-task/eight-dataset mapping of a single regulated domain into formats usable by both embeddings and LLMs. For embeddings, the same source material becomes bitext mining, classification, clustering, pair classification, retrieval, reranking, and fault-tree retrieval tasks; for LLMs it becomes translation, classification, multiple-choice QA, manual matching, open QA, and tree-structured reasoning. The most distinctive component is the fault-tree structured QA, built from real B737/A320 troubleshooting cases: each case is converted into a tree with the observed fault at the root, causes at intermediate nodes, and the root cause at leaves, and models are ask","core_discovery":"The authors' central claim is that CAMB, their civil aviation maintenance benchmark, is effective to distinguish the performance of different embedding models and LLMs. The benchmark contains eight datasets mapped to seven tasks: alignment bilingual terminology, aircraft fault system localization, aircraft text chapter localization, civil aviation maintenance multiple choice, fault description and FIM manual match, civil aviation maintenance QA, and fault-tree structured QA. On these tasks, Qwen3-Embedding-8B ranks first among embeddings with a mean task score of 66.27, while Qwen3-235B-A22B-Instruct leads LLM multiple-choice accuracy at 68.87%. Across models, the authors find that embedding","pith_inferences":["Because the five embedding models used to build the hardest negatives and curated corpus are the same five later ranked against smaller models, I infer that part of the measured ranking could reflect each model's own similarity preferences in the gold labels; an independent construction would test this.","The fault-tree format, with its acceptance of multiple valid diagnostic paths, seems transferable to other safety-critical maintenance domains such as power plants or marine engines, where troubleshooting knowledge is similarly hierarchical.","The contrast between CMMLU scaling gains and CAMB thinking-mode stagnation suggests a testable hypothesis: test-time scaling helps when the model already holds the relevant knowledge, and stalls when the knowledge is missing entirely.","The 60-70% accuracy band across very differently sized models hints that the binding constraint is domain coverage in training data rather than model capacity, which would predict that retrieval augmentation should outperform model scaling in this domain."],"forward_implications":["If CAMB works as claimed, model selection for civil aviation maintenance can be based on task-level scores instead of parameter count or general benchmarks.","The reported 60-70% multiple-choice ceiling gives a concrete target: domain fine-tuning, RAG optimization, or specialized prompts have clear headroom before models reach competence.","The finding that thinking mode stalls without domain knowledge implies that increasing inference-time compute alone will not fix maintenance reasoning; the bottleneck is knowledge, not deliberation.","Embedding-model users in regulated industries should treat semantic-similarity rankings as insufficient and benchmark factual retrieval separately, as CAMB does.","The open-sourced benchmark lets future models be compared against the same corpus and tasks, making progress in this vertical measurable."],"supporting_citations":[{"why":"Supplies the precedent of an aviation-language LLM benchmark that CAMB extends from language understanding into maintenance.","marker":"[13]"},{"why":"Provide existing civil-aviation safety benchmarks whose task design CAMB complements with maintenance-specific tasks.","marker":"[15, 16]"},{"why":"Shows a prior aerospace evaluation in which top models were insufficient, motivating domain-specific benchmark construction.","marker":"[5]"},{"why":"Supplies a QA benchmark design and RAG/in-context evaluation approach that CAMB adapts to maintenance.","marker":"[19]"},{"why":"Demonstrates benchmark construction designed to avoid test-set leakage, informing CAMB's data-contamination stance.","marker":"[24]"},{"why":"The survey of LLM evaluation challenges motivates CAMB's design for reproducibility and coverage.","marker":"[9]"},{"why":"Provides aerospace-manufacturing evaluation metrics for domain knowledge questions, used to frame CAMB's QA scoring.","marker":"[11]"}],"fun_headline_variants":["Aircraft repair benchmark: LLMs score 60-70%","Qwen3 tops aircraft maintenance LLM and embedding tests","New benchmark puts LLMs to the test in aircraft maintenance","Aviation maintenance benchmark: Qwen3 leads with 66% score"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The hardest evaluation data is built with the same embedding models that are later ranked, so the gold labels may encode those models' similarity preferences; if so, the benchmark's ability to distinguish models is partly an artifact of its own construction.","fun_headline_variants_meta":{"raw":{"variants":["Aircraft repair benchmark: LLMs score 60-70%","Qwen3 tops aircraft maintenance LLM and embedding tests","New benchmark puts LLMs to the test in aircraft maintenance","Aviation maintenance benchmark: Qwen3 leads with 66% score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000905,"raw_usage":{"total_tokens":3737,"prompt_tokens":759,"completion_tokens":2978,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2915}},"tokens_in":503,"tokens_out":2978,"duration_ms":23931,"temperature":1.0,"reasoning_tokens":2915,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:04:20.483185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the sentence-pair negatives and the curated retrieval corpus using only human experts or a held-out model that is not among the eight ranked embedding systems, then re-run the benchmark; if the resulting model ranking changes materially, the claimed discriminative power depends on construction bias.","supporting_citations":[{"cited_title":"Aviation language understanding evaluation (alue)–large language model benchmark with aviation datasets","cited_arxiv_id":null,"evidence_quote":"Supplies the precedent of an aviation-language LLM benchmark that CAMB extends from language understanding into maintenance."},{"cited_title":"Retrieval-augmented generation and in-context prompted large language models in aircraft engineering","cited_arxiv_id":null,"evidence_quote":"Supplies a QA benchmark design and RAG/in-context evaluation approach that CAMB adapts to maintenance."},{"cited_title":"LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering","cited_arxiv_id":"2501.17183","evidence_quote":"Provides aerospace-manufacturing evaluation metrics for domain knowledge questions, used to frame CAMB's QA scoring."}],"review_version":1}