Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that leading LLMs, scoring 53.5% on single-correct industrial reliability questions, are fragile to perturbations and fall short of expert-level reliability, so static accuracy overstates deployment readiness.

desk verdict A genuinely new industrial QA benchmark with broad evaluation, but its gold labels are contested and under-audited; conditionally worth engaging. read the letter →

arxiv 2506.03278 v1 pith:CJSPAIOY submitted 2025-06-03 cs.CL

Christodoulos Constantinides , Dhaval Patel , Shuxin Lin , Claudio Guerrero , Sunil Dagajirao Patil , Jayant Kalagnanam This is my paper
classification cs.CL
keywords failuremodessensorrelationshipsmultiple-choiceQAcondition-basedmaintenancelargelanguagemodelsperturbationrobustnessindustrialassetsfeatureselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FailureSensorIQ is a multiple-choice benchmark of 8,296 questions that asks models to connect failure modes to the sensors that detect them for ten industrial assets. The questions are generated from a gold relevance matrix built from ISO guidelines and expert input, with two task orientations: failure-mode-to-sensor and sensor-to-failure-mode. The paper's central claim is that current large language models are not reliably domain-ready: the top-10 models average 53.5% on the 2,667 single-correct-answer questions, below the 66.19% maximum reached by human reliability experts, and accuracy drops 5-20% under simple or complex perturbation. It also shows that reasoning prompts help medium-sized models, while retrieval-augmented agents do not, and that LLM-suggested features correlate with the true signal in about half of the real-world cases tested. The benchmark is static, testing knowledge at a point in time rather than temporal failure dynamics, a limitation the authors acknowledge.

What carries the argument

The load-bearing object is the gold-standard relevance matrix that maps failure modes to sensor and parameter readings for each of ten assets, built from ISO condition-monitoring guidelines and expert-curated information. From that matrix, templates generate two task orientations, FM2Sensor (which sensors indicate a failure mode) and Sensor2FM (which failure modes a sensor indicates), in both selection and elimination formats, yielding 2,667 single-correct and 5,629 multi-correct questions. The evaluation machinery adds perturbed versions of each question, including reordered options, label changes, and paraphrase, together with a consistency-based accuracy metric that counts only questions answered correctly in both original and perturbed form; it also extends questions to ten options to test complexity. These mechanisms allow the paper to separate what a model knows from what it retains under stress.

What would settle it

Have an independent panel of reliability engineers reconstruct the failure-mode-to-sensor mapping for, say, electric motors, compressors, and power transformers without seeing the published matrix, and measure disagreement cell by cell; the expert study's moderate inter-rater agreement of Cohen's kappa 0.462 already hints that substantial disagreement would surface, and any disputed cell directly contaminates every question generated from it.

Watch

Extended reading notes

Core claim

The paper sets out to establish that large language models lack robust, transferable knowledge of industrial asset-failure relationships. On its own expert-curated multiple-choice questions, the strongest model scores 60.4% and the top-10 average 53.5%, whereas human reliability engineers reach at most 66.19% with a mean of 60.20%; consistency-based accuracy after perturbation falls below 50% for top models. The paper argues that static accuracy overstates real knowledge capacity, because perturbing formatting or wording drops scores by 5-20%, adding distractor options caps accuracy near 12%, and models need about three selections to reach 90% coverage. It further finds that access to external corpora through a search-and-reason agent does not help, indicating that the task tests internal reasoning over fine-grained distractors rather than surface fact retrieval. The intended upshot is that the benchmark can serve as a qualification test before LLMs are trusted with condition-based maintenance decisions such as sensor selection, root-cause analysis, and proactive maintenance planning.

Load-bearing premise

The whole benchmark inherits the correctness of the gold relevance matrix, which maps which failure modes relate to which sensors and is assembled from ISO documents and expert judgment; if those mappings are wrong, ambiguous, or asset-specific, every accuracy, perturbation, and expert-comparison number built on them shifts.

Editorial extensions

If this is right

  • If the central claim holds, static leaderboard accuracy on industrial QA cannot be read as deployment readiness; a model must also pass perturbation-consistent accuracy, which current models do not.
  • For predictive-maintenance agents, the 5-20% perturbation drop means small formatting changes in sensor logs or alerts can materially change a model's diagnosis, so invariance testing should precede use.
  • The finding that retrieval-augmented agents do not improve scores implies that simply adding document search will not close the gap; progress must come from better internal reasoning and distractor discrimination.
  • LLMFeatureSelect's partial success, with the truly most-correlated sensor appearing in the top-5 in 3 of 6 test cases, suggests LLM-driven feature selection is promising but not yet dependable for failure prediction pipelines.
  • Reasoning prompts such as chain-of-thought lifting a 70 billion parameter model from 41.69% to 51.18% indicate that evaluation should always report prompting condition, since prompting changes the measured capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's sensitivity to option order and distractors suggests that a robust industrial LLM will need explicit order-invariance guarantees; one testable extension is to require exact agreement across all option permutations before accepting a diagnosis.
  • Because the gold matrix is expert-curated and experts disagree at only a moderate level, a natural next step is to release per-cell confidence or alternative plausible matrices, turning the benchmark into a probabilistic knowledge graph rather than a single ground truth.
  • The static design covers point-in-time relationships; extending to temporal questions about sensor-trend evolution, which the authors name as future work, would test whether models can reason about failure progression, not just failure presence.
  • The search-and-reason agent result suggests that retrieval strategies tuned for industrial entity vocabularies, such as sensor names and failure-mode synonyms, might outperform generic web search; this is a hypothesis the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces FailureSensorIQ, a multi-choice QA benchmark for industrial sensor-failure-mode relationships, built from ISO documents and expert-curated relevance matrices that map failure modes to sensors for 10 asset types. The authors generate 8,296 questions (2,667 single-answer, 5,629 multi-answer) and evaluate over 24 LLMs along several axes: perturbation robustness (SimplePert, ComplexPert, OptionsPert), uncertainty calibration, question complexity, asset-level knowledge gaps, ReAct agent with external retrieval, and a human expert study. A separate contribution is LLMFeatureSelector, an LLM-driven feature selection pipeline tested on three real-world datasets. The headline results are that top models average about 53.5% accuracy on single-answer questions, below expert human performance (max 66.19%), and that perturbation leads to 5-20% performance drops.

Significance. If the gold labels are trustworthy, FailureSensorIQ fills a clear gap in domain-specific LLM evaluation for industrial condition-based maintenance, and the perturbation-uncertainty-complexity analysis is a valuable evaluation methodology. The release of code, the Hugging Face leaderboard, and the LLMFeatureSelector tool are concrete strengths, and the paper includes a human expert study, which is more than most benchmark papers do. The central weakness is that the benchmark's validity rests entirely on an unreleased and independently unvalidated relevance matrix; the paper's own expert study (κ=0.462, expert accuracies 50-67%) demonstrates that the labels are substantially contested. This issue must be resolved before the reported accuracy numbers can be interpreted as measuring domain knowledge rather than agreement with a single curated mapping.

major comments (4)
  1. [§2.2, Table 7] The gold relevance matrix that defines the ground-truth answers for all 8,296 questions is neither released nor independently validated. The human expert study in Table 7 shows only moderate inter-rater agreement (Cohen's kappa = 0.462) and expert accuracies of 50-67% on a balanced sample, indicating that a substantial fraction of the labels are contested. Every accuracy, perturbation, and expert comparison inherits this label uncertainty. The authors should release the full relevance matrix, provide an independent label audit, and report benchmark performance separately for items with high versus low expert agreement. Without this, the 'hard benchmark' claim conflates task difficulty with label ambiguity.
  2. [§3.1, Appendix B.3] The ComplexPert dataset is generated by paraphrasing questions with llama-3.3-70b-instruct, but the paper does not verify that the paraphrases preserve the original meaning and the correct answer. The ACC@Consist metric assumes that the perturbed question has the same ground-truth label as the original; if a paraphrase alters semantics (for instance by changing a negation or swapping a sensor name), the observed 5-20% performance drops could be partly due to invalid perturbations rather than genuine model fragility. The authors should add a human or automated verification of a random sample of ComplexPert items and report the paraphrase validity rate.
  3. [Introduction, §3.5] The claim that a 53.5% average accuracy on the SC-MCQA subset is a hallmark of a 'hard' dataset should be calibrated against the random-guessing baseline of the actual test distribution. With 487 of 2,667 questions having only 2 options (18.3%), the expected random accuracy is approximately 27.5%, so the top models are only about 26 points above random. A single aggregate number obscures that difficulty varies sharply by option count. The authors should report per-option-count accuracy and compare each against the corresponding random baseline, especially since the OptionsPert experiment shows near- or below-chance performance that needs a clearer baseline statement.
  4. [§4, Table 7] The human evaluation reporting is incomplete. The text states that 'Human experts achieved a maximum accuracy of 66.19% and a mean accuracy of 60.20% across three participants,' but Table 7 lists five participants, including expert E (61.11%, with 22 'don't know' responses) who is not included in the mean. The inclusion and exclusion criteria for the reported three participants should be stated. More importantly, the paper should provide an item-level analysis of where the experts disagree with the gold labels, since this directly bears on the benchmark's reliability claim.
minor comments (6)
  1. [Appendix A] The bullet lists in Appendix A contain malformed icon artifacts such as '/brainMCQA', '♂searchReAct', and '/da◎abase', which appear to be corrupted Unicode or Markdown remnants; these should be cleaned up.
  2. [§5] The heading reads 'LLMFeatureSelect' while the abstract and the GitHub repository use 'LLMFeatureSelector'; please unify the naming throughout the paper.
  3. [Table 2] The caption 'PDR ↑' is ambiguous because the table reports negative values; please clarify that the arrow denotes the direction of the perturbation and that negative values indicate a performance drop.
  4. [§3.4] The definition of UAcc is not fully explained, and the values in Table 3 do not appear to reproduce the stated formula (for example, mistral-large with Acc=60.83 and SS=3.02 would give about 45.0 with |Y|=5, not 50.03); please clarify the computation.
  5. [Appendix B, Algorithm 1] The pseudocode references the input field 'mc_targets' without defining it; add a one-line description of this field in the surrounding text.
  6. [§4, ReAct experiments] The description of the ReAct agent is sparse; the prompt template, tool-call format, and retrieval configuration (top-k, query formulation) should be provided either in the appendix or by reference to the released code.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all benchmark metrics are empirical evaluations against an externally constructed gold matrix, with no fitted parameter or self-citation chain feeding back into the conclusions.

full rationale

The paper's derivation chain is: (1) an ISO/expert-curated relevance matrix (Table 1 and Section 2.2: "Currently, we use ISO documents [11], along with expert-curated information, to establish an initial mapping"), (2) deterministic template-based question generation (Algorithm 1, Appendix B), and (3) empirical evaluation of LLMs and humans against the resulting gold answers. None of these steps defines its output in terms of the quantity it claims to measure. Accuracy, ACC@Consist, uncertainty set sizes, expert quiz scores, and asset-correlation observations are all measurements, not quantities derived from a model fit or from the benchmark's own claims. The paper contains no fitted parameter that is later renamed a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via self-citation. The only plausible concern is that the gold relevance matrix is both the source of the labels and curated by IBM experts, and the human study shows moderate agreement (kappa = 0.462), meaning labels are contested for some items; but that is a benchmark-validity and independence limitation, not a circular derivation. The Limitations section candidly notes the static-knowledge scope and uneven online knowledge availability, which further supports that the authors are not concealing a dependency loop. Therefore no circular step meets the evidentiary bar; score 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions rather than free parameters: the benchmark inherits the ISO/expert relevance mapping as ground truth, and the perturbation, exposure-proxy, and feature-selection analyses each assume an unvalidated relationship between their measurement and the target concept. Only the top-k choice in LLMFeatureSelector is a numeric design choice.

free parameters (1)
  • top-k in LLMFeatureSelector = 5
    Section 5 always reports the top-5 recommended sensors; the conclusion that 3 of 6 tasks include the max-correlated sensor depends on this hand-chosen number.
assumptions (4)
  • domain assumption The expert-curated relevance mapping from ISO documents is the ground truth for sensor-failure relationships across the 10 assets.
    Section 2.2: 'we use ISO documents [11], along with expert-curated information, to establish an initial mapping.' The entire dataset and all evaluations inherit this mapping.
  • ad hoc to paper Rephrased questions in ComplexPert preserve the same correct answers as the originals.
    Section 3.1 uses llama-3.3-70b-instruct to rephrase questions; no human verification or consistency check is reported, yet ACC@Consist treats original and perturbed labels as equivalent.
  • domain assumption Document counts from Wikipedia, arXiv, CrossRef, and Google approximate LLM training exposure for an asset.
    Section 3.6 compares asset-level accuracy with retrieved document counts and interprets the mild correlation as evidence of training exposure, an unvalidated proxy.
  • domain assumption Correlation between an LLM-recommended sensor and the target variable is a valid measure of feature selection quality.
    Section 5 and Table 8 score recommendations by absolute correlation with the next-timestep target, a linear proxy that may miss nonlinear or conditional relevance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes." pith.science (2026). https://pith.science/paper/CJSPAIOY

@misc{pith2026250603278,
  author       = {Pith},
  title        = {Pith review of: FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJSPAIOY}},
  note         = {Machine review of arXiv:2506.03278}
}
read the original abstract

We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.

Figures

Figures reproduced from arXiv: 2506.03278 by the authors.

Figure 1
Figure 1. Example of AI Tasks for Industry 4.0 Applications [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Question distribution by [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Real-world knowledge capacities assessed by ACC@Consist. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: Response Pattern Analysis between Original Data and Complex Perturbed Data on SC-MCQA for o1. Model SimplePert ComplexPert o1 -0.1204∗∗ -0.1065∗∗ o3-mini -0.0056 -0.0517∗∗ llama-4-mav. -0.0214∗∗ -0.0671∗∗ llama-4-scout -0.216∗∗ -0.2460∗∗ gpt-4.1 –0.0165 -0.0772∗∗ o1-pr…
Figure 6
Figure 6. Figure 6: Accuracy vs. Sample Volume by Asset Type 4 Experimental Results: Reasoning, Agent and Experts Impact of Reasoning-Based Prompting. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Accuracy comparison between Direct Prompting and ReAct across llama models [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Knowledge Graph Interaction B.3 Perturbed Pipeline The following [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Pipeline for Preparing Perturb Dataset 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Input Prompt. Ground truth is B, Exhaust Pressure. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Reasoning Trace by o1. Mistral-Large-Instruct-2407 Reasoning and Response Reasoning: Let me think step by step: 1. A fuel filter blockage would restrict the flow of fuel to the engine. 2. This restriction would lead to a reduction in the engine’s ability to burn fuel …
Figure 12
Figure 12. Figure 12: Reasoning Trace by Mistral-Large-Instruct-2407. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Reasoning Trace by Llama-3.3-70B. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Google: Accuracy vs. Sample Volume [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Wikipedia: Accuracy vs. Sample Volume [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: CrossRef: Accuracy vs. Sample Volume 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: LLMFeatureSelect input prompt and ChatGPT response. [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Screenshot from the quiz used for the user study. [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

    cs.CL 2026-05 conditional novelty 6.0 of 10

    In a 430k-evaluation study, plain baseline prompting beats most elaborate prompting techniques on non-reasoning LLMs across MCQA benchmarks, with only small role-framing variants gaining about 3 percentage points.

  2. Towards Building General Purpose Embedding Models for Industry 4.0 Agents

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Fine-tuning embedding models on nine ISO-derived industrial tasks yields large reported retrieval gains, but the evaluation is weakened by very small test sets and test-set hyperparameter selection.

  3. Retrieval-based and Fine-tuned LLM Approaches for Industrial Asset Health Monitoring and Decision Support

    cs.IR 2026-07 conditional novelty 3.0 of 10

    On IBM's FailureSensorIQ dataset, QLoRA-fine-tuned Qwen2.5-7B answers machine-failure diagnosis questions at 88% accuracy, outperforming semantic retrieval (73%) and zero/few-shot prompting (55%/54%).

Reference graph

Works this paper leans on

53 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [1]

    J., J AVAHERIPI , M., K AUFFMANN , P., ET AL

    ABDIN , M., A NEJA , J., B EHL , H., B UBECK , S., E LDAN , R., G UNASEKAR , S., H ARRISON , M., H EWETT , R. J., J AVAHERIPI , M., K AUFFMANN , P., ET AL . Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

    arXiv 2024

  2. [2]

    Rest meets react: Self-improvement for multi-step reasoning llm agent, 2023

    AKSITOV , R., M IRYOOSEFI , S., L I, Z., L I, D., B ABAYAN, S., K OPPARAPU , K., F ISHER , Z., G UO, R., P RAKASH , S., S RINIVASAN , P., Z AHEER , M., Y U, F., AND KUMAR , S. Rest meets react: Self-improvement for multi-step reasoning llm agent, 2023

    work page 2023

  3. [3]

    Qwen technical report

    BAI, J., B AI, S., C HU, Y., C UI, Z., D ANG , K., D ENG , X., F AN, Y., G E, W., H AN, Y., HUANG , F., ET AL . Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  4. [4]

    BALEPUR , N., AND RUDINGER , R. Is your large language model knowledgeable or a choices- only cheater? In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024) (Bangkok, Thailand, Aug. 2024), S. Li, M. Li, M. J. Zhang, E. Choi, M. Geva, P. Hase, and H. Ji, Eds., Association for Computational Linguistics, pp. 15–26

    work page 2024

  5. [5]

    Wind power forecasting, 2020

    B HASKARPANDIT , S. Wind power forecasting, 2020

    work page 2020

  6. [6]

    A coefficient of agreement for nominal scales

    COHEN , J. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46

    work page 1960

  7. [7]

    Opencompass: A universal evaluation platform for foundation models

    CONTRIBUTORS , O. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    work page 2023

  8. [8]

    Granite 3.0 language models, October 2024

    G RANITE TEAM , I. Granite 3.0 language models, October 2024

    work page 2024

Show all 53 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    GUO, D., Y ANG , D., Z HANG , H., S ONG , J., Z HANG , R., X U, R., Z HU, Q., M A, S., W ANG , P., BI, X., ET AL . Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  2. [10]

    O., AND PFISTER , T

    HAN, S., Y OON , J., A RIK , S. O., AND PFISTER , T. Large language models can automatically engineer features for few-shot tabular learning. In Proceedings of the 41st International Conference on Machine Learning (21–27 Jul 2024), R. Salakhutdinov, Z. Kolter, K. Heller, A. We...

    2024

  3. [11]

    Condition monitoring and diagnostics of machines — general guidelines

    ISO . Condition monitoring and diagnostics of machines — general guidelines. In Condition Monitoring and Diagnostics of Machines — General Guidelines (Geneva, Switzerland, 2018), International Organization for Standardization (ISO). This publication was last reviewed and confi...

    2018

  4. [12]

    P., L IPTON , Z

    JEONG , D. P., L IPTON , Z. C., AND RAVIKUMAR , P. Llm-select: Feature selection with large language models. arXiv preprint arXiv:2407.02694 (2024)

    2024 arXiv

  5. [13]

    Q., S ABLAYROLLES , A., M ENSCH , A., B AMFORD , C., C HAPLOT , D

    JIANG , A. Q., S ABLAYROLLES , A., M ENSCH , A., B AMFORD , C., C HAPLOT , D. S., DE LAS CASAS , D., B RESSAND , F., L ENGYEL , G., L AMPLE , G., S AULNIER , L., L AVAUD, L. R., LACHAUX , M.-A., S TOCK , P., S CAO, T. L., L AVRIL , T., W ANG , T., L ACROIX , T., AND SAYED, W. ...

    2023

  6. [14]

    Perteval: Unveiling real knowledge capacity of llms with knowledge-invariant perturbations, 2024

    LI, J., H U, R., H UANG , K., Z HUANG , Y., L IU, Q., Z HU, M., S HI, X., AND LIN, W. Perteval: Unveiling real knowledge capacity of llms with knowledge-invariant perturbations, 2024

    2024

  7. [15]

    Q., X IA, L., AND DU, X

    LI, R., W ANG , Z., T RAN , S. Q., X IA, L., AND DU, X. MEQA: A benchmark for multi-hop event-centric question answering with explanations. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2024)

    2024

  8. [16]

    Are clinical t5 models better for clinical text?, 2024

    LI, Y., H ARRIGIAN , K., Z IRIKLY, A., AND DREDZE , M. Are clinical t5 models better for clinical text?, 2024

    2024

  9. [17]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    LIN, S., H ILTON , J., AND EVANS, O. Truthfulqa: Measuring how models mimic human falsehoods, 2022. 10

    2022

  10. [18]

    E., M OTZFELDT , A

    LIÉVIN , V., H OTHER , C. E., M OTZFELDT , A. G., AND WINTHER , O. Can large language models reason about medical questions?, 2023

    2023

  11. [19]

    A., A NTYPAS , D., B ORKAKOTY , H., KIM, E., P EREZ -A LMENDROS , C., A YELE , A

    MYUNG , J., L EE, N., Z HOU , Y., J IN, J., P UTRI , R. A., A NTYPAS , D., B ORKAKOTY , H., KIM, E., P EREZ -A LMENDROS , C., A YELE , A. A., B ASULTO , V. G., I BANEZ -GARCIA , Y., LEE, H., M UHAMMAD , S. H., P ARK , K., R ZAYEV, A. S., W HITE , N., Y IMAM , S. M., PILEHVAR ,...

    2024

  12. [20]

    Human-in-the-loop large-scale predictive maintenance of workstations

    NIKITIN , A., AND KASKI , S. Human-in-the-loop large-scale predictive maintenance of workstations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022), pp. 3682–3690

    2022

  13. [21]

    Predictive maintenance dataset - air compressor, 2023

    O KUDAN , A. Predictive maintenance dataset - air compressor, 2023

    2023

  14. [22]

    o1-preview, 2024

    O PENAI. o1-preview, 2024

    2024

  15. [23]

    L., M ISHKIN , P., CLARK , T., P EEBLES , J., A MODEI , D., ET AL

    OUYANG , L., W U, J., J IANG , X., A LMEIDA , D., WAINWRIGHT , C. L., M ISHKIN , P., CLARK , T., P EEBLES , J., A MODEI , D., ET AL . The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2210.11416 (2022)

    2022 arXiv

  16. [24]

    CiteME: Can language models accurately cite scientific claims? In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2024)

    PRESS , O., H OCHLEHNERT , A., P RABHU , A., U DANDARAO , V., P RESS , O., AND BETHGE , M. CiteME: Can language models accurately cite scientific claims? In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2024)

    2024

  17. [25]

    R., K OTHA , R., G UDA, V., AND RAMADEVI , Y

    PUTCHALA , S. R., K OTHA , R., G UDA, V., AND RAMADEVI , Y. Transformer data analysis for predictive maintenance. In Proceedings of Second International Conference on Advances in Computer Engineering and Communication Systems: ICACECS 2021 (2022), Springer, pp. 217–230

    2022

  18. [26]

    L., S TICKLAND , A

    REIN , D., H OU, B. L., S TICKLAND , A. C., P ETTY, J., P ANG , R. Y., D IRANI , J., M ICHAEL , J., AND BOWMAN , S. R. Gpqa: A graduate-level google-proof qa benchmark, 2023

    2023

  19. [27]

    M., AND WINGATE , D

    ROBINSON , J., R YTTING , C. M., AND WINGATE , D. Leveraging large language models for multiple choice question answering, 2023

    2023

  20. [28]

    Large language models for failure mode classification: An investigation, 2023

    STEWART, M., H ODKIEWICZ , M., AND LI, S. Large language models for failure mode classification: An investigation, 2023

    2023

  21. [29]

    Gemini: A family of highly capable multimodal models, 2024

    TEAM , G., A NIL , R., B ORGEAUD , S., AND ETC . Gemini: A family of highly capable multimodal models, 2024

    2024

  22. [30]

    S., L OVE, J., ET AL

    TEAM , G., M ESNARD , T., H ARDIN , C., D ADASHI , R., B HUPATIRAJU , S., P ATHAK , S., SIFRE , L., R IVIÈRE , M., K ALE , M. S., L OVE, J., ET AL . Gemma: Open models based on gemini research and technology, 2024. URL https://arxiv. org/abs/2403.08295 2 (2024), 10–19

    2024 arXiv

  23. [31]

    TOUVRON , H., M ARTIN , L., S TONE , K., A LBERT , P., A LMAHAIRI , A., B ABAEI , Y., BASHLYKOV , N., B ATRA, S., B HARGAVA , P., B HOSALE , S., B IKEL , D., B LECHER , L., FERRER , C. C., C HEN , M., C UCURULL , G., E SIOBU , D., F ERNANDES , J., F U, J., F U, W., FULLER , B....

    2023

  24. [32]

    DireCT: Diagnostic reasoning for clinical notes via large language models

    WANG , B., C HANG , J., Q IAN , Y., CHEN , G., C HEN , J., J IANG , Z., Z HANG , J., N AKASHIMA , Y., AND NAGAHARA , H. DireCT: Diagnostic reasoning for clinical notes via large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets a...

    2024

  25. [33]

    K.-W., AND LIM, E.-P

    WANG , L., X U, W., L AN, Y., H U, Z., L AN, Y., L EE, R. K.-W., AND LIM, E.-P. Plan-and- solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    2023

  26. [34]

    MMLU-pro: A more robust and challenging multi-task language understanding benchmark

    WANG , Y., M A, X., Z HANG , G., N I, Y., C HANDRA , A., G UO, S., R EN, W., A RULRAJ , A., H E, X., J IANG , Z., L I, T., K U, M., W ANG , K., Z HUANG , A., F AN, R., Y UE, X., AND CHEN , W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. ...

    2024

  27. [35]

    W., J IAO, Y

    WEI, J., K ARINA , N., C HUNG , H. W., J IAO, Y. J., P APAY, S., G LAESE , A., S CHULMAN , J., AND FEDUS , W. Measuring short-form factuality in large language models, 2024

    2024

  28. [36]

    H., L E, Q

    WEI, J., W ANG , X., S CHUURMANS , D., B OSMA , M., I CHTER , B., X IA, F., C HI, E. H., L E, Q. V., AND ZHOU , D. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing System...

    2022

  29. [37]

    Benchmarking complex instruction-following with multiple constraints composition

    WEN, B., K E, P., G U, X., W U, L., H UANG , H., Z HOU , J., L I, W., H U, B., G AO, W., X U, J., LIU, Y., TANG , J., WANG , H., AND HUANG , M. Benchmarking complex instruction-following with multiple constraints composition. In The Thirty-eight Conference on Neural Informatio...

    2024

  30. [38]

    Wizardlm: Empowering large language models to follow complex instructions, 2023

    XU, C., S UN, Q., Z HENG , K., G ENG , X., Z HAO, P., F ENG , J., T AO, C., AND JIANG , D. Wizardlm: Empowering large language models to follow complex instructions, 2023

    2023

  31. [39]

    D., H O, J

    XU, R., S HI, W., Y U, Y., Z HUANG , Y., Z HU, Y., W ANG , M. D., H O, J. C., Z HANG , C., AND YANG , C. BMRetriever: Tuning large language models as better biomedical text retrievers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (M...

    2024

  32. [40]

    Predictive maintenance for general aviation using convolutional transformers

    YANG , H., L ABELLA , A., AND DESELL , T. Predictive maintenance for general aviation using convolutional transformers. In Proceedings of the AAAI Conference on Artificial Intelligence (2022), vol. 36, pp. 12636–12642

    2022

  33. [41]

    Predictive maintenance for general aviation using convolutional transformers, 2022

    YANG , H., L ABELLA , A., AND DESELL , T. Predictive maintenance for general aviation using convolutional transformers, 2022

    2022

  34. [42]

    W., S ALAKHUTDINOV , R., AND MANNING , C

    YANG , Z., Q I, P., Z HANG , S., B ENGIO , Y., C OHEN , W. W., S ALAKHUTDINOV , R., AND MANNING , C. D. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)

    2018

  35. [43]

    React: Synergizing reasoning and acting in language models

    YAO, S., Z HAO, J., Y U, D., D U, N., S HAFRAN , I., N ARASIMHAN , K., AND CAO, Y. React: Synergizing reasoning and acting in language models. ArXiv abs/2210.03629 (2022)

    2022 arXiv

  36. [44]

    Benchmarking llms via uncertainty quantification

    YE, F., Y ANG , M., P ANG , J., W ANG , L., W ONG , D., Y ILMAZ , E., S HI, S., AND TU, Z. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems 37 (2024), 15356–15385

    2024

  37. [45]

    Automating dataset updates towards reliable and timely evaluation of large language models

    YING , J., C AO, Y., BAI, Y., S UN, Q., W ANG , B., T ANG , W., D ING , Z., Y ANG , Y., H UANG , X., AND YAN, S. Automating dataset updates towards reliable and timely evaluation of large language models. In The Thirty-eight Conference on Neural Information Processing Systems ...

    2024

  38. [46]

    Trustuqa: A trustful framework for unified structured data question answering, 2024

    ZHANG , W., J IN, L., Z HU, Y., C HEN , J., H UANG , Z., W ANG , J., H UA, Y., L IANG , L., AND CHEN , H. Trustuqa: A trustful framework for unified structured data question answering, 2024. 12

    2024

  39. [47]

    Multiple-choice questions are efficient and robust llm evaluators, 2024

    ZHANG , Z., J IANG , Z., X U, L., H AO, H., AND WANG , R. Multiple-choice questions are efficient and robust llm evaluators, 2024

    2024

  40. [48]

    Large language models are not robust multiple choice selectors, 2024

    ZHENG , C., Z HOU , H., M ENG , F., Z HOU , J., AND HUANG , M. Large language models are not robust multiple choice selectors, 2024

    2024

  41. [49]

    f ailure_mode

    ZHU, Y., D U, S., L I, B., L UO, Y., AND TANG , N. Are large language models good statis- ticians? In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- ber 10 - 15,...

    2024

  42. [50]

    A fuel filter blockage would restrict the flow of fuel to the engine

  43. [51]

    This restriction would lead to a reduction in the engine’s ability to burn fuel efficiently

  44. [52]

    Inefficient fuel burning would result in a decrease in output power and an increase in exhaust pressure due to incomplete combustion

  45. [53]

    https://en.wikipedia.org/w/api.php

    Other sensors like air flow, cylinder pressure, and engine temperature might not directly indicate a fuel filter blockage as they are influenced by various factors. Therefore, the most indicative sensors for a fuel filter blockage would be exhaust pressure and output power. An...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.