pith. sign in

arxiv: 2606.11204 · v1 · pith:QYQ3QVAKnew · submitted 2026-04-22 · 💻 cs.CL · cs.IR

Benchmarking Large Language Models for Safety Data Extraction

Jonas Grill , Thomas Bayer , S\"oren Berlinger This is my paper

Pith reviewed 2026-07-05 02:44 UTC · model glm-5.2

classification 💻 cs.CL cs.IR
keywords dataextractionaccuracyacrossmodelssafetychain-of-thoughtclaude
0
0 comments X

The pith

Text-based LLM extraction hits 84% on safety data sheets, falls short of autonomous use

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper benchmarks four large language models — Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, and Llama 3.1-70B — for automatically extracting structured information from Safety Data Sheets (SDS), the regulatory documents that accompany hazardous chemicals in industrial settings. The authors test each model across three prompting strategies (zero-shot, few-shot, chain-of-thought) and two processing pipelines: text-based (PDF converted to Markdown first) and multimodal (PDF fed directly as an image). Across more than 50,000 extracted fields from ten SDS documents, text-based extraction consistently beat multimodal processing by 4–9 percentage points in accuracy while also being faster and cheaper. Gemini 1.5 Pro with chain-of-thought prompting achieved the best accuracy at 84%, ahead of GPT-4o (81%) and Claude 3.7 Sonnet (79%), with Llama 3.1-70B trailing at 66–71%. No configuration reached the 90% accuracy threshold that industrial safety applications typically require. The authors conclude that general-purpose LLMs are promising but not yet reliable enough for unsupervised deployment, and that model choice matters far more than prompting strategy.

Core claim

The central finding is that converting PDFs to text before feeding them to an LLM yields substantially better, faster, and cheaper extraction than asking multimodal models to process the PDF as an image — a result that held across every model tested. Within text-based processing, Gemini 1.5 Pro with chain-of-thought prompting was the best single configuration at 84% accuracy, but the gap between the top three commercial models was narrow (84% vs 81% vs 79%), and all fell short of the 90% bar for autonomous use. The open-source Llama 3.1-70B lagged considerably. A secondary finding is that few-shot prompting did not help: zero-shot matched or exceeded few-shot accuracy while costing less, an

What carries the argument

The central object is the comparison matrix of model × prompting strategy × processing pipeline (text vs. multimodal), evaluated on a fixed benchmark of ten SDS documents with 235 fields each, scored on accuracy, false-positive rate, not-found rate, BERTScore, latency, and token cost. The weighted performance score (70% accuracy, 20% latency, 10% cost) serves as the unified ranking mechanism.

If this is right

  • Industrial SDS processing pipelines should prefer text-based extraction over multimodal approaches whenever the source PDF contains embedded text, saving both accuracy and cost.
  • Human-in-the-loop verification remains necessary for safety-critical fields (hazards, first aid, PPE) until accuracy exceeds 90%.
  • Domain-specific fine-tuning on SDS documents could plausibly close the 6-point gap between the best current result (84%) and the autonomous deployment threshold (90%).
  • Zero-shot prompting may be preferable to few-shot for structured extraction tasks on long documents, avoiding the 'lost in the middle' attention degradation that few-shot exemplars introduce.
  • Open-source models at the 70B parameter scale remain substantially behind commercial models for structured extraction from technical documents, at least without domain adaptation.

Load-bearing premise

The evaluation uses only ten SDS documents with manually validated ground truth but no reported inter-annotator agreement and no statistical significance testing between model configurations. The 84% vs 81% vs 79% differences between the top three models could fall within noise at this sample size, making the model ranking uncertain.

What would settle it

If a replication on a larger, independently annotated SDS dataset (e.g., 100+ documents with inter-annotator agreement) showed no statistically significant difference between Gemini, GPT-4o, and Claude, or if multimodal processing matched or exceeded text-based extraction on such a dataset, the paper's central ranking and pipeline recommendations would not hold.

read the original abstract

Accurate extraction of structured information from Safety Data Sheets (SDS) remains challenging in industrial safety due to heterogeneous document formats and the limitations of traditional rule-based methods. This study benchmarks state-of-the-art Large Language Models (LLMs) for automated SDS data extraction, comparing text-based and multimodal processing pipelines. We systematically evaluate four models: Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, and Llama 3.1-70B, across three prompting strategies: zero-shot, few-shot, and chain-of-thought. The evaluation framework assessed accuracy, latency, and cost across more than 50,000 extracted data fields. Results show that text-based extraction consistently outperforms multimodal processing across all metrics. Gemini 1.5 Pro combined with a Chain-of-Thought prompt achieved the highest accuracy (84%), outperforming GPT-4o (81%) and Claude 3.7 Sonnet (79%). However, no model surpassed the 90% accuracy threshold commonly required for reliable real-world deployment. These findings indicate that general-purpose LLMs are not yet robust enough for unsupervised industrial use, though performance suggests strong potential with task-specific fine-tuning. Future research should focus on domain-adapted training, model calibration, and the integration of Human-in-the-Loop verification to ensure safety-critical reliability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 7 minor

Summary. This paper benchmarks four LLMs (Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, Llama 3.1-70B) on structured data extraction from Safety Data Sheets (SDS), comparing text-based and multimodal pipelines across three prompting strategies (zero-shot, few-shot, chain-of-thought). Using 10 SDS documents with 235 fields each, the authors find that text-based extraction outperforms multimodal processing, that Gemini 1.5 Pro with chain-of-thought achieves the highest accuracy (84%), and that no configuration reaches a 90% accuracy threshold deemed necessary for autonomous deployment. A weighted score combining accuracy, latency, and cost is used for aggregate comparison.

Significance. The paper addresses a practically important problem in industrial safety compliance and provides a systematic, controlled comparison across model architectures, modalities, and prompting strategies. The inclusion of latency and cost alongside accuracy is a strength, as is the schema-constrained evaluation protocol. The finding that zero-shot matches or exceeds few-shot performance is interesting and well-motivated by the 'lost in the middle' literature. The negative result (no model reaching 90%) is useful for the community. However, the significance of the comparative ranking claims is limited by the small dataset and the confounded nature of the headline comparison.

major comments (4)
  1. §4.2, Table 3/Table 4: The headline ranking 'Gemini (84%) > GPT-4o (81%) > Claude (79%)' compares each model's best configuration across different prompting strategies (Gemini's 84% is with chain-of-thought; GPT-4o's 81% is with zero-shot). Within-strategy comparisons tell a different story: for zero-shot, Gemini (0.82) and GPT-4o (0.81) are nearly tied; for few-shot, GPT-4o (0.80) outperforms Gemini (0.79). Only for chain-of-thought does Gemini clearly lead. The paper should either (a) present the ranking as 'best configuration per model' with this caveat stated explicitly, or (b) provide within-strategy comparisons as the primary analysis. As written, the comparative ranking is confounded by prompting strategy and is not a controlled comparison.
  2. §3.3, §3.4: No statistical significance testing is reported. With only 10 SDS documents, the differences between top configurations (84% vs 81% vs 79%) may not be statistically meaningful. The 50,000 extracted fields figure reflects 235 fields × 21 configurations, not 50,000 independent observations; the effective sample size for any single configuration is 10 documents × 235 fields = 2,350 fields, and the document-level n is 10. The authors should add significance tests (e.g., bootstrap confidence intervals or paired tests at the document level) or substantially soften the ranking claims.
  3. §3.3: No inter-annotator agreement is reported for the manually validated ground truth. The authors acknowledge this in §4.5, but the entire evaluation rests on the correctness of this ground truth. At minimum, a subset of documents should be double-annotated and agreement reported (e.g., Cohen's κ or similar). Without this, it is unclear how much of the observed error is model error versus annotation ambiguity, particularly for free-text fields like First Aid Measures.
  4. §3.3, Eq. for Score: The weighted score formula uses weights 0.7/0.2/1 for accuracy/latency/cost, but these are not justified. The min-max normalization across configurations means the score is relative to the specific set of models tested. The paper should either justify these weights (e.g., from domain requirements) or present a sensitivity analysis showing whether the ranking changes under alternative weightings.
minor comments (7)
  1. §3.3 states 'approximately 50,000 labeled fields' but 235 fields × 10 documents = 2,350 fields per configuration. The 50,000 figure appears to be 2,350 × 21 configurations. This should be clarified to avoid misleading readers about the dataset size.
  2. §4.3 and §4.4 have identical titles ('Aggregated Cost-Function Evaluation') and substantially overlapping content. One should be removed or merged.
  3. Table 3: The BERTScore column header and values are present but the metric is not discussed in the results section. Either discuss BERTScore findings or note that it is reported for completeness.
  4. §3.3: The accuracy formula includes TN (true negatives / correct omissions). For SDS fields, it would help to clarify how 'correct omissions' are defined—i.e., when is a field considered absent from the ground truth?
  5. Figure references: 'Fig. 3.1' and 'Fig. 3.3' are used in the text but figures are numbered 2, 3, 4. Cross-references should be corrected.
  6. §3.4: 'Claude 3.7 Sonnet' is referenced; as of the manuscript date, the latest Claude 3.x Sonnet is 3.5. If 3.7 is correct, a citation or clarification would help; if it is a typo, it should be corrected.
  7. Table 4: Llama 3.1-70B multimodal results are marked '–' (not applicable) but no explanation is given in the text for why multimodal experiments were not run for this model. A brief note would help.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points concerning (1) confounded model comparisons across prompting strategies, (2) absence of statistical significance testing, (3) lack of inter-annotator agreement for the ground truth, and (4) unjustified weights in the composite score. We agree that all four points identify genuine weaknesses in the current manuscript. Below we address each point and describe the revisions we will make.

read point-by-point responses

  1. Referee: §4.2, Table 3/Table 4: The headline ranking 'Gemini (84%) > GPT-4o (81%) > Claude (79%)' compares each model's best configuration across different prompting strategies (Gemini's 84% is with chain-of-thought; GPT-4o's 81% is with zero-shot). Within-strategy comparisons tell a different story. The paper should either (a) present the ranking as 'best configuration per model' with this caveat stated explicitly, or (b) provide within-strategy comparisons as the primary analysis.

    Authors: The referee is correct. The headline ranking as currently presented conflates model identity with prompting strategy, making it an uncontrolled comparison. We will revise the manuscript to adopt option (a): we will explicitly label the ranking as 'best configuration per model' and state clearly that the top configurations use different prompting strategies (Gemini with chain-of-thought, GPT-4o with zero-shot). We will additionally add a within-strategy comparison table or paragraph showing that for zero-shot, Gemini (0.82) and GPT-4o (0.81) are nearly tied; for few-shot, GPT-4o (0.80) outperforms Gemini (0.79); and only for chain-of-thought does Gemini clearly lead (0.84 vs. 0.79). This within-strategy analysis will be presented alongside the best-configuration ranking so that readers can interpret both perspectives. The abstract and conclusion will be adjusted to avoid implying a controlled cross-strategy comparison. revision: yes

  2. Referee: §3.3, §3.4: No statistical significance testing is reported. With only 10 SDS documents, the differences between top configurations (84% vs 81% vs 79%) may not be statistically meaningful. The 50,000 extracted fields figure reflects 235 fields × 21 configurations, not 50,000 independent observations. The authors should add significance tests (e.g., bootstrap confidence intervals or paired tests at the document level) or substantially soften the ranking claims.

    Authors: We agree that no statistical significance testing was reported and that the effective sample size for any single configuration is 10 documents (with 235 fields per document, which are not independent observations). We will add bootstrap confidence intervals at the document level for the top configurations and, if feasible, paired permutation or Wilcoxon signed-rank tests comparing the top configurations pairwise across the 10 documents. We acknowledge in advance that with n=10 documents, the power to detect differences of 2–5 percentage points is limited, and we expect that some of the observed differences may not reach statistical significance. We will report the results transparently and substantially soften all ranking claims accordingly, replacing language such as 'outperforming' with 'achieving higher accuracy in our sample.' We will also clarify the description of the 50,000-field figure to explicitly state that this reflects 235 fields × 21 configurations × 10 documents and is not a count of independent observations. revision: yes

  3. Referee: §3.3: No inter-annotator agreement is reported for the manually validated ground truth. The authors acknowledge this in §4.5, but the entire evaluation rests on the correctness of this ground truth. At minimum, a subset of documents should be double-annotated and agreement reported (e.g., Cohen's κ or similar).

    Authors: This is a fair and important point. The current ground truth was manually validated by a single annotator, and without inter-annotator agreement we cannot distinguish model error from annotation ambiguity, particularly for free-text fields such as First Aid Measures. We will have a second annotator independently label a subset of the SDS documents (we plan to double-annotate at least 3 of the 10 documents, covering fields with high ambiguity including First Aid Measures, Handling and Storage, and compositional data). We will report Cohen's κ (or percent agreement where κ is ill-defined for free-text fields) and discuss the implications for interpreting model errors. If the double-annotation reveals substantial disagreement on certain field types, we will note this as a limitation affecting the interpretation of accuracy scores for those fields specifically. revision: partial

  4. Referee: §3.3, Eq. for Score: The weighted score formula uses weights 0.7/0.2/0.1 for accuracy/latency/cost, but these are not justified. The min-max normalization across configurations means the score is relative to the specific set of models tested. The paper should either justify these weights (e.g., from domain requirements) or present a sensitivity analysis showing whether the ranking changes under alternative weightings.

    Authors: The referee is correct that the weights are not currently justified and that min-max normalization makes the score relative to the tested configurations. We will address this in two ways. First, we will add a domain-motivated justification: in industrial safety compliance, accuracy is paramount because errors can have safety consequences, so we weight it heavily (0.7); latency and cost are secondary operational concerns, with latency weighted higher than cost (0.2 vs. 0.1) because processing throughput affects deployment feasibility more directly than per-call token cost. Second, we will add a sensitivity analysis showing the composite score rankings under alternative weightings (e.g., 0.5/0.25/0.25, 0.9/0.05/0.05, 0.33/0.33/0.33) to demonstrate whether the ranking is robust to weight choice. We will also add an explicit note that the min-max normalization makes scores relative to the evaluated configuration set and not absolute. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with externally defined ground truth and no derivation chain to evaluate

full rationale

This paper is an empirical benchmark study evaluating four LLMs on SDS field extraction against a manually validated ground truth. There is no derivation chain, no first-principles prediction, and no fitted-parameter-to-prediction loop. The weighted score formula (Score = 0.7·Accuracy_norm + 0.2·Time_norm + 0.1·Cost_norm) is an explicitly stated design choice with fixed weights, not a derived result. The accuracy metric (TP+TN)/(TP+FP+FN+TN) is a standard definition applied to external data. No model parameters are fitted to the evaluation data and then 'predicted' back. The ground truth is independently constructed from manual validation of SDS documents. While the paper has methodological limitations (small sample size, no significance testing, cross-strategy comparison issues), none of these constitute circularity in the sense of a derivation reducing to its inputs by construction. The self-citations present are to external standards (GHS, REACH) and prior work by other authors, not to the present authors' own unverified prior results. The paper is self-contained as an empirical evaluation.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

No invented entities; this is an empirical benchmark.

free parameters (1)
  • Weighted score weights = 0.7 (accuracy), 0.2 (latency), 0.1 (cost)
    Chosen by the authors in §3.3 without justification or sensitivity analysis. These weights directly determine the composite ranking in Table 3 and §4.3.
axioms (3)
  • domain assumption Manually validated ground truth is correct and consistent
    §3.4 states 'all reference labels independently verified for consistency and correctness' but no inter-annotator agreement is reported. The entire evaluation depends on this ground truth being accurate.
  • domain assumption Ten SDS documents are representative of industrial SDS heterogeneity
    §3.3 selects ten documents from ChemicalSafety.com. The paper claims these represent 'diverse range of manufacturers' but with n=10, generalizability is untested.
  • domain assumption Binary field-level matching captures extraction quality adequately
    §3.3 defines accuracy as binary match (TP+TN)/(TP+FP+FN+TN). Partial correctness (e.g., extracting 'rinse with water' vs 'rinse skin with water/shower') is scored as 0, which may understate semantic extraction quality despite the supplementary BERTScore.

pith-pipeline@v1.1.0-glm · 12197 in / 2885 out tokens · 216532 ms · 2026-07-05T02:44:08.693226+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 1 internal anchor

  1. [1]

    (REACH) (2020)

    Commission, E.: Commission Regulation (EU) 2020/878 ... (REACH) (2020). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32020R0878

    work page 2020

  2. [2]

    Technical report, U.S

    Occupational Safety and Health Administration: Hazard communication standard (hcs). Technical report, U.S. Department of Labor (2012)

    work page 2012

  3. [3]

    Heliyon (2025)

    Khan, M., et al.: A machine learning–driven automated system to extract multiple information fields from safety data sheet documents. Heliyon (2025)

    work page 2025

  4. [4]

    Matlhare, M.: Enhancing chemical safety and regulatory compliance through safety data sheets (2024)

    work page 2024

  5. [5]

    16168894, CAS 7664-93-9)

    Univar Solutions USA, Inc.: Safety Data Sheet: Sulfuric Acid 50%, Technical Grade (Product No. 16168894, CAS 7664-93-9). Revision Date: 10/10/2023. Available at: https://www.univarsolutions.com. Accessed: 12 December 2025 (2023)

    work page 2023

  6. [6]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 6000–6010. Curran Associates Inc., Red Hook, NY, USA (2017)

    work page 2017

  7. [7]

    Zhang, Q., et al.: Document parsing unveiled: Techniques, challenges, and prospects for structured information extraction (2024)

    work page 2024

  8. [8]

    Nature Communications (2024)

    Dagdelen, J., et al.: Structured information extraction from scientific text with large language models. Nature Communications (2024)

    work page 2024

  9. [9]

    Opitz, J.: A closer look at classification evaluation metrics and a critical reflection of common evaluation practice (2024) 16

    work page 2024

  10. [10]

    Vatsal, S., Dubey, H.: A survey of prompt engineering methods in large language models for different nlp tasks (2024)

    work page 2024

  11. [11]

    Sahoo, P., et al.: A systematic survey of prompt engineering in large language models: Techniques and applications (2025)

    work page 2025

  12. [12]

    Schulhoff, S., et al.: The prompt report: A systematic survey of prompt engineer- ing techniques (2025)

    work page 2025

  13. [13]

    Cheng, X., et al.: Revisiting chain-of-thought prompting: Zero-shot can be stronger than few-shot (2025)

    work page 2025

  14. [14]

    In: ACM Symposium on Document Engineering (2021)

    Fenton, K., Simske, S.J.: Engineering of an artificial intelligence safety data sheet document processing system for ehs compliance. In: ACM Symposium on Document Engineering (2021)

    work page 2021

  15. [15]

    ACS Omega (2023)

    Fenton, K., Simske, S.J., Luu, J.: Mitigation of chemical reporting liabili- ties through systematic modernization of chemical hazard and safety data management systems. ACS Omega (2023)

    work page 2023

  16. [16]

    Heliyon (2025)

    Khan, M., Penfield, J., Suman, A.: A machine learning driven automated system to extract multiple information fields from safety data sheet documents. Heliyon (2025)

    work page 2025

  17. [17]

    Scientific Reports (2024)

    Suman, A., Khan, M., Talreja, V.: A machine learning driven automated system for safety data sheet indexing. Scientific Reports (2024)

    work page 2024

  18. [18]

    Annals of Work Exposures and Health (2025)

    Pekel, F., Kalkman, G., Lemcke, E.: Implementing generative pretrained trans- former models for text recognition tasks in safety data sheets. Annals of Work Exposures and Health (2025)

    work page 2025

  19. [19]

    Wiley Interdisciplinary Reviews: Computational Toxicology (2024)

    Moreira-Filho, J., Ranganath, D., Mansouri, K.: Automating data extraction from scientific literature and general pdf files using llms and knime. Wiley Interdisciplinary Reviews: Computational Toxicology (2024)

    work page 2024

  20. [20]

    Chemical Society Reviews (2025)

    Schilling-Wilhelmi, M., Rios-Garcia, M., Shabih, S.: From text to insight: Large language models for chemical data extraction. Chemical Society Reviews (2025)

    work page 2025

  21. [21]

    arXiv preprint arXiv:2402.XXXX (2024)

    Mansouri, K., et al.: Chemteb: Benchmarking chemical text embeddings for clas- sification and information extraction. arXiv preprint arXiv:2402.XXXX (2024)

    work page 2024

  22. [22]

    In: IEEE International Conference on Semantic Computing (2025)

    Lu, B., Pham, D., Chang, T.-C.: Shacl-skos based knowledge representation of material safety data sheets. In: IEEE International Conference on Semantic Computing (2025)

    work page 2025

  23. [23]

    In: International Conference on Learning Representations (ICLR) (2020) 17

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Eval- uating text generation with bert. In: International Conference on Learning Representations (ICLR) (2020) 17

    work page 2020

  24. [24]

    Lost in the Middle: How Language Models Use Long Contexts

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the Middle: How Language Models Use Long Contexts (2023). https: //arxiv.org/abs/2307.03172 18

    work page internal anchor Pith review Pith/arXiv arXiv 2023