{"id":"dcd1a6c0-e726-4ffd-a430-6ab6132c9665","arxiv_id":"2505.22959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HSE-Bench is a new 1,020-question LLM benchmark for HSE compliance reasoning, and the paper claims LLMs rely on semantic matching rather than structured legal reasoning.","lead":"This paper introduces HSE-Bench, a set of over 1,000 multiple-choice questions that test how well large language models assess health, safety, and environment compliance. It reports that models score high when answer choices are given but much lower in open-ended settings, and proposes a multi-expert prompting method that improves accuracy by a few points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic-matching finding rests on an unvalidated comparison between with-options accuracy and BERT-cosine AUC-ROC without options; the gap may be a measurement artifact.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the unjustified comparability of accuracy and AUC-ROC as evidence for the semantic-matching claim. My analysis confirms that this comparison is confounded by two simultaneous changes (options presence and scoring method), and that no control experiment or proxy validation is provided. The paper's 'AUC-ROC is typically higher than or equal to accuracy' argument is a mathematical statement about the same ranking scores, not about the comparability of these two different measurement pipelines. The concrete test I propose would settle whether the BERT-cosine proxy understates true open-ended answer quality; if it does, the central finding collapses to a measurement artifact. This is addressable with additional validation, so the paper should not be rejected, but the current evidence is insufficient to accept the semantic-matching claim as stated. The reader's CONDITIONAL verdict is appropriate, and my stress-test does not change it.","tokens_in":22489,"tokens_out":4140,"duration_ms":43808,"concrete_test":"Take a stratified random sample of 200 HSE-Bench questions (50 per source). For each, collect free-form answers from DeepSeek-R1 and GPT-4o-mini without options. Have two HSE-expert annotators independently rank the four options by how well each matches the free-form answer, and compute an annotator-based AUC-ROC using the same pairwise formula as the paper. Also have annotators mark each free-form answer as correct/incorrect as a standalone response. If the annotator-based AUC-ROC is substantially higher than the BERT-based AUC-ROC (e.g., >10 points) or the standalone correctness rate approaches the with-options accuracy, the accuracy-vs-AUC gap is an artifact of the BERT proxy and the semantic-matching conclusion is unsupported. If annotator-based AUC-ROC matches the BERT-based value, the conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that LLMs rely on semantic matching rather than principled HSE reasoning is inferred from a large drop between accuracy (options provided, direct selection) and AUC-ROC (options omitted, BERT cosine similarity between a free-form answer and each option). This comparison conflates two changes: input format (options present vs absent) and scoring mechanism (direct classification vs embedding-similarity ranking). BERT cosine similarity is a noisy proxy for answer correctness: a correct free-form answer using different legal terminology may score low, while a generic answer may accidentally resemble the correct option. The paper provides no validation of this proxy, no alternative scoring (e.g., human judgment of the same free-form answers, or a second embedding model), and no control for the well-known difficulty difference between recognition and open-ended generation. The case studies in Table 4 are qualitative and show only that reasoning traces differ, not that the AUC-ROC gap is causally due to semantic matching. Consequently, the headline finding about semantic matching is not yet established; the observed gap could be an artifact of the BERT-based measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HSE-Bench, a benchmark of 1,020 multiple-choice questions for Health, Safety, and Environment compliance assessment, constructed by prompting GPT-4o to convert regulations, court cases, safety exams, and fieldwork-video transcripts into IRAC-structured (Issue, Rule, Application, Conclusion) questions, followed by adversarial augmentation and expert double-checking. The authors evaluate 12 LLMs under six prompting strategies and report two metrics: selection accuracy when options are provided and BERT-cosine-based AUC-ROC when options are omitted. They observe a large gap (~24.73% on average) between the two and conclude that LLMs rely on semantic matching rather than principled HSE reasoning; they also find that reasoning models do not beat foundation models and that their proposed Reasoning of Experts (RoE) prompt improves accuracy, with up to 25.01% over CoT on DeepSeek-R1.","tokens_in":22692,"tokens_out":6700,"duration_ms":64388,"significance":"The benchmark and code are publicly released, the data sources are diverse, and the IRAC-based structure is a useful contribution to evaluating LLMs in a high-stakes regulatory domain. The RoE prompt is a simple, plausible intervention, and the reported average improvements are worth investigating. However, the headline conclusion that 'capabilities largely rely on semantic matching' rests on a comparison of two different tasks with two different scoring functions, and the benchmark labels are themselves largely GPT-4o-generated; both issues need to be resolved before the central claims can be accepted. The paper also omits numerical tables, error bars, and human-verification statistics, which are needed to support several secondary claims.","major_comments":[{"comment":"The semantic-matching conclusion is not supported by the reported experiment because accuracy (options provided, direct selection) and AUC-ROC (options omitted, BERT cosine similarity) differ in both input format and scoring mechanism. A drop from ~90% accuracy to ~70% AUC-ROC is therefore compatible with open-ended generation being harder or with BERT similarity being a noisy proxy, rather than with semantic matching. To support the claim, the authors should validate the BERT-similarity score against human judgments on the same free-form outputs (or a second embedding model / log-probability confidence), and ideally compare like with like by scoring both settings with the same mechanism (e.g., forced-choice accuracy on options vs. open-form answers judged by humans or a calibrated model). The case studies in Table 4 are anecdotal and do not quantify the causal role of semantic matching.","section":"Section 4.1 and 4.2.1, Figure 2"},{"comment":"The benchmark labels are generated by GPT-4o and only briefly described as double-checked by two experts; no expert qualifications, annotation instructions, agreement rates, or counts of corrected labels are reported. Because every model is evaluated against these labels, the reported performances may partly reflect agreement with GPT-4o's labeling style rather than HSE competence. The authors should report inter-annotator agreement, the number of questions changed during review, and ideally provide a human-verified gold subset that is independent of the LLM generation process.","section":"Section 3.3 and Table 1"},{"comment":"The fieldwork-video questions appear to be constructed from GPT-4o-generated textual descriptions of videos, and the setup states that for video LLMs 'we follow the existing method... to extract frames from the video and use GPT-4o to generate textual descriptions.' If this means that both vision-language models and text-only models are evaluated on the same text, then the vision-model results do not test visual input at all. Please state explicitly whether vision models received frames/videos; if they did not, the 'vision models' category should be removed or re-scoped to text-only evaluation.","section":"Section 3.1 and Section 4.1 (Experimental Setup)"},{"comment":"The prompt templates are internally inconsistent: the CoT and 0-CoT templates include 'Choose the best answer. Reply only with the letter A, B, C, or D. **Do NOT explain your choice...**' immediately before 'Let's think step by step:'. If the 'do not explain' instruction was actually sent, the CoT conditions may not have elicited the intended reasoning, which would directly affect the comparison with RoE and the conclusion that CoT does not help. In addition, the abstract's 'up to 25.01%' improvement for RoE over CoT on DeepSeek-R1 cannot be traced in the reported text, which gives average gains of about 5%; the full per-cell numerical results behind Figure 3 should be provided.","section":"Section B.1 and Section 4.2.2"},{"comment":"The comparative claims (e.g., reasoning models do not outperform foundation models; the drop is larger for reasoning models; later IRAC phases are harder) are made without confidence intervals or repeated runs. The only stochastic element discussed is CoT+SC, which is run five times, and no variance is reported. Because LLM outputs are random, the authors should report at least three runs with standard deviations for the headline comparisons or use significance tests, and include the underlying numerical tables rather than only figures.","section":"Section 4.2.1 and 4.2.2"}],"minor_comments":[{"comment":"The heading 'Benchmark deatils' contains a typo; it should read 'Benchmark details'.","section":"Appendix C heading"},{"comment":"Figure 2(a-d) labels the y-axis 'Accuracy' but extends it to 1.4, and some AUC-ROC axes in Figure 5 extend above 1.0; since neither metric can exceed 1.0, the axes should be rescaled or the plotted values should be rechecked.","section":"Figure 2(a-d) and Figure 5"},{"comment":"The sentence 'AUC-ROC is typically higher than or equal to accuracy' is used to argue that the observed drop is even greater, but that property holds only when the same scoring function underlies both metrics; since the paper's AUC-ROC uses BERT cosine similarity rather than model confidence, the argument is not valid as stated.","section":"Section 4.2.1"},{"comment":"The paper claims to be the 'first comprehensive HSE compliance benchmark' but does not position itself against LabSafety Bench and AIR-Bench in enough detail; a short comparison of task scope and data sources would make the novelty claim clearer.","section":"Section 2 (Related Works)"},{"comment":"The correct options in Table 3 are marked in bold, but the caption does not state this; please add a note explaining the bold formatting.","section":"Table 3 in Appendix C"},{"comment":"The few-shot prompt template says 'k shot of different...', while the text states that 5 examples are used; please make the placeholder and the actual number consistent.","section":"Section B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a benchmark resource and an evaluation, and the resource itself is potentially useful. The semantic-matching claim is likely over-interpreted given the metric mismatch, and the benchmark's label-generation process needs more transparency. If the authors add validation of the free-form scoring, repeated runs or confidence intervals, annotation statistics, and a clear description of what the vision models actually received, the paper could be publishable as a benchmark/evaluation study. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: HSE-Bench is a genuinely useful new resource, but the paper's headline finding about semantic matching is built on a measurement comparison that hasn't been validated. The benchmark is worth taking seriously; the claim is not yet established.\n\nThe new thing here is the dataset: 1,020 HSE questions drawn from regulations, court cases, safety exams, and video-derived text, structured along IRAC stages. That fills a real gap. The construction pipeline (GPT-4o generation plus adversarial augmentation plus two-expert review) is reasonable, and the paper ships code and data, which makes it reproducible. The evaluation across 12 models and 6 prompting strategies is thorough, and the observation that reasoning models don't beat foundation models on this benchmark is worth discussing.\n\nWhere it goes soft: the central claim that LLMs rely on semantic matching rather than principled reasoning is inferred from the gap between accuracy (options provided, direct selection) and AUC-ROC (options omitted, BERT cosine similarity ranking). That gap conflates two changes: input format and scoring mechanism. BERT similarity is a noisy proxy; the paper doesn't validate it against human judgment or a second embedding model. The case studies show different reasoning traces, but they don't show that the gap is caused by semantic matching. So the headline finding is a plausible hypothesis, not a demonstrated result. I'd want the authors to either validate the proxy or soften the claim.\n\nAlso, the labels were generated by GPT-4o. The two-expert check is good, but the paper doesn't report agreement rates or how many labels were changed. That's a moderate concern, not fatal. The 'up to 25.01%' improvement for RoE is a single best cell; the average gains are single-digit percentages. That's normal but should be presented with the distribution.\n\nThe paper's own limitations section is honest but thin. For a safety-critical domain, the lack of error bars and the single-run nature of most experiments is a real weakness, though not disqualifying.\n\nBottom line: this is a resource paper with an over-claimed interpretation. I'd send it to peer review with the expectation of major revision. The dataset alone justifies the referee time, and the measurement issue is fixable with additional validation.","headline":"HSE-Bench is a genuinely useful resource, but the semantic-matching claim rests on an unvalidated metric comparison.","tokens_in":23213,"tokens_out":2558,"would_cite":true,"duration_ms":23662,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models appear competent on HSE compliance questions by matching words between scenario and options, not by reasoning from the regulations, and a new multi-expert prompting technique produces more accurate decisions.","keywords":["hse compliance assessment","llm evaluation","benchmark dataset","irac framework","semantic matching","legal reasoning","prompt engineering","reasoning of experts"],"falsifier":"Run the same 1,020 questions under a single metric: ask each model to answer without options and then grade those free-form answers the way the accuracy protocol grades choices, or, conversely, present options but grade by whether the model's reasoning names the correct regulation before picking an option. If a model's option-free accuracy matched its optioned accuracy on questions whose distractors share no surface vocabulary with the scenario, the semantic-matching explanation would collapse. A concrete check is available in the released data: compute the accuracy–AUC-ROC gap separately for distractors with high versus low lexical overlap with the scenario; the gap should largely vanish for low-overlap distractors under the paper's own account.","tokens_in":22305,"feed_emoji":"⚖️","tokens_out":9937,"duration_ms":86077,"temperature":0.7,"pith_summary":"This paper tries to establish what large language models actually do when they appear to assess health, safety, and environment (HSE) compliance, and the answer is that they mostly match words, not rules. To show this it builds HSE-Bench, over 1,000 curated questions from regulations, court cases, safety exams, and fieldwork videos, organized by the four-step IRAC legal-reasoning pipeline. Across twelve models, accuracy with answer options visible averages about 90 percent, but when options are withheld the same models' free-form answers score about 25 percentage points lower, which the authors read as evidence that the apparent competence rests on semantic matching rather than principled reasoning grounded in the compliance context. The paper's remedy is a prompting strategy, Reasoning of Expert (RoE), that makes the model deliberate as several domain experts before deciding, and it reports accuracy gains up to 25.01 percent over chain-of-thought on DeepSeek-R1. If these findings hold, organizations should treat LLM compliance advice as unverified until a human checks the reasoning.","feed_headline":"LLMs ace HSE compliance quizzes by word matching, not legal reasoning","feed_subtitle":"New 1,020-question benchmark shows why AI safety-compliance advice still needs human oversight.","key_machinery":"The machinery that carries the argument is a pair of matched evaluation protocols whose score gap is treated as evidence of how models think. In the accuracy protocol the model sees the scenario, question, and all four options and must choose one, which the paper argues rewards surface-level semantic matching. In the AUC-ROC protocol only the scenario and question are shown; the model's free-form answer is embedded with BERT and compared by cosine similarity against each option's embedding, so the model must effectively produce the correct content on its own. The benchmark itself is organized by the IRAC framework (Issue spotting, rule Recall, rule Application, rule Conclusion), a four-step legal-reasoning schema borrowed from legal education, which lets the authors localize failures to specific reasoning stages. The proposed improvement, Reasoning of Expert (RoE), is a prompt that instructs the model to assume several domain-expert roles, produce an independent IRAC analysis for each, and synthesize the opinions into a final judgment that balances regulatory compliance with practical safety.","core_discovery":"The paper's central claim is that current large language models give a misleading impression of HSE compliance competence. Given a scenario and four answer options they select the correct option at roughly 90 percent accuracy, but the paper argues this success is driven by semantic matching between the scenario and the options rather than by principled reasoning grounded in the underlying HSE compliance context. The evidence is the consistent gap between the two evaluation protocols: when only the scenario and question are presented and the model's free-form answer is scored by how strongly it resembles each option, average performance falls by 24.73 percent across all models and data sources. Reasoning-trace case studies show models engaged in option-by-option elimination rather than structured legal analysis, and the paper claims their native reasoning lacks the systematic legal pipeline—issue spotting, rule recall, rule application, rule conclusion—that rigorous HSE assessment requires. The paper further claims that its Reasoning of Expert (RoE) prompt, which has the model produce independent IRAC-based opinions from multiple expert roles and then synthesize a unified decision, substantially improves accuracy over chain-of-thought prompting.","pith_inferences":["The accuracy–AUC-ROC gap would be a stronger argument if the two protocols differed only in whether options are shown; as designed they also differ in scoring method (exact choice versus BERT cosine similarity) and in output format, so some of the 24.73 percent gap may be measurement artifact rather than evidence of semantic matching.","A natural deployment implication the authors do not spell out is that option-free, generation-style probes are the more honest test of an LLM's HSE competence, so organizations could re-run the HSE-Bench questions in open-ended form as a spot audit before trusting model output.","RoE's expert-role deliberation resembles decision processes in other regulated fields, so the prompt is a plausible candidate for testing in financial compliance, medical safety, or privacy-law assessment tasks.","The benchmark could be extended to measure grounding directly by scoring whether a model's answer cites a real rule identifier; that would convert the semantic-matching claim from an interpretation of a score gap into a directly observable behavior."],"forward_implications":["Human oversight remains essential if LLMs are used in HSE compliance assessment, since high multiple-choice accuracy does not indicate command of the underlying regulations.","Chain-of-thought prompting does not close the HSE reasoning gap; in the reported experiments CoT-based prompts often failed to beat zero-shot prompting on both accuracy and AUC-ROC.","Reasoning-oriented models did not outperform foundation models on HSE-Bench, and their accuracy-to-AUC-ROC drop was larger, so general reasoning strength does not transfer automatically to regulatory compliance reasoning.","The Reasoning of Expert (RoE) prompt improved both models tested, with gains up to 25.01 percent accuracy over CoT on DeepSeek-R1 and up to 8.03 percent AUC-ROC, with larger relative gains on the reasoning model.","Rule recall is the weakest stage of the IRAC pipeline across models, with an average AUC-ROC of 60.75 percent, indicating that LLMs struggle to name the specific regulation that governs a spotted issue."],"supporting_citations":[{"why":"Supplies the IRAC-based task decomposition and the legal-benchmark methodology that HSE-Bench adapts to the HSE domain.","marker":"[19]"},{"why":"The prior HSE-focused LLM exploration whose limited data sources and missing integrated reasoning HSE-Bench is designed to overcome.","marker":"[3]"},{"why":"The chain-of-thought prompting baseline against which RoE's accuracy gains (up to 25.01 percent) are measured.","marker":"[43]"},{"why":"Provides the BERT encoder used to embed free-form answers and options for the AUC-ROC cosine-similarity scoring.","marker":"[11]"},{"why":"Source of the IRAC legal-reasoning formulation (issue, rule, application, conclusion) that structures the benchmark's four phases.","marker":"[36]"},{"why":"Supplies the self-consistency prompting baseline evaluated alongside CoT and RoE.","marker":"[42]"},{"why":"Supplies the few-shot prompting baseline used in the prompt-strategy comparisons.","marker":"[6]"},{"why":"Supplies the zero-shot chain-of-thought baseline (0-CoT) used in the prompt-strategy comparisons.","marker":"[41]"},{"why":"The multimodal method for extracting frames from fieldwork videos and generating textual descriptions for the video-based questions.","marker":"[25]"}],"fun_headline_variants":["Word matching, not legal reasoning, drives LLM HSE scores","HSE-Bench exposes LLM compliance as semantic matching, not law","LLMs score high on HSE by pattern matching, not legal analysis","LLMs ace HSE quiz via word cues but lack legal reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline finding depends on the assumption that the accuracy score (options supplied) and the AUC-ROC score (options withheld, measured by BERT cosine similarity) are comparable measures of the same ability, so that the gap between them reflects semantic matching rather than a difference in task difficulty or a weakness of the similarity-based scoring proxy.","fun_headline_variants_meta":{"raw":{"variants":["Word matching, not legal reasoning, drives LLM HSE scores","HSE-Bench exposes LLM compliance as semantic matching, not law","LLMs score high on HSE by pattern matching, not legal analysis","LLMs ace HSE quiz via word cues but lack legal reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001092,"raw_usage":{"total_tokens":4601,"prompt_tokens":1025,"completion_tokens":3576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":3499}},"tokens_in":641,"tokens_out":3576,"duration_ms":26510,"temperature":1.0,"reasoning_tokens":3499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:56:11.651359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,020 questions under a single metric: ask each model to answer without options and then grade those free-form answers the way the accuracy protocol grades choices, or, conversely, present options but grade by whether the model's reasoning names the correct regulation before picking an option. If a model's option-free accuracy matched its optioned accuracy on questions whose distractors share no surface vocabulary with the scenario, the semantic-matching explanation would collapse. A concrete check is available in the released data: compute the accuracy–AUC-ROC gap separately for distractors with high versus low lexical overlap with the scenario; the gap should largely vanish for low-overlap distractors under the paper's own account.","supporting_citations":[{"cited_title":"Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the IRAC-based task decomposition and the legal-benchmark methodology that HSE-Bench adapts to the HSE domain."},{"cited_title":"Andreopoulos","cited_arxiv_id":null,"evidence_quote":"The prior HSE-focused LLM exploration whose limited data sources and missing integrated reasoning HSE-Bench is designed to overcome."},{"cited_title":"Legal reasoning.It’s all about IRAC, 2021","cited_arxiv_id":null,"evidence_quote":"Source of the IRAC legal-reasoning formulation (issue, rule, application, conclusion) that structures the benchmark's four phases."}],"review_version":1}