{"id":"62062cc7-5cbf-48bc-873d-47265f8658e7","arxiv_id":"2505.19165","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"OrgAccess, a 70k-query synthetic RBAC benchmark, shows current LLMs including GPT-4.1 (F1 0.27 on the hardest split) struggle badly with permission adherence.","lead":"This paper introduces OrgAccess, a synthetic benchmark of 70,000 role-based access control queries that tests whether LLMs can grant, restrict, or deny access under one, three, or five concurrent permissions. Even GPT-4.1 scores only 0.27 F1 on the hardest split, suggesting a major gap in enterprise-grade rule following.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth label correctness is the load-bearing assumption; the 'Temporary Contractor Access' seed in Appendix A.4 contradicts itself (Expected 'partial', rationale 'must be rejected'), so low F1 scores may reflect label noise rather than an RBAC reasoning deficit.","rationale":"The reader's verdict is CONDITIONAL, and our stress-test lands on the same load-bearing point: the benchmark's ground-truth labels are assumed correct. The internal contradiction in the published 'Temporary Contractor Access' seed (Expected Response 'partial' vs rationale 'must be rejected') is concrete evidence that the annotation process is fallible. This matters because the entire empirical argument—that even GPT-4.1 fails at RBAC reasoning—is measured against those labels. The paper's QA (Section 2.4) is real but limited: automated checks cover only verifiable permissions, and manual rationale correction touched 750 hard and 84 medium examples out of 70,000. The authors do not report a full audit of expected-response labels, and the absence of human or random baselines leaves no way to calibrate label noise. The open-sourcing of the dataset and code is a genuine strength, and the expert-driven permission curation is credible. But a low F1 on a noisy benchmark is not yet evidence of a reasoning deficit; it may reflect disagreement with erroneous ground truth. We therefore do not change the reader's CONDITIONAL verdict: the central claim warrants acceptance only after an independent human re-annotation of a representative sample confirms that the hard-split labels are largely correct. Our proposed test is the natural next step and would settle the concern directly.","tokens_in":37560,"tokens_out":3485,"duration_ms":31128,"concrete_test":"Independently re-annotate a random sample of 300 hard-split examples (stratified by permission category) with at least two expert annotators blinded to the dataset's expected responses, and compute per-example agreement with the published labels. If agreement is below 90%, or if contradictions of the kind seen in the 'Temporary Contractor Access' seed recur in more than 5% of the sample, the benchmark's ground truth is too noisy to support the central low-F1 claim; re-run GPT-4.1 on the clean subset and compare F1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that current LLMs cannot strictly adhere to organizational RBAC policies, with GPT-4.1 at F1 0.27 on the hard split—assumes the benchmark's expected responses ('full', 'partial', 'rejected') are correct and unambiguous. This assumption is not established. In Appendix A.4, the published 'Temporary Contractor Access' seed lists Expected Response: 'partial', but its own rationale concludes 'Therefore, the request must be rejected.' This is a direct internal contradiction in a hand-authored seed (Section 2.2 says 100 high-quality seed data points per permission were peer-reviewed). If seeds can be inconsistent, the synthetic 70,000 examples generated from them (Section 2.3) and the QA pass in Section 2.4—which corrected only 750 hard-split and 84 medium-split rationales, without a stated full-label audit—cannot guarantee ground-truth correctness. Consequently, the reported F1 values, including the flagship 0.27, may be depressed by mislabeled or ambiguous queries rather than by a genuine LLM reasoning failure. The absence of human or random baselines makes this ambiguity worse: without an estimate of label noise, a below-chance score cannot be interpreted as evidence of a capability gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OrgAccess, a synthetic benchmark for evaluating LLMs on role-based access control (RBAC) reasoning. It defines 40 expert-validated permission types and creates three splits (easy, medium, hard) with 40,000, 10,000, and 20,000 query–response pairs, respectively, using Mistral Small 3.1 for synthetic generation with guided sampling and post-hoc QA. The authors evaluate 16 (or, in one sentence, 35) LLMs of varying sizes and report that even GPT-4.1 achieves only an F1-score of 0.27 on the hard split, concluding that current LLMs are not capable of strictly adhering to organizational access policies. The error analysis attributes failures to categories such as constraint, scope, conflict, and false-partial errors.","tokens_in":37912,"tokens_out":3668,"duration_ms":34449,"significance":"If the benchmark's ground-truth labels are correct, OrgAccess addresses a real and underexplored evaluation gap: compositional rule-following and conflict resolution in enterprise RBAC settings, which standard factual and STEM benchmarks do not cover. The dataset is large, open-sourced, and grounded in NIST frameworks with expert review, and the three-tier difficulty design is a useful contribution. The paper also provides a detailed error taxonomy and per-category performance breakdown, which are valuable for future work. However, the significance hinges on the reliability of the synthetic labels; if label noise is substantial, the reported capability gap is not cleanly interpretable.","major_comments":[{"comment":"The 'Temporary Contractor Access' seed in Appendix A.4 lists Expected Response: 'partial', but its own rationale concludes 'Therefore, the request must be rejected.' This is a direct internal contradiction in a hand-authored seed that Section 2.2 states was rigorously peer-reviewed. Because Section 2.3 uses these seeds as anchors for generating 70,000 synthetic examples, and Section 2.4 reports only a partial QA pass (750 hard-split and 84 medium-split rationales corrected, with no full label audit), the contradiction undermines confidence in the ground-truth labels. The authors should correct the seed, audit the affected data, and provide either a formal decision procedure for the full/partial/rejected distinction or an estimate of label noise (e.g., inter-annotator agreement). Without this, the reported F1 values, including the headline GPT-4.1 score of 0.27, may reflect label ambiguity rather than a reasoning deficit.","section":"Appendix A.4 and Section 2.2"},{"comment":"The final paragraph of Section 3 states 'The evaluation of 35 state-of-the-art LLMs', which conflicts with the '16 LLMs' mentioned in the Introduction and the first sentence of Section 3, and with the 10 models in Table 1 and 7 in Table 2. Please reconcile the model count and provide a complete list of all evaluated models (e.g., in an appendix) so that the reported results are reproducible.","section":"Section 3"},{"comment":"Mistral Small 3.1 was used to generate the synthetic queries and expected responses, and the same model (as Mistral-Small-3.1-24B) is also evaluated in Table 1. This introduces a circularity: the model may be advantaged or disadvantaged by having generated the data it is tested on. The authors should explicitly discuss this risk and, ideally, report results with this model excluded or with an independent verification of a sample of labels by a different model or human annotators.","section":"Sections 2.2/2.3 and Table 1"},{"comment":"No random or majority-class baseline is reported. The hard-split F1 scores (e.g., 0.27 for GPT-4.1) are actually below the random-chance F1 of approximately 0.33 for a balanced three-class problem, which is highly unusual and suggests that either the task is mis-calibrated or the labels contain substantial noise. Reporting random and majority baselines, and ideally human performance on a sample, is necessary to interpret the 'capability gap' claim and to confirm that the low scores are not an artifact of the benchmark design.","section":"Section 3 and Tables 1/2"}],"minor_comments":[{"comment":"The word 'Erros' should be 'Errors'.","section":"Table 3 caption"},{"comment":"In the paragraph beginning 'The observed performance plateau', 'adddress' should be 'address'.","section":"Section 3"},{"comment":"The text references 'as reported in Section 5', but the paper has only four numbered sections; this should be Section 3.","section":"Appendix A.2"},{"comment":"There is an unfinished placeholder, 'Section [Refer to the Section where you introduced the 7 categories]', which should be replaced with the actual section reference.","section":"Appendix A.3"},{"comment":"The permission catalog inconsistently uses 'OrgAccess' and 'ORG-Benchmark'; unify the terminology.","section":"Appendix A.4"},{"comment":"There are formatting artifacts such as raw '\\textbf' commands and garbled quotes (e.g., '˘20...˘2019') in several seeds; these should be cleaned up.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The internal contradiction in the published seed and the unexplained '35 LLMs' count suggest that the manuscript may have been revised hastily. The dataset and code are open-sourced, which is commendable, but the label-reliability issue is central to the paper's interpretation and should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is a genuine contribution. RBAC evaluation for LLMs is a real blind spot, and OrgAccess gives us 40 permission types, three difficulty splits, and a plausible error taxonomy, all open-sourced. The empirical pattern—sharp degradation from easy to hard, with a distinctive false-partial failure mode—is consistent across models and is worth taking seriously. The grounding in NIST frameworks and the expert Delphi process for permissions is a real strength. This is not a paradigm shift, but it is a useful measurement tool for enterprise LLM readiness.\n\nThe soft spots are real but not uniformly fatal. The clearest problem is the seed example in Appendix A.4: Temporary Contractor Access lists Expected Response as 'partial', yet its own rationale concludes the request 'must be rejected.' That is a direct internal contradiction in a hand-authored, peer-reviewed seed. If the seeds are noisy, the 70,000 generated labels inherit that noise, and the QA pass that corrected only 750 hard and 84 medium rationales does not constitute a full label audit. The absence of human and random baselines is the second issue: without an estimate of label noise, a below-chance F1 of 0.27 on the hard split cannot be cleanly interpreted as a reasoning deficit. The '35 LLMs' phrasing alongside Table 1's 16 models is sloppy, and the generator model appearing in the evaluated set is a mild circularity.\n\nNone of this sinks the central claim, but it does mean the headline numbers are provisional. The paper deserves a serious referee because it opens a new evaluation dimension and ships reproducible artifacts. My recommendation: accept for peer review with major revisions. Require an audit of label quality with inter-annotator agreement, add human and random baselines, fix the seed contradiction, and reconcile the model count.","headline":"OrgAccess fills a real evaluation gap, but a self-contradictory seed example and missing human baselines mean the headline F1 numbers need a label audit before they fully carry weight.","tokens_in":38349,"tokens_out":1904,"would_cite":true,"duration_ms":22889,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current LLMs cannot reliably enforce role-based access control in organizational settings.","keywords":["role-based access control","LLM benchmark","organizational hierarchy","permission reasoning","rule following","compositional reasoning","synthetic dataset","enterprise AI"],"falsifier":"One concrete check is to take a random sample of hard-split items, have independent security professionals label them without seeing the generated labels, and measure agreement. If agreement is low, or if a published seed whose rationale contradicts its expected label is not an isolated typo, the F1 gap between models and labels is not a clean measure of RBAC reasoning.","tokens_in":37420,"feed_emoji":"🔐","tokens_out":3791,"duration_ms":33534,"temperature":0.7,"pith_summary":"The paper argues that current large language models cannot reliably enforce role-based access control policies of the kind enterprises would depend on. To test this, it builds OrgAccess, a synthetic benchmark of 70,000 query-response pairs built from 40 expert-validated permission types arranged in easy, medium, and hard splits. Evaluating models across sizes and providers, the authors find consistent collapse as concurrent permissions grow: GPT-4.1, the strongest model tested, reaches only 0.27 F1 on the hard split. The central claim is that this reveals a fundamental deficit in compositional rule following, not a gap that model scaling closes.","feed_headline":"New benchmark shows GPT-4.1 scores 0.27 F1 on enterprise access rules","feed_subtitle":"70,000 permission queries expose a steep, scaling-resistant drop in LLM rule following.","key_machinery":"The load-bearing object is the OrgAccess dataset itself: 40 permission types grounded in NIST control families and refined through a structured multi-round Delphi expert process, then assembled into three difficulty splits. Easy items carry one permission, medium items carry three, and hard items carry five, with combinations chosen by experts to be realistic and conflict-prone. Each item is a user query plus a target decision among \"full\", \"partial\", and \"rejected\" with a rationale. The benchmark isolates rule-following by requiring the model to return the discrete decision from explicit permissions, and the design separates a baseline single-rule skill from the compositional skill of resolving concurrent permissions.","core_discovery":"The central discovery is that LLMs are not inherently capable of strictly adhering to organizational access policies once several permissions must be considered together. On the benchmark, accuracy falls sharply from single-permission tasks to tasks with five concurrent permissions, and the same steep drop appears across model families. The failure is not random: models over-predict \"partial\" access, overlook individual permissions within a combination, and mishandle scope and conflict reasoning. Even a state-of-the-art model like GPT-4.1 earns only 0.27 F1 on the hardest split, which the authors take as evidence that the ability to compose and apply structured rules is largely missing from current systems.","pith_inferences":["A natural follow-up experiment is to hold the query fixed and vary only the permission tuple, which would isolate whether failures come from rule comprehension or from composing multiple rules.","The same design could be extended to temporal and stateful policies, such as expiring grants or session state, to test whether models track access over a dialogue rather than in a single turn.","If label noise is present, as the published \"Temporary Contractor Access\" seed suggests, even the reported F1 numbers could be pessimistic; re-scoring with a clean, adjudicated label set would sharpen the benchmark's conclusions.","The three-way decision format likely understates real enterprise needs, where the correct action is often an explanation or a partial fulfillment with redaction; a benchmark that scores those outputs would measure deployability more directly."],"forward_implications":["Performance collapses consistently from easy to medium to hard across all tested models, so the difficulty gradient itself is a robust measure of compositional rule-following.","Scaling model size within current architectures does not close the gap: larger models improve on easy and medium splits, but hard-split performance stays near or below 0.3 F1.","The dominant \"false partial\" error means models tend to hedge toward partial access instead of committing to full or rejected, undermining their reliability as access gateways.","Error analysis shows scope, constraint, and conflict errors all rise sharply from easy to hard, pointing to a reasoning deficit rather than a surface formatting issue.","The benchmark can serve as a diagnostic tool for hierarchy-aware LLM development and for tracking progress beyond factual and STEM benchmarks."],"supporting_citations":[{"why":"Defines role-based access control, the conceptual model the benchmark operationalizes.","marker":"Sandhu [1998]"},{"why":"Supplies the SP 800-53 control families from which the 40 permissions are derived.","marker":"NIST [2020]"},{"why":"Provides the NIST Cybersecurity Framework control groups used to structure the permission categories.","marker":"CSF [2018]"},{"why":"Provides the Delphi methodology used for expert validation of the permission set and combinations.","marker":"Rashid et al. [2020]"},{"why":"Few-shot prompting is the mechanism used to steer guided synthetic data generation.","marker":"Brown et al. [2020]"},{"why":"Mistral Small 3.1 is the model that generates the 70,000 synthetic data points.","marker":"Mistral [2025]"},{"why":"Best practices for synthetic data inform the post-processing and quality assurance pipeline.","marker":"Liu et al. [2024]"}],"fun_headline_variants":["GPT-4.1 scores 0.27 F1 on hardest OrgAccess permission tasks","LLMs fail organizational access: GPT-4.1 hits 0.27 F1 on 5-permission tasks","OrgAccess: 70k queries show GPT-4.1 gets only 0.27 F1 on hardest RBAC","Complex RBAC trips LLMs: GPT-4.1 scores 0.27 F1 on new OrgAccess"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement rests on the ground-truth labels being correct and unambiguous, so a low model score reflects a reasoning deficit rather than noisy or contradictory answers in the benchmark itself.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4.1 scores 0.27 F1 on hardest OrgAccess permission tasks","LLMs fail organizational access: GPT-4.1 hits 0.27 F1 on 5-permission tasks","OrgAccess: 70k queries show GPT-4.1 gets only 0.27 F1 on hardest RBAC","Complex RBAC trips LLMs: GPT-4.1 scores 0.27 F1 on new OrgAccess"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00061,"raw_usage":{"total_tokens":2853,"prompt_tokens":976,"completion_tokens":1877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1764}},"tokens_in":592,"tokens_out":1877,"duration_ms":7691,"temperature":1.0,"reasoning_tokens":1764,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:18:50.379474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check is to take a random sample of hard-split items, have independent security professionals label them without seeing the generated labels, and measure agreement. If agreement is low, or if a published seed whose rationale contradicts its expected label is not an isolated typo, the F1 gap between models and labels is not a clean measure of RBAC reasoning.","supporting_citations":[],"review_version":1}