{"id":"7cbf7881-bd91-400b-8a51-27032102c697","arxiv_id":"2505.18542","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pseudo-code grounding improves LLM extraction of business rule flows, and reasoning-optimized models handle nonlinear dependencies best, on the new BREX benchmark.","lead":"The paper builds BREX, a benchmark of 409 business documents and 2,855 expert-labeled rules, and tests five prompt designs across 13 LLMs for turning regulations into structured rule flows. It finds that asking models to first write pseudo-code improves rule extraction, and that reasoning-tuned models trace branching and parallel dependencies better.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage I headline claim is not supported by the reported tables: P5's average NER F1 gain over P1/P2 is 0.003, with no significance tests and several per-model reversals.","rationale":"The reader's conditional verdict is appropriate: BREX is a useful benchmark and the ExIde framework is clearly described, but the headline empirical claims are overstated. My read identifies a slightly different load-bearing concern than the reader's weakest_assumption. The reader focuses on the unvalidated dependency-graph labels, which matters for Stage II and the 'reasoning gap' conclusion. However, the Stage I claim that executable grounding significantly outperforms standard prompts is already unsupported by the reported tables themselves, independently of label noise: the aggregate differences are 0.003 NER F1 and 0.004-0.010 logical-operator F1, with no uncertainty or significance statistics, and several models show the reverse ordering. This means even if every label were perfect, the paper would still need to show the difference is not noise. The dependency-label concern is valid and should be addressed by a direct edge-level IAA study, but the statistical-support issue is more directly load-bearing for the abstract's primary claim. I therefore keep the conditional verdict; the required revision is to soften the claims and add paired significance or uncertainty quantification for both Stage I and Stage II comparisons.","tokens_in":16795,"tokens_out":6167,"duration_ms":54786,"concrete_test":"Re-analyze the existing model outputs at the document level: for each of the 13 models, compute per-document NER F1 and logical-operator F1, then run a paired bootstrap (10,000 resamples) or a McNemar-type token-error test comparing P5 against P1 and P5 against P2. Report the 95% confidence interval for the mean difference and the per-model win rate. If no confidence interval excludes zero, or if fewer than a clear majority of the 13 models favor P5, then the 'significantly outperforming' claim is unsupported and should be downgraded to 'marginally higher aggregate score with high variance and non-uniform model-level benefit.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central Stage I claim—that executable grounding (P5) 'significantly outperforms standard prompts'—is not established by the paper's own reported numbers. In Table 2, P5's average NER F1 is 0.882 versus 0.879 for both P1 and P2, a 0.003-point gap. In Table 3, the logical-operator gap is 0.004 over P2 and 0.010 over P1. No confidence intervals, standard deviations, or paired significance tests are reported, so 'significantly' is unsupported. More directly, P5 is not consistently the best prompt per model: e.g., Gemini-2.5-pro P1 0.916 > P5 0.899; GPT-5 P1 0.876 > P5 0.873; Qwen3-235B-Instruct P1 0.898 > P5 0.892; and in logical-operator F1, DeepSeek-3.2-exp P2 0.896 > P5 0.848. The robustness analysis in Figure 4 also lacks error bars and statistical tests. Because the paper's central contribution is framed around the superiority of executable grounding, this is the load-bearing weakness: if the small aggregate differences do not survive a paired or bootstrap test, the headline conclusion reduces to a minor average-rank observation. The dependency-label validity concern raised by the reader is real, but for the Stage I claim the missing statistical support is more directly fatal to the abstract's wording.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BREX, a benchmark of 409 business documents and 2,855 expert-annotated condition–action rules with dependency relations, and ExIde, a five-prompt structure-aware framework for extracting such rules and reconstructing dependency graphs. The authors evaluate 13 LLMs and report two main findings: pseudo-code-based executable grounding (P5) outperforms other prompts in rule extraction, and reasoning-optimized models outperform instruction-tuned models on dependency identification. The dataset and code are released.","tokens_in":17112,"tokens_out":5111,"duration_ms":38885,"significance":"If the findings are established, BREX would be a useful resource for rule-flow extraction, and pseudo-code prompting would be a practical tuning-free intervention. The manuscript's strengths include the multi-domain expert-annotated dataset (over 30 domains), the clear decomposition into Stage I and Stage II, and the breadth of models and prompting variants. However, the headline claims rest on small average differences without statistical support, and the reliability of the dependency labels—central to the Stage II conclusion—is not directly measured. These issues must be addressed before the conclusions are fully supported.","major_comments":[{"comment":"The abstract and §5.2 claim that P5 'significantly outperforms' standard prompts, but the reported numbers do not demonstrate this. The average NER F1 difference between P5 (0.882) and P1/P2 (0.879) is 0.003, and the logical-operator F1 advantage over P2 is 0.004; no confidence intervals, error bars, or paired significance tests are reported. Moreover, P5 is not the best prompt for several models (e.g., Gemini-2.5-pro P1 0.916 vs. P5 0.899; GPT-5 P1 0.876 vs. P5 0.873; Qwen3-235B-Instruct P1 0.898 vs. P5 0.892; DeepSeek-3.2-exp logical-operator P2 0.896 vs. P5 0.848). A paired bootstrap or signed-rank test is needed to support the wording; if the differences are not significant, the conclusion should be tempered to a small average-rank trend.","section":"§5.2, Tables 2–3"},{"comment":"The reported Fleiss Kappa of 0.901 is computed on a NER projection of Slot Type, Reference Value, and Action spans, not on the Sequential/Conditional/Parallel dependency edges that are central to Stage II. The paper itself acknowledges that there is no established metric for graph-structured agreement. Without evidence about expert agreement on the edge labels, the Stage II 'reasoning gap' results may be built on labels whose reliability is unknown. The authors should report agreement on the dependency relations (or a defensible proxy) before drawing strong conclusions from Table 4.","section":"§3.3, A.6"},{"comment":"It is unclear whether the Stage II dependency evaluation uses gold rule pairs or pairs extracted from Stage I outputs. The text says the prompt 'takes rule pairs as input' but does not specify the source of these pairs, and Table 4 reports only F1. If gold pairs are used, the results isolate pairwise classification and should be described as such; if predicted pairs are used, error propagation from Stage I should be discussed. This distinction materially affects the interpretation of the 'reasoning advantage' claim.","section":"§4.2, §5.3"},{"comment":"The robustness analysis claims that P5 shows a 'relatively flat performance curve' and that reasoning-optimized models have 'significantly flatter degradation curves,' but Figure 4 shows aggregate lines without error bars or statistical tests. The claims of significance and the bucket-specific reversal for P5 should be supported by per-model inference or at least by variance estimates.","section":"§5.4, Figure 4"}],"minor_comments":[{"comment":"The title 'Business asRulesual' appears to be a pun; consider spelling out 'Business Rules' in the title for discoverability.","section":"Title and full text"},{"comment":"The reference list contains incomplete entries, e.g., the Gemini 2.5 entry is listed with 'and 1 others.' Please fix.","section":"References"},{"comment":"Table 5 shows 2,855 rules but 3,469 dependency relations; the text could clarify that a rule can participate in multiple dependency edges.","section":"Table 5"},{"comment":"The claim that 'P4 performs worst on average' is based on Table 3, but Table 2 shows P4 is not the worst for NER (P3 is). Please make the statement consistent with both tables.","section":"§5.2"},{"comment":"The IAA example shows annotator disagreement on span boundaries, but the text says the high Kappa confirms consistent interpretation; consider discussing how boundary variations affect downstream evaluation.","section":"A.6, Table 7"},{"comment":"The radar plot would benefit from axis labels and a legend; currently the color distinction between 'Standard Instruction-tuned' and 'Reasoning-optimized' is not defined in the caption.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful dataset and a systematic prompting study, but the headline 'significantly outperforming' claim is not supported by the reported numbers. I would encourage the editor to request a revision that adds statistical tests or softens the claims, and that clarifies the dependency-label reliability. The fit with the journal is otherwise appropriate for an NLP/CL applied benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Couple of things worth knowing about this one. The BREX benchmark is genuinely new: 409 business and regulatory documents, 2,855 expert-annotated condition-action rules with Sequential/Conditional/Parallel dependencies, spanning 30+ domains. That is a useful testbed for logic-aware extraction, more interesting than the action-centric recipe datasets. The annotation schema is clearly described, and the dataset statistics look sane.\n\nThe ExIde prompting study is also competently done as a systematic comparison: five prompts, 13 models, two stages. The case study in the appendix on P4's over-merging of enumerated values under 'equal' is a concrete, useful failure analysis. The framework itself is not a big conceptual leap—P5 is pseudo-code as an intermediate representation, which is a known trick—but the comparison is still useful.\n\nWhere the paper goes soft is the headline claim. The abstract says executable grounding (P5) 'significantly outperforms standard prompts in rule extraction.' Look at Table 2: P5's average NER F1 is 0.882 versus 0.879 for P1 and P2. A 0.003 gap, with no confidence intervals, no paired tests, and several per-model reversals. On logical-operator F1 the gap is 0.004–0.010, again with reversals. That does not support 'significantly' in any statistical sense. The stress-test note has the numbers and they are accurate. This is a load-bearing weakness because the paper's main empirical selling point is the superiority of executable grounding. The fix is easy: report per-model differences with bootstrap CIs or a paired test, or soften the claim to 'P5 is competitive and often best on average.'\n\nThe other soft spot is the dependency-label reliability. The Fleiss Kappa of 0.901 is computed on a NER projection of spans, not on the Sequential/Conditional/Parallel edges themselves. The paper acknowledges this, but it is a real gap: the Stage II results and the 'reasoning models bridge the Logic Gap' conclusion rest on those edge labels. I would want to see either a smaller expert agreement study on the edges or a clear statement that the proxy is the best available and the limitation is accepted.\n\nOne more minor thing: the reader and stress-test both flagged the missing statistical support, and they are right. I do not think the central argument collapses—the benchmark still stands on its own—but the abstract needs rewording and the evaluation section needs proper uncertainty reporting.\n\nWho is this for? People working on procedural text extraction, process mining, or LLM evaluation with structured outputs. The dataset is the contribution; the prompting comparison is secondary. It deserves a serious referee—the benchmark is worth putting in front of reviewers so it gets into the literature with the numbers fixed. I would accept it for review with heavy-revision expectations.\n\nNet: worth engaging, worth citing for the dataset, and worth a revision that makes the claims match the evidence.","headline":"The BREX benchmark is a real resource; the headline 'significant' claim about executable grounding is not backed by the reported numbers.","tokens_in":17607,"tokens_out":3315,"would_cite":true,"duration_ms":25667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Executable grounding via pseudo-code beats standard prompts at extracting business rules from real-world documents.","keywords":["business rule extraction","rule flow modeling","large language models","prompting strategies","executable grounding","pseudo-code","dependency graph","benchmark"],"falsifier":"Re-annotate a stratified sample of BREX documents with dependency-edge labels using the same three-expert protocol and compute Kappa on the edges themselves; if agreement on Conditional and Parallel edges falls substantially below 0.901, the Stage II 'reasoning gap' results rest on noisy ground truth. As a complementary check, run Prompt 5 and Prompt 1 on a subset with exact-match tuple evaluation instead of span-based NER F1 to see whether the pseudo-code advantage survives stricter scoring.","tokens_in":16566,"feed_emoji":"📋","tokens_out":4198,"duration_ms":28502,"temperature":0.7,"pith_summary":"This paper introduces BREX, a benchmark of 409 real-world business documents with 2,855 expert-annotated rules, each a condition–action pair linked by Sequential, Conditional, or Parallel dependencies. It argues that existing benchmarks treat procedures as linear action sequences and therefore miss the branching and concurrent logic that dominates regulatory and administrative texts. The paper proposes ExIde, a two-stage prompting framework that separates rule extraction from dependency reasoning, and tests five prompting strategies across 13 large language models. Its central claim is that executable grounding—asking the model to translate text into pseudo-code before producing structured rules—is a stronger inductive bias than standard prompts, and that reasoning-optimized models beat instruction-tuned models at tracing long-range, non-linear dependencies. If correct, this offers a fine-tuning-free lever for logic-intensive extraction in process automation.","feed_headline":"Pseudo-code prompts beat standard LLM prompts for business rules","feed_subtitle":"A 409-document benchmark shows executable grounding outdoes plain prompts for logic-heavy rule extraction.","key_machinery":"The load-bearing mechanism is the intermediate pseudo-code representation used in Prompt 5 of ExIde. The model first rewrites the business text into simple procedural primitives such as select_from() and execute_action(), then extracts rule tuples from that scaffold. This 'executable grounding' forces early resolution of nested conditions and control flow, and it is the component the paper credits for the consistent performance gains in Stage I. The second mechanism is the two-stage decompose-and-reason design: Stage I extracts atomic condition–action rules, and Stage II performs pairwise dependency classification over rule pairs to reconstruct a typed adjacency matrix.","core_discovery":"The paper's discovery, stated on its own terms, is that the 'Logic Gap' between free-form business regulations and executable, condition-dependent control flow can be partially closed by introducing intermediate executable representations during prompting. Concretely, Prompt 5, which has the LLM translate the business text into pseudo-code primitives before extracting atomic rules, achieves the highest average NER F1 (0.882) and logical-operator F1 (0.850) across 13 models, outperforming implicit alignment, explicit traceability, clarified context, and logic-definition injection. The paper also finds that reasoning-optimized models (e.g., the 'Thinking' variants) show the largest gains on Conditional and Parallel dependency identification, with the gap widening as the number of rule pairs grows, indicating that global rule-flow reconstruction requires state tracking rather than pattern matching.","pith_inferences":["The pseudo-code inductive bias likely transfers to other structured extraction tasks, such as form-filling or compliance checklist generation, because it is a general way to force control-flow awareness into LLM outputs.","The paper's dependency-graph evaluation is only as strong as the annotation of the edges; if experts disagree more on Conditional and Parallel relations than on spans, the reported reasoning gap could shrink under cleaner labels.","A testable extension is to compare P5 against a variant that uses formal rule notation (e.g., JSON with explicit branches) instead of pseudo-code, isolating whether code syntax or execution semantics drives the gain.","The 'reasoning-optimized' advantage may erode as instruction-tuned models improve; the 6% gap observed for Qwen3-30B suggests that structured reasoning mechanisms matter now, but the landscape is moving quickly."],"forward_implications":["If executable grounding works as described, practitioners can improve rule extraction on regulated documents without fine-tuning, simply by adding a pseudo-code step to their prompts.","Reasoning-optimized models are the better default for dependency-heavy documents, particularly when conditions branch and rules run in parallel.","The over-30% share of Conditional and Parallel relations in BREX suggests that action-centric benchmarks understate the logical complexity of real-world procedures.","The observed robustness of P5 under high rule density indicates that pseudo-code scaffolding mitigates the lost-in-the-middle effect for long regulatory texts.","The case study's failure mode—pseudo-code reifying discourse connectors into spurious meta-rules—means deployment should check precision on high-density documents."],"supporting_citations":[{"why":"Provides the PAGED benchmark, the synthetic action-flow baseline that BREX contrasts with on logical depth and data realism.","marker":"Du et al., 2024"},{"why":"Chain-of-thought prompting underpins all five ExIde prompting strategies, which share the same output schema and CoT-style reasoning format.","marker":"Wei et al., 2022"},{"why":"Supplies the Intraclass Correlation Coefficient used to assess inter-annotator reliability of the text quality ratings.","marker":"Shrout and Fleiss, 1979"},{"why":"Supplies the Fleiss' Kappa method used to compute the reported 0.901 inter-annotator agreement on the NER projection.","marker":"Artstein, 2017"},{"why":"Documents the lost-in-the-middle effect that the paper claims executable grounding mitigates in high-density documents.","marker":"He et al., 2024"},{"why":"Represents the action-flow extraction paradigm from tutorials and manuals that the rule-centric BREX benchmark is designed to move beyond.","marker":"Quishpi et al., 2020"}],"fun_headline_variants":["Executable grounding wins for business rule extraction","Pseudo-code prompts sharpen LLM rule extraction from regs","BREX: 409 docs, 2,855 rules, best prompts revealed","Reasoning models excel at tracing complex rule flows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported expert agreement, a Fleiss' Kappa of 0.901, was measured only on named-entity spans (slot types, reference values, actions), not on the dependency edges (Sequential, Conditional, Parallel) that the Stage II results depend on.","fun_headline_variants_meta":{"raw":{"variants":["Executable grounding wins for business rule extraction","Pseudo-code prompts sharpen LLM rule extraction from regs","BREX: 409 docs, 2,855 rules, best prompts revealed","Reasoning models excel at tracing complex rule flows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000982,"raw_usage":{"total_tokens":4176,"prompt_tokens":963,"completion_tokens":3213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3145}},"tokens_in":579,"tokens_out":3213,"duration_ms":20117,"temperature":1.0,"reasoning_tokens":3145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:30:06.508017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a stratified sample of BREX documents with dependency-edge labels using the same three-expert protocol and compute Kappa on the edges themselves; if agreement on Conditional and Parallel edges falls substantially below 0.901, the Stage II 'reasoning gap' results rest on noisy ground truth. As a complementary check, run Prompt 5 and Prompt 1 on a subset with exact-match tuple evaluation instead of span-based NER F1 to see whether the pseudo-code advantage survives stricter scoring.","supporting_citations":[],"review_version":1}