Pith.

Artificial Intelligence · arXiv:2609.37658 v2 · submitted 29 Sep 2026

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

Min Yang, Yichen Pan, Jinghua Piao, Dandan Song, Yongshun Gong, Yong Li

The paper says

“LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs.”

more of the abstract

However, existing enterprise and financial benchmarks mainly test static capabilities such as information extraction, numerical calculation, domain knowledge, and financial QA, leaving interactive and long-horizon decision-making underexplored. To bridge this gap, we introduce EnterpriseBench, a benchmark that evaluates LLM agents across this spectrum, from static question answering to dynamic decision-making. Specifically, EnterpriseBench reorganizes existing enterprise and financial QA datasets into a unified foundational suite annotated by capability and difficulty, and introduces three professional interactive settings: Consulting, based on management-consulting-style business cases for client problem diagnosis through multi-turn information seeking; the Beer Game, adapted from a classic supply-chain management simulation for inventory control under delayed feedback; and Enterprise Digital Twin, a project-based business simulator for workforce, risk, and project planning. Experiments with nine agent methods under four backbone models show that current agents have not yet achieved stable, comprehensive, and cross-task reliability in enterprise scenarios. These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 30 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Honest two-layer benchmark whose qualitative non-transfer finding holds up, but whose specific interactive method rankings sit inside the evaluator's own noise. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §4.2, Tables 1–3 and 8

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The Consulting backbone-reversal claim (GEPA best under DeepSeek-V3 at 8.28 vs DC 8.00; DC best under GPT-4.1 at 8.08 vs GEPA 7.89) rests on gaps of 0.28 and 0.19 points, which are smaller than the paper's own reported judge noise: the expanded audit (Table 8, Appendix A.10) reports an overall MAE of 0.679 with 85% within-one-point agreement, and the prompt-robustness test (Fig. 3a) shifts scores by up to 0.12.

more of objection 1

The paper reports these audit statistics honestly but never propagates the judge uncertainty into the method-level rankings, so the statement that the 'best-performing method varies' across backbones for Consulting is not resolvable at the reported precision. Please compute confidence intervals or a significance test on the method differences (e.g., per-case paired scores rather than means) and either qualify the reversal claim or restrict it to margins that exceed the judge's reliability band.

The paper says, at §4.2, Appendix A.6.4/A.6.6

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

EDT results for each method are based on a single stochastic episode: extension and follow-on events are gated by random draws (A.6.4, A.6.6) yet no seeds or variance are reported. Under DeepSeek-V3, AMEM (6.76M), Reflexion (6.20M), and GEPA (6.04M) lie within 0.72M of one another, while the same method across backbones spans 5.54M–8.16M (Tables 1, 2, 5, 6) — a spread comparable to any cross-method effect.

more of objection 2

The EDT-based part of the ranking-instability claim (AMEM best under DeepSeek-V3, GEPA best under GPT-4.1) is therefore consistent with Monte Carlo noise. Please run multiple seeds per method/backbone, report mean ± std (or interquartile range), and test whether the leading-method changes are significant relative to the simulator's own variance.

The paper says, at Table 5, Appendix A.5

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The Beer Game is run as a single 25-step episode with one demand script and one set of cost coefficients (A.5), and under DeepSeek-V4-Pro (Table 5) six of the seven methods report essentially identical total costs around 4.67×10^4, with only AMEM deviating (2.99×10^4).

more of objection 3

An identical outcome across six distinct agent methods strongly suggests the episode is non-discriminative or degenerate for that backbone (the agent's ordering decisions may be dominated by the simulation dynamics), which undercuts the claim that method rankings vary meaningfully across backbones for the Beer Game. Please (i) investigate and report why six methods produce the identical cost, and (ii) report results over multiple demand scripts / seeds with variance so that cross-backbone ranking comparisons are meaningful. A single fixed configuration for each of the three interactive tasks leaves the observed non-transfer (part a) and ranking instability (part b) open to the concern that they are configuration-specific rather than general agent limitations.

4 minor objections

The paper says, at References

Pith says: minor objection 1 of 4

The GEPA reference is typeset as 'Gepa: Reflective prompt evolution...' in the bibliography and in sentence b182; the capitalization should be consistent with the method name GEPA used in the text.

The paper says, at Appendix A.5, Eq. (1)

Pith says: minor objection 2 of 4

The opponent-policy coefficients α=1 and β=1 are stated as fixed values without justification; a sentence explaining the choice (or noting it as a default from prior Beer Game studies) would help readers judge sensitivity of the results to these settings.

show 2 more

The paper says, at Appendix A.11

Pith says: minor objection 3 of 4

Table 9 reports accuracy on only 50 evaluation instances as part of resource measurement; please add a sentence clarifying that this is an illustrative resource probe and not the benchmark's reported accuracy metric, to avoid confusion with the full-scale results in Tables 1–2.

The paper says, at §4.4

Pith says: minor objection 4 of 4

The AOA improvement (0.725 to 0.729) is presented as a proof of concept; the paper should state explicitly that no significance test is attached to this 0.004 gain, given it is smaller than the difficulty-annotation audit's 0.47-point mean absolute difference on the analogous scoring scale.

What would settle it

Re-run EDT on the same template scenario (A.6) with at least 20 independent seeds per (method, backbone), recording the extension/follow-on draws, and report mean ± std of accumulated earnings; then test whether AMEM (6.76M), Reflexion (6.20M), and GEPA (6.04M) under DeepSeek-V3 are statistically separable. In parallel, use the 500-transcript judge audit to bootstrap per-method confidence intervals on the Consulting overall score and test whether GEPA (8.28) vs DC (8.00) under DeepSeek-V3, and DC (8.08) vs GEPA (7.89) under GPT-4.1, differ beyond the judge's uncertainty. If either set of top-method gaps falls within the noise band, the paper should confine its claim to the robust cross-task non-transfer result and report the per-task rankings as statistically indistinguishable.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 30 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 31 s · 27,317 tokens read, 4,744 written
  3. Editor wrote the verdict above 36 s · 29,116 tokens read, 3,342 written
  4. Referee major revision: 3 major objections, 4 minor 26 s · 27,840 tokens read, 3,787 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The finding that survives scrutiny is the qualitative one: static QA performance does not transfer to interactive enterprise tasks. That claim is robust. The cross-task gaps are large and consistent across all four backbones in Tables 1, 2, 5, and 6, so the paper's core motivation for building interactive benchmarks stands. What is weaker is the companion claim that 'the best-performing method varies across tasks and backbones.' The stress-test note is right about why. The Consulting top-two gaps (0.28 under DeepSeek-V3, 0.19 under GPT-4.1) sit inside the judge's own MAE of 0.679 from the authors' expanded 500-transcript audit. EDT is stochastic — extensions and follow-ons are gated by random draws (A.6.4), yet each method is run once, with no seeds or variance reported. Under DeepSeek-V3, AMEM, Reflexion, and GEPA land within 0.72M of one another, while the same method spans 5.5M to 8.2M across backbones. Those specific method rankings need error bars and multi-seed replication before anyone cites them. What is genuinely good: the Consulting task itself. The gated-information LLM interviewer, a judge drawn from a different model family than the evaluated backbone, and two human audits (r=0.71 on 20 cases, r=0.887 on 500 transcripts) is a serious attempt at a hard evaluation problem, better than most agent benchmarks manage. The Beer Game and EDT are adaptations of established simulations — less novel, but the MCP interfaces are reusable and the configurations are fully specified in the appendices. The authors also disclose the resource asymmetry (GEPA spends 48.9K tokens per sample vs CoT's 1.1K) rather than burying it, and the limitations section is plain about the controlled-environment scope. Proportionate concerns: each interactive task uses a single fixed configuration, so the cross-task generalization claim rests on one demand script, one template scenario, one set of opponent coefficients. The human audit also shows quantitative reasoning is the weakest-judged dimension (67.4% within-one agreement) — exactly the dimension that most often separates top methods. The foundational layer is admittedly a reorganization of existing QA datasets; novelty there is limited to the capability/difficulty annotation, which is at least human-validated at r=0.96. This is an honest, useful benchmark for the agent-evaluation community. Send it to peer review — the measurement-power gap is addressable, and the Consulting protocol deserves scrutiny and reuse. I would not cite the method rankings, but I would reuse the Consulting design.

The strongest objection

The central claim (b26–b27) has two parts: (a) strong QA does not transfer to interactive tasks, and (b) the best-performing method varies across tasks and backbones, with no single method consistently best. Part (a) is robust — cross-task gaps are large. The load-bearing weakness is part (b): the specific method rankings that establish it are reported without propagating measurement uncertainty, exactly where the margins are smallest. Consulting: under DeepSeek-V3 the top two methods differ by 0.28 (GEPA 8.28 vs DC 8.00); under GPT-4.1 by 0.19 (DC 8.08 vs GEPA 7.89). The authors' own expanded judge audit (Table 8 / A.10) reports an overall MAE of 0.679 with 85% within-one-point agreement, and prompt-robustness reordering shifts scores by up to 0.12 (Fig. 3a). The observed top-method gaps are inside the evaluator's noise band, so the Consulting backbone-reversal (GEPA best under DeepSeek-V3, DC best under GPT-4.1) rests on differences the paper cannot statistically resolve. EDT is worse: the simulator is stochastic — extension and follow-on events are gated by random draws (A.6.4, A.6.6) — yet each method is run once, with no seeds or variance reported. Under DeepSeek-V3, AMEM (6.76M), Reflexion (6.20M), and GEPA (6.04M) lie within 0.72M of one another, while the same method across backbones spans 5.54M–8.16M (Tables 1, 2, 5, 6) — a spread as large as any cross-method effect. With unseeded stochastic draws, those ranks may be noise. The paper honestly reports the audit, but it does not propagate measurement noise into the ranking claims. The qualitative non-transfer conclusion survives; the specific 'best method varies' rankings do not yet have statistical support. This is a measurement-power gap, addressable by error bars and multi-seed replication.

The reader's weakest_assumption concerned the single fixed configuration per interactive task possibly favoring particular methods and making rankings configuration artifacts. My concern is adjacent but sharper: it targets the statistical identifiability of the rankings that support b27. The reader's two concrete gaps — missing error bars on stochastic interactive results and Consulting gaps comparable to evaluator noise (MAE 0.679) — are precisely the evidence for my concern, so there is substantial overlap. I call it partial because the reader framed the issue as benchmark-configuration validity and generalization, whereas I frame it as measurement power: the reported top-method differences do not exceed the evaluator's known noise, and the EDT simulator's stochasticity (extensions/follow-ons) is unseeded with single runs, so the specific 'best method varies' rankings lack statistical support. This does not change the qualitative cross-task non-transfer conclusion (b26), which survives on large margins and four backbones, so it does not move the verdict. The reader's CONDITIONAL already captures the need to tighten the ranking claims; my analysis confirms that condition should stand, hence UNCHANGED. The paper deserves credit for reporting the judge audit (r=0.887, MAE 0.679) and prompt robustness honestly; the gap is in not applying that uncertainty to the rankings.

The referee's report

If the central claims hold, EnterpriseBench fills a genuine gap: existing enterprise/financial benchmarks are almost entirely static QA, and the paper's qualitative finding that interactive decision-making is not predictable from static QA accuracy would be practically important for benchmark design. The paper's strengths are real: the materials are drawn from professionally grounded sources (MBA casebooks, Sterman's Beer Game, the transentis EDT simulation); the Consulting judge is held in a different model family from the evaluated backbone (reducing self-evaluation bias); the judge is honestly audited against humans (r=0.887, 85% within-one-point); difficulty annotations are audited (r=0.96); and four backbones are covered. The 'QA does not transfer' message (part a of the claim) is robust, since the cross-task gaps are large. However, the 'best method varies across tasks and backbones' message (part b) is, as written, not statistically supported: the smallest margins on which it rests are inside the paper's own reported measurement noise, and the stochastic simulator results carry no variance information.

Pith's reading of the paper (a machine's summary, not the authors')

EnterpriseBench proposes that evaluating LLM agents for enterprise work needs two tiers: a static QA suite covering information extraction, numerical calculation, domain knowledge, and complex reasoning, and an interactive tier with three tasks — a simulated consulting interview, a delayed-feedback supply-chain game, and a project-planning simulator. Running nine agent methods on four backbones, the authors find that strong performance on the static QA tasks does not predict performance on the interactive tasks, and the best method changes depending on both the task and the backbone model. The paper argues this means interactive enterprise decision-making must be assessed with interactive benchmarks, and agent choice should be task- and model-dependent.

What carries the argument. The load-bearing object is the EnterpriseBench benchmark itself, built as a two-layer evaluation. The foundational layer reorganizes existing financial QA datasets into four capability-difficulty categories. The interactive layer contains three tasks: Consulting, where an LLM interviewer runs a hidden-information case interview up to 12 turns and an LLM judge scores the transcript on structure, quantitative reasoning, business sense, and communication; the Beer Game, a 25-step supply-chain simulation with delayed shipments and a demand shift where the agent controls the retailer and minimizes total cost; and Enterprise Digital Twin, a project-based simulator where the agent commits to a scenario-level decision — consultant count, revenue risk, project portfolio — before the episode runs and maximizes accumulated earnings. The paper's AOA proof-of-concept derives routing rules from execution traces to select a method per QA instance.

Core claim. The central claim is that current LLM agents have not achieved stable, cross-task reliability in enterprise scenarios, and that static QA competence is not evidence of interactive decision-making competence. The evidence: under DeepSeek-V3 the leading method differs across the three interactive tasks — GEPA on Consulting, Discussion on the Beer Game, AMEM on EDT — while under GPT-4.1, DC leads Consulting and GEPA leads both Beer Game and EDT. Foundational tasks also show that numerical-calculation and domain-knowledge accuracy falls sharply from easy to hard, while information extraction is less discriminative. The authors treat this as evidence that enterprise agent capability decomposes into distinct skills that must be measured separately.

The premise it rests on. The load-bearing premise is that the three interactive tasks, each implemented with a single fixed configuration (one Beer Game demand script and opponent-policy coefficients, one EDT template), are representative of enterprise decision-making; the authors acknowledge the environments are controlled and text-only, so if those configurations are unrepresentative, the observed non-transfer and ranking instability could be artifacts rather than general agent limitations.

If the paper is right (Pith's inferences, not the authors')

Enterprise agent benchmarks should include interactive, feedback-driven tasks alongside static QA, because QA accuracy does not predict interactive success.

Reported leadership by an agent method on one enterprise task should not be read as general enterprise capability, since the best method changes across tasks and backbones.

Agent selection should be treated as task- and model-dependent; the paper's routing proof-of-concept shows a small gain (0.725 to 0.729 on QA) from assigning instances to methods dynamically.

The evaluation protocol can be made robust: prompt-variant scores move by at most 0.12 points and the LLM-based consulting judge agrees with human ratings on 500 transcripts (r=0.887).

If the non-transfer result generalizes, enterprise agent evaluation should separate QA skill and interactive skill into independent axes, and future benchmarks could explicitly measure a 'transfer gap' between them.

The task-specific ranking instability suggests that different interactive challenges — hidden-information dialogue, delayed-feedback control, long-horizon planning — engage different model sub-skills, which is why routing or ensemble approaches look promising even when each individual method stays mediocre.

A testable extension is to vary the interactive-task configurations (more Beer Game demand scripts, multiple EDT templates) and check whether the instability persists; if it does not, the paper's central claim would need qualification.

One open question the paper leaves implicit is how human experts would perform on the same interactive tasks; without that anchor, the benchmark shows that agents lag static QA expectations but not how far they are from enterprise-grade performance.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs. However, existing enterprise and financial benchmarks mainly test static capabilities such as information extraction, numerical calculation, domain knowledge, and financial QA, leaving interactive and long-horizon decision-making underexplored. To bridge this gap, we introduce EnterpriseBench, a benchmark that evaluates LLM agents across this spectrum, from static question answering to dynamic decision-making. Specifically, EnterpriseBench reorganizes existing enterprise and financial QA datasets into a unified foundational suite annotated by capability and difficulty, and introduces three professional interactive settings: Consulting, based on management-consulting-style business cases for client problem diagnosis through multi-turn information seeking; the Beer Game, adapted from a classic supply-chain management simulation for inventory control under delayed feedback; and Enterprise Digital Twin, a project-based business simulator for workforce, risk, and project planning. Experiments with nine agent methods under four backbone models show that current agents have not yet achieved stable, comprehensive, and cross-task reliability in enterprise scenarios. These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-30

The authors' main claim, as the classifier picked it out 0.54

“These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.26 · human participants 0.11 · controlled comparison 0.14 · randomized assignment 0.02 · states quantitative result 0.54 · compares with prior results 0.65 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs. background 1.00 0.05 0.02 asserted 0.89
However, existing enterprise and financial benchmarks mainly test static capabilities such as information extraction, numerical calculation, domain knowledge, and financial QA, leaving interactive and long-horizon decision-making underexplored. gap 1.00 0.08 0.03 asserted 1.00
To bridge this gap, we introduce EnterpriseBench, a benchmark that evaluates LLM agents across this spectrum, from static question answering to dynamic decision-making. contribution 1.00 0.88 0.07 asserted 0.98
Specifically, EnterpriseBench reorganizes existing enterprise and financial QA datasets into a unified foundational suite annotated by capability and difficulty, and introduces three professional interactive settings: Consulting, based on management-consulting-style business cases for client problem diagnosis through multi-turn information seeking; the Beer Game, adapted from a classic supply-chain management simulation for inventory control under delayed feedback; and Enterprise Digital Twin, a project-based business simulator for workforce, risk, and project planning. method 0.50 0.93 0.04 asserted 0.99
Experiments with nine agent methods under four backbone models show that current agents have not yet achieved stable, comprehensive, and cross-task reliability in enterprise scenarios. result 1.00 0.96 0.48 asserted 1.00
These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making. interpretation 0.99 0.81 0.77 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 5,061 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Overview of EnterpriseBench. (a) EnterpriseBench organizes enterprise-oriented evaluation into a layered
Figure 2 of the paper
Figure 2. Comparison of human and LLM scores on 20
Figure 3 of the paper
Figure 3. (a) Prompt robustness analysis for Consulting using ACE. Reorder-I and Brief-I denote interviewer

Figures are the authors', reproduced from the paper.

Cited by 2 Pith papers

Reviewed papers on Pith that cite this one. The full citation record

  1. DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents cs.SE · Nov 2026
  2. FDE-Gym: The Last Mile of AI Delivery cs.AI · Oct 2026

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment