Pith. sign in

REVIEW 1 major objections 4 minor 61 references

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents

T0 review · 1 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces GB/T-Bench, a benchmark for structured review of national standard documents, and reports that the best LLM reaches 0.328 CMCS versus 0.664 for experts, while its multi-agent framework raises the best score to 0.5094.

desk verdict A genuinely useful benchmark idea for an underevaluated domain, but the exact-match protocol is built on two incompatible error-code taxonomies, so the headline numbers are not reproducible as written. read the letter →

arxiv 2608.06312 v1 pith:NQDEUJWQ submitted 2026-08-06 cs.CL

Tao Wang , Qihao Yang , Rongjiao Liang , Lianghong Lin , Haitao Wang , Xinyu Cao , Tianyong Hao This is my paper
classification cs.CL
keywords GB/Tstandardsrule-intensivedocumentreviewLLMbenchmarkmulti-agentframeworkerrortaxonomycounterexamplegenerationexact-matchevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a benchmark, GB/T-Bench, for a task it calls rule-intensive review: checking a Chinese national standard document against its own drafting rules down to the exact error location, review dimension, and error type. It injects 7,306 traceable errors into 488 real GB/T documents and scores models by exact matches on all three fields. The paper reports that the strongest standalone LLM reaches 0.3280 CMCS against 0.6640 for human experts, and that its multi-agent framework, GB/T-Reviewer, raises the best score to 0.5094. The central claim is that structured skill decomposition, combining a global reviewer, dimension specialists, error-type agents, and deterministic rule scanners, narrows the human-LLM gap by roughly half, and that a diagnosis-oriented benchmark is needed to measure such progress honestly.

What carries the argument

The load-bearing machinery is a combination of three pieces. First is the GB/T Review Taxonomy, a two-level schema of five review dimensions (structure, scope, normative modality, terminology, normative references) decomposed into 25 diagnosable error types; this provides the shared vocabulary that both annotation and prediction must use. Second is the controllable counterexample generation mechanism, which combines deterministic rules with constrained LLM rewriting to inject 7,306 traceable errors into 488 real GB/T documents, each error carrying its original section, corrupted section, location, dimension, type, and description so that ground truth is known and auditable. Third is the evaluation protocol and the GB/T-Reviewer framework: the protocol counts a prediction correct only when its section number, review dimension, and error type all match ground truth, while the framework parses the document into sections, runs a Direct-All Reviewer alongside dimension specialists, error-type agents, and deterministic rule scanners, and then filters, merges, and deduplicates candidates before producing the final structured predictions.

What would settle it

Compare the error-type definitions in the annotation taxonomy (Appendix Tables 5–9) with those in the shared evaluation prompt (Table 11). If the meaning of any error-type code differs between the two sources, for instance E-D_STR-01 being 'Incorrect heading numbering format' in one and 'Missing mandatory element' in the other, then re-running the evaluation with aligned codebooks would produce different CMCS values, and the reported human-LLM gap and GB/T-Reviewer gain would be partly an artifact of mismatched taxonomies.

Watch

Extended reading notes

Core claim

On the paper's own terms, the finding is twofold. First, rule-intensive review of national standard documents is a measurable task at which current LLMs remain far from expert-level: the best standalone model attains CMCS 0.3280, recall 0.5203, and DMTR_10 0.2174, while human experts attain 0.6640, 0.8308, and 0.8479 respectively. Second, a multi-agent architecture that decomposes review knowledge into specialized skills and then verifies and merges the results recovers a substantial portion of the deficit: the GB/T-Reviewer built on GPT-5.5 reaches CMCS 0.5094, recall 0.7314, and DMTR_10 0.7860. The paper interprets the consistent improvement across all 14 evaluated models and the ablation results, in which every component contributes and the global reviewer is the most critical, as evidence that structured skill coordination is a viable route toward more trustworthy AI-assisted review of structured, rule-governed documents.

Load-bearing premise

The scores rest on the assumption that the error-type codes used to label the ground-truth instances mean exactly the same things as the codes in the prompts that the models see, and the appendix tables and the prompt table give different definitions to E-D_STR-01, so that assumption may not hold.

Editorial extensions

If this is right

  • Across all 14 tested models, the GB/T-Reviewer framework improves CMCS, with the best standalone score rising from 0.3280 to 0.5094 and DMTR_10 rising from 0.2439 to 0.7860.
  • Models are systematically stronger at structural and scope errors (D_STR, D_SCP) than at normative modality, terminology, and reference errors (D_NMS, D_TER, D_NREF), a pattern that locates the current bottleneck in rule-constrained semantic judgment rather than document-level pattern recognition.
  • Ablation results show the Direct-All Reviewer is the most critical component: removing it drops CMCS from 0.5094 to about 0.22, below the single-prompt baseline, indicating that global document context is a prerequisite for the specialist agents.
  • The paper's high-threshold metric DMTR_10 improves from 0.2439 to 0.7860 for the best model, meaning the framework substantially increases the share of documents where nearly all errors are correctly diagnosed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Applied beyond GB/T standards, the same formulation, reviewing a rule-bearing document against its own stated rules, could cover contracts, building codes, or internal compliance policies, where the document serves as both input and object of inspection.
  • The framework's separation of a global reviewer from narrow specialists and deterministic scanners suggests a general recipe for LLM agents in high-stakes document tasks: establish whole-document context first, delegate to focused experts, then verify and merge.
  • A near-miss scoring scheme that gives partial credit for a correct location with a wrong error type would reveal whether LLM failures are primarily localization failures or classification failures, information the current binary matching discards.
  • Because the paper's token analysis shows that higher token consumption does not predict better scores, a controlled comparison of single-pass long-context models against the multi-agent framework at equal total token budgets would test whether the gain comes from orchestration or simply from multiple review passes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. This paper proposes GB/T-Bench, a benchmark for structured review of Chinese national standard (GB/T) documents, together with a multi-agent framework GB/T-Reviewer. The benchmark comprises 488 documents and 7,306 injected, traceable errors across five review dimensions and 25 error types, generated by deterministic rules and constrained LLM rewriting. Evaluation requires exact matching of section, dimension, and error type, with metrics DMTR_k, Recall, and CMCS. Experiments on 14 LLMs report a large human-LLM gap (best LLM CMCS 0.3280 vs human 0.6640) and claim that GB/T-Reviewer raises the best CMCS to 0.5094. The paper's central conclusion is that LLMs remain far below expert review but that structured skill coordination with rule scanners recovers roughly half the gap.

Significance. The task is well-motivated and the resource is potentially valuable: rule-intensive review of standards is under-benchmarked, and the combination of traceable error injection, two-stage human validation, inter-annotator agreement (kappa around 0.95), and a granular diagnosis protocol is a genuine contribution. The multi-agent framework is a plausible design and the ablation study is informative. However, the significance hinges on the validity of the exact-match evaluation. As written, the error-type codes are defined inconsistently between the taxonomy tables and the model prompts, so the central quantitative claims are not supported. The absence of released data or code and the unmatched human-evaluation subset further prevent independent verification. With a consistent taxonomy and a re-run of all experiments, the benchmark could become a useful resource for the community.

major comments (1)
  1. [GB/T-Reviewer, Expert Review Module; Table 14] The deterministic rule-based scanners in Table 14, such as comparing the set of cited reference identifiers against the normative-reference list for D_NREF or constructing a terminology index for D_TER, implement checks that closely mirror the deterministic injection rules used to construct the benchmark ground truth. Because GB/T-Reviewer merges these scanner outputs with LLM-generated candidates before scoring, the CMCS improvement attributed to GB/T-Reviewer may largely reflect rule-based recovery rather than coordinated LLM reasoning. The ablation in Figure 4 removes the rule/local scanners as a whole, but the magnitude of the effect and its per-dimension breakdown are not reported. Please provide scanner-only performance, per-dimension results, and a version of GB/T-Reviewer without deterministic scanners on D_NREF and D_TER to support the claim that structured skill coordination, rather than rule replication, drives the improvement.
minor comments (4)
  1. [Figure 1 and Appendix A] Figure 1 uses the code 'E-DSTR-05' while the taxonomy and prompt tables use 'E-D_STR-05'; please normalize the code notation to a single convention.
  2. [Evaluation Metrics, CMCS] The main-text CMCS equation is typeset without parentheses and is unreadable; Appendix B gives a clear per-document formulation and should be used in the main text.
  3. [Tables 2 and 3] The text refers to 'Recall' but the tables append a footnote that it means Diagnosis Recall; please rename the metric consistently (e.g., 'Diag-Recall') in all tables, figures, and the metric definitions.
  4. [References] Several citations are to very recent or concurrent preprints and a few entries appear in multiple contexts; please verify the bibliographic details and years before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified: the benchmark, taxonomy, and reviewer are explicitly disclosed and not fitted to each other; their shared rule vocabulary is a design feature, not a circular derivation.

full rationale

The paper's central claims are empirical: constructing a traceable benchmark, evaluating LLMs, and proposing a multi-agent reviewer. None of these claims reduces by definition to its inputs. The GB/T Review Taxonomy is explicitly stated, and the counterexample generation is described as combining deterministic rules with constrained LLM rewriting. The GB/T-Reviewer's rule-based scanners use the same public domain knowledge (e.g., citation-set comparison for E-D_NREF-01) that the benchmark's generation mechanism presumably inverts, but this is disclosed design rather than an unstated fitted parameter: the reviewer receives only the document under review at inference time, and the rules are general GB/T drafting requirements, not statistics learned from the benchmark instances. The statement that GB/T-Reviewer 'remains independent of the counterexample construction process' is an operational independence claim (no access to labels or construction details), not an epistemic guarantee that no shared rules exist; sharing domain rules with the generator does not make the evaluation circular. The self-citations in the paper (Yang et al. 2025b; Yang et al. 2026) appear only in related-work context and are not load-bearing for any result or method choice. The more serious issue is not circularity but internal consistency: Appendix A Table 5 defines E-D_STR-01 as 'Incorrect heading numbering format' and E-D_STR-05 as 'Incorrect header/footer format', while Table 11's shared prompt defines E-D_STR-01 as 'Missing mandatory element' and E-D_STR-05 as 'Incorrect appendix classification'; Figure 1's example agrees with Table 11. Because the evaluation metric requires exact error_type string matching, this taxonomy mismatch threatens the validity and reproducibility of the reported CMCS values, but it is a correctness/reproducibility flaw, not a circularity pattern. No step in the paper's derivation is equivalent by construction to its inputs, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central evaluation depends on taxonomic consistency and on the assumption that generated errors represent real review errors. The CMCS metric adds hand-set hyperparameters. The rule scanners are carefully engineered to the benchmark's own error definitions.

free parameters (4)
  • lambda (CMCS missed-error penalty) = 1.0
    Hand-set in all experiments; controls how strongly missed errors reduce the score, and the paper reports no sensitivity analysis.
  • alpha (CMCS redundancy penalty scale) = 0.15
    Hand-set; penalizes redundant predictions in CMCS.
  • beta (CMCS redundancy penalty saturation) = 100.0
    Hand-set; controls saturation of the redundancy penalty.
  • Per-document error count range = 10-18
    The construction process appears to control errors per document to this range, which affects DMTR_k thresholds.
assumptions (4)
  • domain assumption GB/T 1.1 drafting rules define the correct review taxonomy and error labels.
    The taxonomy is the basis for all ground-truth annotations and evaluation; the paper asserts but does not fully derive it.
  • domain assumption Ground-truth error-type codes match the error-type codes shown in the model prompt.
    Exact-match scoring requires this; Appendix A and Table 11 currently contradict each other for D_STR codes.
  • domain assumption LLM-rewritten semantic errors are natural and resemble real human review errors.
    The benchmark substitutes generated errors for naturally occurring ones; no natural error corpus is provided for comparison.
  • domain assumption The reviewer's rule-based scanners are independent of the counterexample generation mechanism.
    The paper asserts this independence, but scanner rules in Table 14 mirror the deterministic rules used to create benchmark errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents." pith.science (2026). https://pith.science/paper/NQDEUJWQ

@misc{pith2026260806312,
  author       = {Pith},
  title        = {Pith review of: Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQDEUJWQ}},
  note         = {Machine review of arXiv:2608.06312}
}
read the original abstract

Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency. Existing benchmarks focus on domain knowledge and question answering, largely overlooking intrinsic quality review for professional documents. Such reviews rely heavily on human experts, making them costly and difficult to scale. To bridge this gap, we introduce GB/T-Bench, the first benchmark for the structured review of national standard documents. Its GB/T Review Taxonomy is a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. A controllable counterexample generation mechanism combines deterministic rules and constrained LLM rewriting to process 488 documents into 7,306 traceable review error instances for evaluation. We also develop a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics. We further propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Experiments with 14 mainstream LLMs reveal a substantial human-LLM gap: the strongest model achieves only 0.3280 CMCS versus 0.6640 for experts. GB/T-Reviewer raises the best CMCS to 0.5094, showing the value of structured skill coordination for rule-intensive document review. This work paves the way for trustworthy AI in standardization and other high-stakes document domains.

Figures

Figures reproduced from arXiv: 2608.06312 by the authors.

Figure 1
Figure 1. A GB/T document review case and dataset statistics [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of GB/T-Bench and GB/T-Reviewer, consisting of a traceable error-injection-based dataset construction [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Location, dimension, and diagnosis recall across [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study results of GPT-5.5. Recall refers to [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Hierarchical recall of 14 standalone LLMs on GB/T-Dataset. Location, Dimension, and Diagnosis represent increasingly [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Average per-task input and output token consumption across the evaluated LLMs. Each stacked bar shows the input [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages

  1. [1]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2308.12966 , year=

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond , author=. arXiv preprint arXiv:2308.12966 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2501.15383 , year=

    Qwen2.5-1M Technical Report , author=. arXiv preprint arXiv:2501.15383 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2605.26494 , year=

    The minimax-m2 series: Mini activations unleashing max real-world intelligence , author=. arXiv preprint arXiv:2605.26494 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2606.19348 , year=

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  7. [7]

    Advances in neural information processing systems , volume=

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models , author=. Advances in neural information processing systems , volume=

  8. [8]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    LexGLUE: A benchmark dataset for legal language understanding in English , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

Show all 61 references
  1. [9]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Lawbench: Benchmarking legal knowledge of large language models , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  2. [10]

    Pubmedqa: A dataset for biomedical research question answering , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  3. [11]

    arXiv preprint arXiv:2311.11944 , year=

    Financebench: A new benchmark for financial question answering , author=. arXiv preprint arXiv:2311.11944 , year=

    arXiv

  4. [12]

    Advances in neural information processing systems , volume=

    Finben: A holistic financial benchmark for large language models , author=. Advances in neural information processing systems , volume=

  5. [13]

    Fineval: A chinese financial domain knowledge evaluation benchmark for large language models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Paper...

    2025

  6. [14]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    E-EVAL: A comprehensive Chinese K-12 education evaluation benchmark for large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  7. [15]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    MoZIP: A multilingual benchmark to evaluate large language models in intellectual property , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    2024

  8. [16]

    Journal of Artificial Intelligence Research , author=

    IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models , volume=. Journal of Artificial Intelligence Research , author=. 2025 , month=. doi:10.70891/JAIR.2025.040011 , abstractNote=

    2025

  9. [17]

    arXiv e-prints , keywords =

    MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv e-prints , keywords =. doi:10.48550/arXiv.2205.00445 , archivePrefix =. 2205.00445 , primaryClass =

    doi arXiv

  10. [18]

    The Eleventh International Conference on Learning Representations , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  11. [19]

    Advances in neural information processing systems , volume=

    Toolformer: Language models can teach themselves to use tools , author=. Advances in neural information processing systems , volume=

  12. [20]

    URL https://arxiv

    Gorilla: Large language model connected with massive apis, 2023 , author=. URL https://arxiv. org/abs/2305.15334 , year=

    2023 arXiv

  13. [21]

    International Conference on Learning Representations , volume=

    Toolllm: Facilitating large language models to master 16000+ real-world apis , author=. International Conference on Learning Representations , volume=

  14. [22]

    AutoGen: Enabling Next-Gen

    Qingyun Wu and Gagan Bansal and Jieyu Zhang and Yiran Wu and Beibin Li and Erkang Zhu and Li Jiang and Xiaoyun Zhang and Shaokun Zhang and Jiale Liu and Ahmed Hassan Awadallah and Ryen W White and Doug Burger and Chi Wang , booktitle=. AutoGen: Enabling Next-Gen. 2024 , url=

    2024

  15. [23]

    arXiv preprint arXiv:2311.17541 , year=

    Taskweaver: A code-first agent framework , author=. arXiv preprint arXiv:2311.17541 , year=

    arXiv

  16. [24]

    International Conference on Learning Representations , volume=

    Critic: Large language models can self-correct with tool-interactive critiquing , author=. International Conference on Learning Representations , volume=

  17. [25]

    27th USENIX Security Symposium (USENIX Security 18) , pages=

    Polisis: Automated analysis and presentation of privacy policies using deep learning , author=. 27th USENIX Security Symposium (USENIX Security 18) , pages=

  18. [26]

    Question answering for privacy policies: Combining computational and legal perspectives , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages=

    2019

  19. [27]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    PolicyQA: A reading comprehension dataset for privacy policies , author=. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    2020

  20. [28]

    IEEE Transactions on Software Engineering , volume=

    Nlp-based automated compliance checking of data processing agreements against gdpr , author=. IEEE Transactions on Software Engineering , volume=. 2023 , publisher=

    2023

  21. [29]

    ACM Computing Surveys (CSUR) , volume=

    Natural language processing for requirements engineering: A systematic mapping study , author=. ACM Computing Surveys (CSUR) , volume=. 2021 , publisher=

    2021

  22. [30]

    txt, a cheap Shallow Parsing approach for Regulatory texts , author=

    SPAR. txt, a cheap Shallow Parsing approach for Regulatory texts , author=. Proceedings of the Natural Legal Language Processing Workshop 2021 , pages=

    2021

  23. [31]

    Scientific data , volume=

    CODE-ACCORD: A Corpus of building regulatory data for rule generation towards automatic compliance checking , author=. Scientific data , volume=. 2025 , publisher=

    2025

  24. [32]

    arXiv preprint arXiv:2603.23519 , year=

    MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios? , author=. arXiv preprint arXiv:2603.23519 , year=

  25. [33]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  26. [34]

    IEEE access , volume=

    A review on large language models: Architectures, applications, taxonomies, open issues and challenges , author=. IEEE access , volume=. 2024 , publisher=

    2024

  27. [35]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  28. [36]

    arXiv preprint arXiv:2508.02324 , year=

    Qwen-image technical report , author=. arXiv preprint arXiv:2508.02324 , year=

    arXiv

  29. [37]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Finrpt: Dataset, evaluation system and llm-based multi-agent framework for equity research report generation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  30. [38]

    arXiv preprint arXiv:2605.10267 , year=

    IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs , author=. arXiv preprint arXiv:2605.10267 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2506.16400 , year=

    Physical-layer signal injection attacks on ev charging ports: Bypassing authentication via electrical-level exploits , author=. arXiv preprint arXiv:2506.16400 , year=

    arXiv

  32. [40]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , url =

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and ichter, brian and Xia, Fei and Chi, Ed and Le, Quoc V and Zhou, Denny , booktitle =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , url =

  33. [41]

    The Eleventh International Conference on Learning Representations , year=

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  34. [42]

    Training language models to follow instructions with human feedback , url =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and As...

  35. [43]

    L -Eval: Instituting Standardized Evaluation for Long Context Language Models

    An, Chenxin and Gong, Shansan and Zhong, Ming and Zhao, Xingjian and Li, Mukai and Zhang, Jun and Kong, Lingpeng and Qiu, Xipeng. L -Eval: Instituting Standardized Evaluation for Long Context Language Models. Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024 doi

  36. [44]

    2024 , url=

    Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen , booktitle=. 2024 , url=

    2024

  37. [45]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Edubench: A comprehensive benchmarking dataset for evaluating large language models in diverse educational scenarios , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  38. [46]

    The Twelfth International Conference on Learning Representations , year=

    On the humanity of conversational ai: Evaluating the psychological portrayal of llms , author=. The Twelfth International Conference on Learning Representations , year=

  39. [47]

    Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    ContractNLI: A dataset for document-level natural language inference for contracts , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    2021

  40. [48]

    and Gardner, Matt

    Dasigi, Pradeep and Lo, Kyle and Beltagy, Iz and Cohan, Arman and Smith, Noah A. and Gardner, Matt. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu...

    2021 doi

  41. [49]

    ACORD : An Expert-Annotated Retrieval Dataset for Legal Contract Drafting

    Wang, Steven H and Zubkov, Maksim and Fan, Kexin and Harrell, Sarah and Sun, Yuyang and Chen, Wei and Plesner, Andreas and Wattenhofer, Roger. ACORD : An Expert-Annotated Retrieval Dataset for Legal Contract Drafting. Proceedings of the 63rd Annual Meeting of the Association f...

    2025 doi

  42. [50]

    Advances in Neural Information Processing Systems , volume=

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations , author=. Advances in Neural Information Processing Systems , volume=

  43. [51]

    Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (long papers) , pages=

    A dataset of peer reviews (PeerRead): Collection, insights and NLP applications , author=. Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (long papers) , pages=

    2018

  44. [52]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Maud: An expert-annotated legal nlp dataset for merger agreement understanding , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  45. [53]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Finmme: Benchmark dataset for financial multi-modal reasoning evaluation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  46. [54]

    D oc M ath-Eval: Evaluating Math Reasoning Capabilities of LLM s in Understanding Long and Specialized Documents

    Zhao, Yilun and Long, Yitao and Liu, Hongjun and Kamoi, Ryo and Nan, Linyong and Chen, Lyuhao and Liu, Yixin and Tang, Xiangru and Zhang, Rui and Cohan, Arman. D oc M ath-Eval: Evaluating Math Reasoning Capabilities of LLM s in Understanding Long and Specialized Documents. Pro...

    2024 doi

  47. [55]

    L ong D oc URL : a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

    Deng, Chao and Yuan, Jiale and Bu, Pi and Wang, Peijie and Li, Zhong-Zhi and Xu, Jian and Li, Xiao-Hui and Gao, Yuan and Song, Jun and Zheng, Bo and Liu, Cheng-Lin. L ong D oc URL : a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Lo...

    2025 doi

  48. [56]

    Marathon: A Race Through the Realm of Long Context with Large Language Models

    Zhang, Lei and Li, Yunshui and Liu, Ziqiang and Yang, Jiaxi and Liu, Junhao and Chen, Longze and Luo, Run and Yang, Min. Marathon: A Race Through the Realm of Long Context with Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024 doi

  49. [57]

    LONGAGENT : Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration

    Zhao, Jun and Zu, Can and Hao, Xu and Lu, Yi and He, Wei and Ding, Yiwen and Gui, Tao and Zhang, Qi and Huang, Xuanjing. LONGAGENT : Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration. Proceedings of the 2024 Conference on Empirical Me...

    2024 doi

  50. [58]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 , pages =

    Mohammadi, Mahmoud and Li, Yipeng and Lo, Jane and Yip, Wendy , title =. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 , pages =. 2025 , isbn =. doi:10.1145/3711896.3736570 , abstract =

    2025

  51. [59]

    2021 , url=

    Dan Hendrycks and Collin Burns and Anya Chen and Spencer Ball , booktitle=. 2021 , url=

    2021

  52. [60]

    Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

    LLM-based collaborative agents with pedagogy-guided interaction modeling for timely instructive feedback generation in task-oriented group discussions , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

  53. [61]

    Proceedings of the AAAI Conference on Artificial Intelligence , pages =

    HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , pages =

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.