The Next 700 Challenge Problems for Reasoning with Higher-Order Abstract Syntax Representations: Part 1-A Common Infrastructure for Benchmarks

A variety of logical frameworks support the use of higher-order abstract syntax (HOAS) in representing formal systems. Although these systems seem superficially the same, they differ in a variety of ways; for example, how they handle a context of assumptions and which theorems about a given formal system can be concisely expressed and proved. Our contributions in this paper are three-fold: 1) we develop a common infrastructure for representing benchmarks for systems supporting reasoning with binders, 2) we present several concrete benchmarks, which highlight a variety of different aspects of reasoning within a context of assumptions, and 3) we design an open repository ORBI, (Open challenge problem Repository for systems supporting reasoning with BInders). Our work sets the stage for providing a basis for qualitative comparison of different systems. This allows us to review and survey the state of the art, which we do in great detail for four systems in Part 2 of this paper (Felty et al, 2015). It also allows us to outline future fundamental research questions regarding the design and implementation of meta-reasoning systems.