Degenerate Hamiltonian operator in higher-order canonical gravity -- the problem and a remedy

Different routes towards the canonical formulation of a classical theory result in different canonically equivalent Hamiltonians, while their quantum counterparts are related through appropriate unitary transformation. However, for higher-order theory of gravity, although two Hamiltonians emerging from the same action differing by total derivative terms are related through canonical transformation, the difference transpires while attempting canonical quantization, which is predominant in non-minimally coupled higher-order theory of gravity. We follow Dirac's constraint analysis to formulate phase-space structures, in the presence (case-I) and absence (case-II) of total derivative terms. While the coupling parameter plays no significant role as such for case-I, quantization depends on its form explicitly in case-II, and as a result, unitary transformation relating the two is not unique. We find certain mathematical inconsistency in case-I, for modified Gauss-Bonnet-Dilatonic coupled action, in particular. Thus, we conclude that total derivative terms indeed play a major role in the quantum domain and should be taken care of a-priori, for consistency.