Reflection symmetry instability at high spins in 162,164Yb

A shape evolution of 162,164Yb in yrast states is traced using the self-consistent Skyrme Hartree-Fock calculations. We found that nonaxial octupole deformations (in particular, Y_{31} term) become favorable at large rotational frequencies (> 0.4 MeV) in 162Yb, while in 164Yb a nonaxial quadrupole shape is dominant at fast rotation. The cranked Nilsson model and random phase approximation are used to understand the dynamics of octupole correlations in both nuclei. We demonstrate that the disappearance of one of the octupole vibrational modes in the rotating frame gives rise to the nonaxial octupole deformations in 162Yb, while the octupole modes are nonzero in 164Yb.