Visualization of Unconventional Rashba Band and Vortex Zero Mode in Topopogical Superconductor Candidate AuSn$_{4}$

Topological superconductivity (TSC) is a promising platform to host Majorana zero mode (MZM) for topological quantum computing. Recently, the noble metal alloy AuSn$_{4}$ has been identified as an intrinsic surface TSC. However, the atomic visualization of its nontrivial surface states and MZM remains elusive. Here, we report the direct observation of unconventional surface states and vortex zero mode at the gold (Au) terminated surfaces of AuSn$_{4}$, by ultra-low scanning tunneling microscope/spectroscopy. Distinct from the trivial metallic bulk states at tin (Sn) surfaces, the Au terminated surface exhibits pronounced surface states near Fermi level. Our density functional theory calculations indicate that these states arise from unconventional Rashba bands, where two Fermi circles from different bands share identical helical spin textures, chiralities, and group velocities in the same direction. Furthermore, we find that although the superconducting gap, critical temperature, anisotropic in-plane critical field are almost identical on Au and Sn terminated surfaces, the in-gap bound states inside Abrikosov vortex cores show significant differences. The vortex on Sn terminated surfaces exhibits a conventional Caroli-de Gennes-Matricon bound state while the Au surface shows a sharp zero-energy core state with a long non-splitting distance, resembling an MZM in a non-quantum-limit condition. This distinction may result from the dominant contribution of unconventional Rashba bands near Fermi energy from Au terminated surface. Our results provide a new platform for studying unconventional Rashba band and MZM in superconductors.