We compared 4 programs designed to identify species of bats from their echolocation calls (Bat Call ID, EchoClass, Kaleidoscope Pro, and SonoBat) using field data collected in Nebraska, USA (29,782 files). Although we did not know the true identity of these bats, we could still compare the pairwise agreement between software packages when identifying the same call sequences. If accuracy is high in these software packages, there should be high agreement in identification. Agreement in identification by species averaged approximately 40% and varied by software package, species, and data set. Our results are not consistent with the high accuracy often claimed by some software packages and may be a warning about the importance of understanding accuracy of acoustical identification in designing ecological experiments and interpreting results.