Listeners recognize talker identity by storing exemplars or by forming abstract representations. However, it is unclear which strategy they use for nonnative-accented talkers, whose speech requires more cognitive effort to process. We examined whether native English listeners rely on abstraction or exemplars when learning to recognize Mandarin-accented and American-accented talkers. We trained listeners to identify voices that varied in glottal pulse rate and vocal tract length. They then made recognition judgments for two types of stimuli: ring-shaped tokens, at the perimeter of a talker’s voice space and heard during training; center tokens, at the average of the trained distribution but not heard during training. Results showed higher accuracy for center tokens and crucially, this pattern emerged for both native and nonnative talkers, suggesting that talker recognition relies on abstraction-based encoding strategy regardless of listeners’ prior experience with specific talker types.