Perceptual learning is sometimes characterized by rapid improvements in performance within the first hour of training (fast perceptual learning), which may be accompanied by changes in sensory and/or response pathways. Here, we report rapid physiological changes in the human auditory system that coincide with learning during a 1-hour test session in which participants learned to identify two consonant vowel syllables that differed in voice onset time. Within each block of trials, listeners were also presented with a broadband noise control stimulus to determine whether changes in auditory evoked potentials were specific to the trained speech cue. The ability to identify the speech sounds improved from the first to the fourth block of trials and remained relatively constant thereafter. This behavioral improvement coincided with a decrease in N1 and P2 amplitude, and these learning-related changes differed from those observed for the noise stimulus. These training-induced changes in sensory evoked responses were followed by an increased negative peak (between 275 and 330 msec) over fronto-central sites and by an increase in sustained activity over the parietal regions. Although the former was also observed for the noise stimulus, the latter was specific to the speech sounds. The results are consistent with a top–down nonspecific attention effect on neural activity during learning as well as a more learning-specific modulation, which is coincident with behavioral improvements in speech identification.