Although, acquired amusia is a common deficit following stroke, relatively little is still known about its precise neural basis, let alone to its recovery. Recently, we performed a voxel-based lesion-symptom mapping (VLSM) and morphometry (VBM) study which revealed a right lateralized lesion pattern, and longitudinal gray matter volume (GMV) and white matter volume (WMV) changes that were specifically associated with acquired amusia after stroke. In the present study, using a larger sample of stroke patients (N = 90), we aimed to replicate and extend the previous structural findings as well as to determine the lesion patterns and volumetric changes associated with amusia recovery. Structural MRIs were acquired at acute and 6-month post-stroke stages. Music perception was behaviorally assessed at acute and 3-month post-stroke stages using the Scale and Rhythm subtests of the Montreal Battery of Evaluation of Amusia (MBEA). Using these scores, the patients were classified as non-amusic, recovered amusic, and non-recovered amusic. The results of the acute stage VLSM analyses and the longitudinal VBM analyses converged to show that more severe and persistent (non-recovered) amusia was associated with an extensive pattern of lesions and GMV/WMV decrease in right temporal, frontal, parietal, striatal, and limbic areas. In contrast, less severe and transient (recovered) amusia was linked to lesions specifically in left inferior frontal gyrus as well as to a GMV decrease in right parietal areas. Separate continuous analyses of MBEA Scale and Rhythm scores showed extensively overlapping lesion pattern in right temporal, frontal, and subcortical structures as well as in the right insula. Interestingly, the recovered pitch amusia was related to smaller GMV decreases in the temporoparietal junction whereas the recovered rhythm amusia was associated to smaller GMV decreases in the inferior temporal pole. Overall, the results provide a more comprehensive picture of the lesions and longitudinal structural changes associated with different recovery trajectories of acquired amusia.