Individual differences in pitch discrimination have been associated with the volume of both the bilateral Heschl’s gyrus and the right inferior frontal gyrus (IFG). However, most of these studies used samples composed of individuals with different amounts of musical training. Here, we investigated the relationship between pitch discrimination and individual differences in the gray matter (GM) volume of these brain structures in 32 adult musicians, 28 adult non-musicians, and 32 children without musical training. The results showed that (i) the individuals without musical training (whether children or adults) who were better at pitch discrimination had greater volume of auditory regions, whereas (ii) musicians with better pitch discrimination had greater volume of the IFG. These results suggest that the relationship between pitch discrimination and the volume of auditory regions is innately established early in life, and that musical training modulates the volume of the IFG, probably improving audio-motor connectivity. This is the first study to detect a relationship between pitch discrimination ability and GM volume before beginning any musical training in children and adults.