For some people, the type of music you like is effectively what defines who you are as a person and where you fit into society. They consider that music to be a core part of their identity and their life, rather than something you just listen to and pass the time. Perhaps most of their social life involves going to gigs, meaning that if you want to be friends with them, liking the same sort of music really helps. A band t-shirt helps them to basically display which "club" they belong to and to be recognised on that basis.
It's probably not all that different to how people wear items to show that they belong to a particular religion, or are from a particular country/culture, support a particular football team, have a certain amount of wealth (and thereby belong to a particular socioeconomic class)... People wear this type of thing, or sometimes actively refrain from wearing them, just because they're trying to basically show the world who they are.