Boys would soon decide they wouldn't like girls walking around without bras on, thats what pulls the boobs up, for those with bigger chests, everything soon starts to decend. Especially those who have babies or are getting older, would you find it attractive to see an 80 year old woman just enjoying the breeze as she walked past you?
Tribal women who walk around topless are used to that, plus I don't think they're seen as a sexual attribute in their societies, just that they produce milk for the babies. When girls wear bras for most of their lives it feels completely different to go without it. One time I forgot to put my bra on before I went to school and only realised when I arrived, the entire day it felt uncomfortable and I kept having to cross my arms over or under my boobs to get some support and make it look less werid.