Women's breasts are sexual. They're supposed to be, evolutionarily. They're very distracting for men in a way which is not often appreciated by women. Men's breasts on the other hand do not have the same influence on women.. so there's nothing to do about it.
Being very flat-chested and mistaken for a male, you'll probably find nobody cares. Being found out as a girl, it's perceived as weird because of course most women cover up. At this point is an entirely social issue, and you're given weird looks for "deviant behaviour" more than anything else.
But generally the so-called "double standard" exists because men and women respond differently to the other sexes breasts.