Listen, its not a big deal, but you might be an "XXY Male". i.e. you have an extra female chromosome. Its not that uncommon, and generally most men don't even realise they have this and they lead normal lives without ever knowing.
Somtimes it means having a 'femenine' body. This means you could be curvy and possibly even have excess breast tissue (gynecomastia), which can easily be removed by operation. I just happen to have read about this in medical journals, etc.
I would reccomend building some muscle if it bothers you that much. building up muscle on your upper body would hide curvyness...have you seen how masculine some of those female body builders look!?