Well, do you think you will wear it over a hoodie? Do you plan to - and how does the small one you tried fit you normally, is it a perfect fit, slightly too big or small?
Mine is actually a boys age 9-10 gilet (I really wanted one for that awkward time between spring and summer, where it's too warm to wear a coat and too cold to wear nothing), and it's perfectly comfortable most of the time. But I rarely wear it over a hoodie, it's normally just with a top and cardigan.