I've never understood the need to label yourself tbh.
Dunno if it's just me, but I find that stuff like this rarely comes up in conversation. But I find that because I don't drink alcohol for numerous reasons, (medication, can't have most alcohol anyway due to wheat intolerance, etc) that tends to come up in coversation a lot more. Aside from that conversation with that militant veggie who told me I shouldn't eat meat and should be veggie. She couldn't give me a reason as to why I should limit my diet more than I already need to. (wheat intolerant; so can't eat a lot of stuff anyway)
You shouldn't need to justify yourself. Me, I'd just say that I don't like the texture or something. I don't eat certain meats for that reason.