You are homophobic only if you as the recipient of the comment are bigoted towards homosexuals. The users are homophobic because they choose to use it in a derogatory sense.
The usage is rather like swearing at you for instance by calling someone 'tosser' or 'bastard' or 'dick'. By definition >99% of men masturbate, 50% of children are born out of wedlock and (notwithstanding medical problems) all men have a penis. The words still have the power to offend even though the great majority of us can associate with some or all of them.
Taking offence means the perpetrator has achieved their aim and, unfortunately, taking offence continues propagating the term in the bigoted sense but it does not necessarily mean you are homophobic unless you choose to be.