Nasty Man Confronted me for Using Disabled Toilet (I have Crohns)

It sucks, it really does but you wouldn't have the key if you didn't need it!

Hey, Okay so I have crohns disease and I basically have a need to use toilets frequently and I have never had this before and it has really upset me to be honest.

I went out to my local pub, a spoons pub actually and I know there is a really good campaign they are doing to promote not every disability is visible but this one does not have the sticker yet on the door.

Anyway as I was in there I heard a knock and I said "Just a minute" I wasnt sure who it was usually its staff wanting to clean but whoever it was then tried to open the door with the key (Luckily I locked the door properly and yes I have my own key too) and I washed my hands and when I opened the door there was an old man outside (He has a reputation for being pretty nasty in this pub towards people) I held the door open for him and said "there you go" as usually I close it after and as i walked past he said "Are you disabled because this toilet is for disabled people only" in such a nasty tone and it threw me i turned round and thought should I say something but I just smiled, shrugged and continued walking.

I feel I should have said something but I thought why is it his business how or what illness or needs I have, My friends told me and even my family said I should have said something and put him in his place rather than ignore him.

I guess it shocked and upset me because I find it difficult with the condition and to be confronted like that wasn't nice. And I feel I didn't handle it well as a mate told me my "silence" could be seen as guilt for using it which wasn't the case it all.

Anybody else has a similar experience and how they handle it? I think next time I will simply repeat what the sign says "not all disabilities are visible" and continue walking.