From you saying that you've fainted in other situations I would think that the anxiety comes from you feeling like you're about to faint rather that you fainting from anxiety.
Fainting usually occurs when blood pressure drops, but when people are anxious and panicky their blood pressure usually rises.
I'd advise going to the doctor about it and seeing if there is anything that could help stop you fainting, as an intense phobia of fainting may eventually develop and increase your anxiety even more.
When you feel the signs of fainting coming on (eugh they're horrible aren't they) try to calmly sit down and make yourself as cool as possible. Hopefully if you sort of show yourself that it can be handled calmly with no-one fussing and embarrassing you, the anxiety about going out will decrease.