Privilege is really hard to get your head around because when you have it, you don't always realise.
Take this as an example: I spent a short time in a wheelchair when I broke my leg. I had never realised that when you're in a wheelchair, it's actually really difficult to get around, because not every building has ramps or lifts. I'd never noticed that when people talk to someone in a wheelchair, they talk down to them and assume they are stupid. Wherever I went, people stared at me! It felt really demeaning. I hadn't ever noticed before, though, because I had always had what some people would refer to as the 'privilege' of being able-bodied.
Having privilege basically means you have a few less barriers in your way. I'm not claiming that being in a minority means you can blame the system for your failure, it just means that on a day-to-day basis you have more things to contend with - in a way, you are less equal.
Another couple of examples: if you're not gay, you might not notice that representation of homosexuality in the media is basically non-existent (though that is changing). You wouldn't notice that school sex ed never discusses homosexual sex, only heterosexual sex. You won't ever have anyone tell you, casually, that they think your 'lifestyle' is disgusting, or that they think that you should have less rights due to your sexuality.
If you're not trans*, you never experience the terror of realising your body is betraying you, and you're unlikely to be subjected to the vitriol or hostility many trans* people encounter everyday. If you're not female, you might not notice street harassment, or understand the humiliation and fear it breeds. And it goes both ways, too: if you're not male you may never have considered how our macho culture affects growing young men.
The good news is that as humans we possess both empathy and curiosity. Having privilege doesn't make you blind to others. Nor should it make you the subject of attack or distrust. When we can all consider, empirically and without bias, how our lives differ to the lives of others, then we'll be a big step along the way to equality. Additionally, the groups often referred to in terms of privilege (men/women, gay/straight, cis/trans etc) are all generalisations. Not all men will have more inherent privilege than all women. Everything entwines and interlocks. Generally, though, white, straight, cisgendered, able-bodied men will have more privilege than others who do not fit those criteria. Generally. Not always.