I took ages to pass and wasn't that good for a while. Don't worry.
My instructor let me take my time learning, because I have to really know stuff before I'm brave enough to try something new. He understood that which was cool, it sounds a bit like yours is the same. Although he's telling you what most people do, he seems aware that everyone's different.
I didn't really enjoy driving until a few months after I passed, and even now I still often don't like it. I look at it mainly as a method of getting around, and while I miss my car now I'm at uni, it's mainly because I'd forgotten how horrible walking everywhere is. I get nervous in situations where there's a lot of traffic, but deal with it. The one thing I positively enjoy is situations like motorway driving, where it's not too crowded and you don't have to worry too much about where exactly you're going.
PM me if you want any tips, but don't give up and it will get better.