I started learning to drive in 2009 (I think) when I was 17 and I hated every second of it.
I was shy anyways and driving felt so unnatural. I failed 4 tests and spent so much money on lessons, I was beginning to think that I'd never pass. Like you, I did manage my theory first time though.
In November 2014 I decided I had to start again for work and whatnot. I'd been away at University, I'd done some travelling (a lot of travelling alone) and I was more relaxed and more confident about life in general than I was at 17. I think this definitely helped a lot.
Looking back, I just don't think I had the confidence in myself at 17 and driving just wasn't a natural instinct to me (I wouldn't look at junctions and I couldn't grasp roundabouts at all). Moving away and doing other things that scared me honestly helped me become a more confident driver. Obviously I'm not suggesting you just move away and it will help, but maybe try doing other things that scare you. It definitely helped with me more than I ever thought it would and if I ever got scared while driving, I'd think of something I was more scared of.
I recently passed my test after taking my time finding a calm, friendly instructor who taught me in the way I understood. I absolutely 100% feel it was worth the wait. I feel more confident now aged 23 than I ever did at 17 and I have no problems jumping in my car and driving wherever I need to. I was the last of ALL of my friends to pass, but I don't feel thats a issue. It really isn't a race! It's surely better that YOU feel confident at the wheel, than having a licence and being terrified to drive. Speaking as someone who's just passed, it's absolutely NOT the end of the world if it takes you a few goes. I took 5 tests (way more than everyone I know) but I feel so much more confident having those extra lessons, and coming across situations other people haven't (snow, driving at night...).
So I guess the point of this post is take all the time you need. So what if it takes you a year, or 2, or more? The more important thing is that you feel confident, comfortable and relaxed at the wheel. Take the time to learn at night, to learn in the winter and to drive in different cities while you have an experienced instructor beside you. I miss the days I could turn to my instructor and ask about things I'm not sure of. Now it's all up to me, which is scary in itself!
It's all about confidence. And if I can do it, you certainly can.