For me, from my experience, I would say it's hard. Not the action of driving itself (i.e. being able to move the car, shift gears etc), but being safe around other drivers and having really good awareness of cars around you.
Passing the test is easy. I considered myself a bit bad at driving but breezed through my practical with just 1 or 2 minors, no problems. But I started driving literally just 4 days ago, and it has been a very scary experience. I'm really terrified. I keep getting beeped at. One guy stopped in front of me and swore at me (and he had every right). The problem is that you have to be absolutely and completely flawless. You really can't afford to make even a single mistake. It could make your insurance premium rocket sky high. Or it could result in someone dying.
That's what's hard about driving. The safety and awareness aspect. Especially when you drive with lots of traffic around on unfamiliar roads with plenty of lanes and lots and lots of cars, large complex roundabouts, complex road junctions, exits, slip roads...it's hell. My first driving experience is in a really big city, my route takes me across some really big, scary busy roundabouts. You're going to absolutely hate it for a few days at least. Hell, I'm even stalling my car so much.