On a road with no parked cars and empty or quiet pavements, I will usually overtake a cyclist at whatever speed I was doing before. However, if this places me over the centre white line, I will slow down a little, and I will make sure there are no large vehicles or motorcycles coming in the opposite direction (and if possible, that there are no cyclists being overtaken on that side of the road as well).
On a road with parked cars or busy pavements, however, I will wait until I can see that I can overtake the cyclist safely, and if that means crawling 5 miles along the same road at 15mph, so be it. When there are parked cars, you need to leave sufficient space that if someone suddenly opens a car door and the cyclist has to swerve, you can still swerve around them. Same with busy pavements - there may be small children or dogs, or just idiots who step into the road.
OP, never ever overtake a cyclist because a driver behind gets annoyed. If it happens every time you're stuck behind a cyclist, maybe see if an instructor can give you a couple of sessions in a cyclist-heavy area? But if it's once or twice, even if you feel flustereed every time, do NOT give in to the pressure from other drivers, because it will end badly, and "he used his horn so I thought I had to go" won't be helpful in an insurance claim!
I've never had this pressure overtaking cyclists before, but I've had temporary traffic lights placed so that the traffic in the opposite dicrection usually back up BEYOND the entrance when it was rush hour, and I had to sit through THREE green lights once before it was clear AND green and I could go. So many people were tooting their horns, but if I had gone, I would have caused a crash, or blocked up the traffic in both directions, because there was no way that traffic from the other direction would see me if I went on my red light (their green) until it was too late! As proven by the two idiots who overtook me at the same lights a couple of days later (well into the red light, when they could have seen the oncoming traffic very clearly) and mercifully avoided a crash but ending up reversing very slowly to numerous car horns from either direction of traffic - if there had been a police officer present, no doubt they would have been taken aside!