No, in my opinion, it doesn't matter.
5 years is hardly a big age gap anyway (unless you are underage then, going on personal experience here, I'd be a bit wary of what he was after and that he wasn't looking to take advantage)
Personally it comes down to:
Initial attraction/ sexual chemistry/ a spark (not being superficial here, just realistic)
Personality
Compatibility
Similar attitudes/ looking for the same thing out of the relationship
Age never really factors into the equation for me, well, apart from the upper limit I've got fixed in my mind. If you like him, go for it. Just, as with any new partner, make sure that they want to be with you for the right reasons so you don't get hurt.