Not a big deal. It takes me ages to remember a new phone number. The one I've got right now... I've had for about two years and I still get it wrong sometimes.
When someone wants my number, I usually just give them my phone and tell them to call themselves. Much easier for everyone.