No way, that sort of age is a great time to start! Much younger than that and you're just too weak (in terms of both issuing pain and receiving it) to really get anywhere. I mean yeah, kids can practice martial arts, but it's usually a karate type affair -- you stand in a line, kick and punch the air and shout "eyuuuuuuuuurgh!" at the top of your pre-pubescent lungs. It's essentially a dance lesson.
Once you're over the 18-ish mark you can start sparring properly. Considering that you said that you're seeking to be adept at defending yourself, sparring is very important, as you (gradually/painstakingly) learn to handle yourself whilst under fire. You could spend your entire life doing kata and pummeling focus pads, but all of this becomes useless when you've got a big drunk bastard windmilling towards you with his sovereigned club-hands flailing wildly.