If you were able to stick to that amount for the rest of your life, then yeah, the health effect would probably be negligible (other than maybe a cough/more regular colds). But that's a big if. I used to be the same. I went multiple years on 1-2 a day, if that. Going a week or two without smoking wasn't particularly difficult.
But then you go to a festival. It's ok, it's a festival! Normal rules don't apply! You don't normally smoke 5 in a day, but hey, other people are, and it passes the time while you're waiting for acts etc.
But you're fine, and within a week or two, you're back to your usual amount. Maybe a touch more, but it's barely noticeable.
Next year, numbers go up again. This time, you smoke a bit more than 1-2 a day afterwards, but hey, you're just coming off all that nicotine, you shouldn't go cold turkey.
Then something hard happens. You smoke a bit more. You have to, to cope. Then you have that piece of work to do. So stressful. Better have a cigarette to relax.
Now you're bored with friends. Have a cigarette. Anxious/bored at a party. Cigarette.
Maybe you try a cigar, or an ecig, with more nicotine than you're used to. Hey, this is nicer than cigarettes.
More stress.
Cigarette.
More work.
Cigarette.
More boredom.
Cigarette.
Before you know it, you're up to 5-10 a day. But you can't quit like you used to be able to. Going without makes you constantly crave it. You find yourself deeply inhaling whenever you smell cigarette smoke.
This post was typed with a cigarette in hand. A few years ago, I was like you. You may think it won't happen to you. You're different. You'll be able to resist. You'll keep your smoking occasional. Others do, why can't you?
It can happen. Easily. And it sneaks up on you. Be careful.