Yes, it's wrong; meaningful celibacy is generally referring to sexual activity in general - though it depends. If it's religious celibacy then certainly you should be abstaining.
Surely you have a reason why you have decided that you want to be celibate. If this is the case then you should just do what you feel comfortable with. If you want to abstain from all sexual activity then that is fine. If you feel that you only want to stop sex or sexual contact with other people that is also ok. Its your body and you just need to think about what is best for you.