It depends.
Vegans and vegetarians lack in many essential minerals such as zinc and magnesium as they do not eat meat, they also lack in healthy fats most of the time. Therefore you will have to supplement these vitamins and minerals and eat plenty of nuts/olive oil to replace the meat.
Vegetarians and vegans have been shown to have lower levels of testosterone in general, id say eating a bit of meat everyday will not be harmful. Just eat chicken or turkey and limit red meat.