If someone treats you badly, they are not worth your effort or time. Do what you want to do and what is best for you. If you want to keep in contact with him, have you told him how you feel? If he seems to not care then I would say leave it. Actions speak louder than words - if he ignores you a lot but still tries to say he cares, then most likely he doesn't care that much.