Firstly, don't have a go at him, especially through text; that'll only be counterproductive.
Next time you meet him, sit him down and nicely explain to him how his actions make you feel. Don't personally attack him or make it out like he's doing something wrong; he's not.
I myself am a terrible texter, I take ages to reply unless I'm in a convo I'm super interested in. Ask him if there are any reasons that are preventing him from wanting to converse with you over text. Perhaps he is very private, and wants to talk about certain things but is afraid to open up. Perhaps he feels he'll be judged. Perhaps he doesn't find the conversations stimulating enough (happens to me all the time).
Whatever it is, try to understand it, and at the same time he should try to understand where you're coming from. Both of you have to be selfless and compassionate enough to care for the other person, rather than just focusing on your own problems (not saying you do that).
I don't think you're being petty, it's a legitimate concern/issue if you're both in a relationship in my opinion.
Good luck.
Oh, and one more thing. Don't take stress over it; very little in life is worth taking stress over. Your mental well-being should take precedence, take solace in the fact that whatever is going to happen will happen, and you can only do and be the best that you can.