You definitely need to be talking about this. If you can't talk to your partner about any problems you're having with the relationship, what hope does it have in the long term.
You are bothered by it. There is no should or shouldn't - you feel what you feel and thats not right or wrong, but if you want things to change, you need to talk to your boyfriend about whats bothering you and listen to what bothers him and try to come to a solution or compromise.