Even the smaller universities have thousands of students so even though you may see this person around occasionally, it is very unlikely that you will see them enough or have to speak to them to make it a nuisance.
Also, starting university is a scary experience so he will probably be too nervous about making new friends and getting used to the place to be fussed about bothering you. If he ever does bother you, you just have to remember that you are the bigger person and ignore them and not retaliate.