Aw don't let your insecurities get in the way. In a way it'd be selfish to do that because you're putting your feelings over his. Give him a chance. The amount you like someone can grow overtime, so whilst you may not be head over heels now, give yourself some time!
You should let him know your concerns though. It'll make you feel much better and it'll help establish where you both stand in each others lives.
Good luck, you'll be great!