I mean I personally love all of those items but when you say 'playing it safe on a date' it depends on other factors such as where you go for the date and how well you know your date. I would say play it smart casual, so if your waistcoat is formal maybe skip it for the first date as you wouldn't want to act as if you're trying too hard. White socks are always usually a good choice for most things I think? (I don't really pay attention to socks to be fair!)
If you're going somewhere reasonably expensive or 'fancy' then maybe avoid the rugby shirt and opt for something like a nice t shirt or shirt. If you're just going out and walking around a park or something more 'casual' then sure, go for the rugby shirt if you feel its right!
Ultimately as you say, wear something you feel comfortable in, although theres no harm adjusting your style depending on the situation. If they really don't like your style and that affects your date tremendously then they aren't cut out for you anyways. Hope this helps?