I would hope I could swim, I am a Lifeguard. Trust me, swimming with clothes on is a lot more tiring than swimming in a pool in ordinary swimming gear. Every month I go and train and I have to wear a t-shirt and shorts on top of my swimsuit and even that is tiring, especially doing the swim test (2 and half lengths under 60 seconds (20 meter pool)) or practicing towing people.
Most people (even strong swimmers) would struggle falling into a lake with ordinary clothes on, not to mention the addition of shock from the cold and sudden change.