Further to that, the security argument lies on the fact that there's nobody else that we could buy from. Indeed, we'd need to be at war with every single food supplier in order for the argument to hold.
All it would take would be the trade routes (still generally sea) to be blocked by whomever we were at war with. But yes, as the UK as long as we weren't at war with Europe we'd be fine.
Why? It's well documented that minority groups that are visible (in this case,the beneficiaries of protectionism) tend to have a lot more political influence than invisible majorities who are dispersed (in this case, the beneficiaries of free trade).
'Expect' as in 'I expect you to be on your best behaviour', not as in 'I expect United to win the league'.
Food is different. Clothes are much more durable and less necessary. Besides, the tariffs do not serve that purpose for textiles. The EU textiles industry couldn't survive without its imports of raw materials. Furthermore, the EU textile industry does not produce even similar products to imported clothing. The EU produces branded and specialist clothing for industrial purposes. They are not a direct substitute for the cheap everyday clothing which is imported. The tariff essentially just acts as a tax on cheap clothing, more likely to cause people to substitute consumption to other markets than to EU textiles.