For me, that move would annoy me due to it taking money out of the UK when we need to improve the circulation of money to get out economy going properly. Switzerland aren't exactly struggling. Generally tax evasion annoys me more when it's politicians. They completely abuse the position they are in, and abuse the financial benefits of the job, avoiding tax on earnings because they 'need a 2nd home in my constituency'. I could go on about this but I'm probably drifting off topic.
Isn't it obvious why it annoys people? Because most people who are on a pay-as-you-earn taxed salary arrangement have no choice but to pay their taxes - only the rich and large corporations are able to afford complicated tax avoidance advice and arrangements. Similarly, many small businesses have to pay their taxes in full, unable to exploit loopholes.
Yes, these arrangements are legal but that doesn't make them morally correct. Large companies based here in the UK who pay almost no tax here due to "arrangements" they make are abusing their presence - they benefit in all kinds of ways from the state here, including in simply having a stable, calm, defended economy to operate in, yet their tax sheltering trickery means they avoid their responsibilities back to that state. Classic examples are Vodafone, which pretends all its profits are earned in Luxembourg (quite bizarre that the UK government allows them to get away with this) and Topshop, which has been "transferred" to Phil Green's wife, who "lives" in Monaco.
The big e-tailers like Amazon, Ebay, etc, despite earning billions here, are also paying vanishingly small amounts of tax in the UK due to adroit placing of their corporate bases in Luxembourg.
What really needs to happen is closure of the tax havens like Luxembourg, Switzerland, Monaco, etc. It has been calculated that shutting down the European tax havens would lead to salary earners getting a 7p in the £ reduction in income tax, whilst at the same time boosting spending on the NHS, education and defence by 25%. We would all benefit from such a position.