There are downsides to disaffiliating. At my undergrad uni, even as part of the NUS, a Conservative contingent mobilised amongst a largely politically apathetic student body and took over. As a result, my uni's SU voted *in favour* of tuition fee increases and the uni couldn't send an official group to the related protests. If you'd asked individual students to vote, there's no way they would've voted in favour.
Disaffiliation might amplify this effect. For disaffiliation or affiliation to work in favour of the majority and result in a truly representative SU, you need a politicised student body which is willing to participate, at least in SU elections.
I've seen disaffiliation work and affiliation fail, so there's no "one size fits all" answer.