You've given a very incomplete and biased comparison, which is to be expected, as you've never worked for a smaller firm.
There is no doubt you get more responsibility, I don't know how many managing directors/finance directors you've spoken to, but I spoke to dozens in just my first three months at a mid-tier firm. Fair enough, managing directors of companies turning over 250k-1mil, but MD's none the less. In 3 years you're expected to be offering business advice(rather than just accountancy advice) to these clients, advice that is value added and that makes a real and noticeable difference. Improving business function of a client turning over 100k saving them 10k makes a far bigger difference than doing the same for a client turning over 1mil.
The variety of work is also just not on the same level, I was expected to do corporation tax comps, prepare accounts, auditing, other ad-hoc duties such as drafting important letters, the odd bit of corporate finance etc. Thats not the case with a big 4 firm. Preparing accounts is also very different to auditing, you have to have a real eye for detail and a very broad knowledge of accounting and tax to do a good job.
As for "The quality of fellow workers. Everyone at a big four firm could have gotten into a smaller firm. The same isn't true the other way around." I don't think you could be very much more wrong. Hiring for smaller firms is actually far more selective due to the fact that there is a small workforce and you're not part of a machine, as such. Hiring is done directly by the partner of the firm and it is his values that are important, HR don't exist, and there are no recruitment policies other than those decided by that partner.
The partner of my firm only recruited trainee accountants with AAA+ in arts subjects or AAB+ in science subjects for a-levels, and he only recruited graduates if they graduated in maths/languages/sciences. Why? No reason, thats just his personal preferences, which already makes it far more selective than somewhere like a big 4 firm. So all of your colleagues who got BBC and did Business Studies at the University of Essex wouldn't have even been given a second glance, where as there is definitely a large amount of those sorts of people working in the big 4.
I don't know why auditing FTSE350 companies is important to you, or anyone, for that matter. How about making a real difference and having some real input into the way somebody's business is run? How about being a business advisor rather than just an auditor? These are options you have at a small firm which you just don't at a large one, at least not for a very long time.
And your comment about pay is incorrect, again, because the firms are small pay is solely performance related(market rates were irrelevant at my firm), and often this is an arbitrary figure calculated by perceived value added. There were newly qualifieds being paid £50k at my firm and the newest partner(who'd only been at the firm for 6 years, incidently) was paid 100k. You do not get that sort of money in that timescale at big 4 firms. If you are good you will almost undoubtedly rise to the top much quicker in a smaller firm, as there is much less beurocracy and your contribution is much more visible.
Lastly, don't kid yourself into thinking the big 4 is a selective organisation to work for. A quick search of their "experienced vacancies" page will show you that they are literally DESPERATE for qualified/part-qualified candidates. You are not a cut above everybody else working in accountancy, despite what you may think. Selection for graduates is only perceived to be competitive because of the very large amounts of rubbish applicants that apply. Having a 10-1 applicant/offer ratio means nothing if 9 of those applicants are trash. As an aside point, if working for the big 4 is that great, why is the turnover almost a shocking 20%, and why do you find many big 4 workers flocking to smaller firms in the hope of more responsibility and variety, and a better work life balance?