Immediately after leaving the big 4 audit firms, I did a couple of year long contracts with two fund administration companies as a fund accountant (permanent was an option, but contracting pays better). I then moved into work as a trustee with a relatively large (top 20) global bank's custody business (using audit skills, along with knowledge of the particular collective investment schemes I was assigned to) which I did for two and a half years. During this time I completed the ICSA professional exams and became a chartered secretary. After that I moved to Australia for two and a half years joining my current employer (top US investment bank) where I held roles in governance and controls. Recently, I returned to the UK where I govern the projects implementing regulatory change. I plan to do this for a few years whilst completing some legal study (undecided as to what that will be!) before I look at another international move.
What I would say is that there is life after the big 4, certainly if you still want to work in finance, but don't underestimate the usefulness of having completed a big 4 training contract. Had I not failed that one exam and just worked that bit harder, I think I would have much better job opportunities now. Everyone I know who finished their training contracts, including a couple whose work ethic didn't come close to matching their exam performance, are in more senior positions today than I am. As I said, it took about four years to reach the point that I could have been at in just one more year in the big 4.
However, the original poster doesn't seem to share that interest (in which case I would say leave. The sooner you can start building a career in something else, the better. Don't get stuck in a career you hate, it gets harder to leave when your salary starts to increase).