I have been with one of the Big4 for over 10 years now, seen many graduates come and go, don't worry.
As with the issue of division, it's much easier to move when I joined and I have moved to other divisions once I fully qualified, I know it's more rigid nowadays but still possible.
Since you haven't been to the other firm, get a feel from the interview with the partner, no need to make up your mind yet, I hope you didn't tell them you have signed a contract with one of their major competitors. Good luck.