Civil/Uniformed Services Offer Dilemma

Hello,
I've recently accepted a provisional offer of employment at a public service pending onboarding and vetting (16-24 weeks then until the rest of my cohort can start). In the past couple of days I've received an unexpected interview invite to a civil service department, which I wasn't expecting due to a feeling of impostors syndrome. I'm wondering if my interview goes well and I feel like this department is the right fit for me. Would I be able to withdraw my acceptance of my first offer? I did ask about rights of withdrawal during my interview at my interview for the first offer and how long it would take. I just wonder if it would be okay to withdraw or reconsider my previous offer. If any of you guys have been in a similar situation let me know how it turned out/how you decided.
Thanks and all the best to you all.