If you really enjoy Physics, it's not that difficult really. Would strongly recommend taking Mathematics if you are taking Physics. The two subjects slot very nicely into each other, and doing Maths makes the maths involved in Physics a walk in the park, so you can devote more time to studying the principles, information and skills you'll need to learn.
Though if you want to get into engineering, you'll probably know already how essential Maths is as well as Physics :P