Hi,
I think it's normal to have these concerns. As someone who will start medicine in September, this is something that I considered very seriously. Since I am going to medicine as a mature student, an understanding that there is essentially no easy job out there helped me. Of course medicine would be on a different level as you have the reposibilty of people's lives but I am in academia right now and things are really tough. You regularly do unpaid work late into evenings and the weekends. In order to progress as a lecturer or a prof involved securing extremely hard to find grants, constantly worrying about publications (which is not directly related to how hard you work). My friends are in banking and while they make a decent living, their working hours are cruel.
I think at the end of the day, as well as taking your passion into consideration, you should also take into account the sacrifices you will be okay to make in making something out of your life. And remembering that work life is not easy and every job will have really stressful times.