I have made an exactly similar decision quite a while ago, here's my reasoning.
I enjoyed physics because I loved maths, and physics was, to me, the best way you can use mathematics. It's intellectually interesting, and most importantly challenging. Computer science has a lot of these qualities too, it's an application of mathematics, it's definitely challenging and interesting, but I found physics problems much, much more fun.
Here's a piece of advice, if you want to know whether or not you will like physics, do the physics olympiad papers. They're hard and challenging and will make you think. After doing them you will definitely feel brainstunned, but you will also leave with one of two sentiments, you will either be extremely hooked to the subject and this kind of problems and thinking, or you won't. If not, I suggest you go with computer science as the prospects for computer science students is usually better than that for physicists.
Whatever your decision may be, rest assured the two fields are similarly challenging and interesting in their own rights. You just have to find what lights your spark
. Also, remember a very, very important thing. Computer science is as much about programming as chemistry is about mass spectrometry or physics is about building telescopes. Some physics courses do more programming than computer science *Computational physics @ Edinburgh". Computer science is more about the theory of computation, how can you calculate things efficiently, how to find employable algorithms which can solve different problems or families of problems, how to organise data, how to protect data, and so on.
Physics is also more about solving equations, thinking about physical situations and trying to model them mathematically, finding shortcuts and if you go deep enough, physics becomes more of a sea of mathematics that's just so close to being absolutely pure and abstract, than it is about electrons popping in and out of existence or dark matter or black holes.