I haven't read through the posts so I might be saying the same thing someone else has already said and in that case, my apologies.
I personally have taken both in HL and here's my opinion.
Chemistry – each topic is different. Organic Chemistry is used in almost all topics, but except that, you won't lose any marks in a topic cuz you didn't do well in another. For example, you won't lose any mark in Energetics if you don't know anything about Stoichiometry or Acids or Bases or whatever.
In extra material for HL, you don't learn "new topic", but you just go into more detail and solving problems can get "trickier" due to that. Difficulty wise it's not that difficult, but it can get pretty "tricky" to get the right answer.
In a nutshell, chemistry is easy if you take it as SL, but becomes very tricky (not hard, but tricky) when taken as HL.
Physics – even if you take it as SL, it's hard – and tricky. When you take it as HL, you don't necessarily go much more in depth, but you rather learn more material. So if you're gonna take a SL, I would say chem would be much more easier, but if you're taking HL, both would be almost equally challenging.
If you're gonna be doing either of them, here are some hints that I wish I knew earlier.
For both subjects, ALWAYS check the syllabus. The syllabus CLEARLY outlines what EXACTLY you need to know for the IB. There will be – I repeat – there WILL be materials that your teacher provide you that's not necessarily in the syllabus, but helps you to understand what IS in the syllabus. But for exam purposes, you can discard them. So please check with the syllabus for both physics and chemistry. If the syllabus says you need to "define" a term, you should be able to. On the other hand, if the syllabus doesn't say you need to define it, you don't need to regardless how important that word/concept might be. If the syllabus says "explain", you really gotta know your **** about that etc. The syllabus is pretty much useless for most subjects, but for natural sciences, I think they're extremely helpful. (And for social sciences as well, to some degree)
2nd tip: for physics, you really need to know what each symbols represent. Even if it's the same alphabet, say F, it could mean elastic force (F=kx), gravitational force (F=Gmm/r^2), electric force (F=kqq/r^2) or F=BILsinx or F=qvBsinx etc... so you really MUST know which is which.
3rd tip: for physics, you MUST know about topic 4: waves and oscillation (topic 11 also if you're HL), topic 5: electric currents (topic 12 also for HL) and topic 7: atomic and nuclear physics (topic 13 also for HL). All the other topics are based on these 3 topics (6 for HL), so if you can't do these, you're basically ****ed – which I was and had to work my ass off to catch up before the IB.
I hope this helps. I'm getting my result tomorrow and it's quite amusing to see new people entering the IB. Best of luck to ya.