I took chemistry HL for the same reason you're considering it, and to be honest, I hate it; but it isn't overwhelmingly difficult. If you're willing to put enough work into it, it'll be fine. Also, if your teacher chooses the human biochemistry option (C, I think) you'll have a nice overlap with biology.
Oh, and the syllabi are absolutely essential in the sciences. The biology syllabus especially. The syllabi and the Oxford Study Guides are what I'm relying on to get through this.
I'm doing math, physics and chemistry at HL (bio at SL). I think physics is by far the hardest science. If you're good at biology you should be good at chemistry, as most of the course is just memorising, like biology. You only really need to do basic calculations. Chemistry for me at least is the easiest higher level, and biology at HL is meant to be easier than chemistry so it can definitely be done.
I'm taking Biology, Chemistry and Psychology at higher level, and like everyone else says: it's all about having a good study technique! Seriously. I personally consider Chem to be far more difficult than Bio. Bio is all about memorising. And well, some common sense when it comes to logic (I freakin HATE the data base questions, they always bring me down a grade-level on every test)