It is not a question of difficulty, but luck. I have come to despise HL english because the assessment criteria is so subjective; how well you do is more-or-less completely dependent on the mood of your examiner at the time your paper is marked.
I did HL English for IB1 but have dropped it for next year for two reasons. First, because it's my 4th higher, I don't need for university and it's eating up my time unnecessarily. Second, because even though I'm predicted a 7, statistically it's just not going to happen (my school got one 7 last year).