I found S1 and M1 to be sort of closely matched in terms of difficulty...but I found S1 to be a little bit harder, ever so slightly. Doing physics helps slightly with M1, and the concepts in S1 are a little tricky to get the hang of at first. I found S1 to be quite boring, it didn't really capture my interest, whereas I greatly enjoyed doing M1 and M2, my two favourite modules in the whole of A level maths.
I'd say D1 is the hardest module in A level maths, hands down (not counting further maths of course). It's completely different from all the rest. It's not impossibly difficult, but core maths (C1 to 4) is a joke compared to decision maths. I didn't like the style of the D1 paper.