As a preamble, I should say I think things have rather gone off the rails in this thread - with various different methods being suggested, most of which are beyond the material you've covered so far. That said, you're actually pretty close to getting the question out with what you've done, but there's a fair bit of inefficiency and wrong directions.
There are two simplifications that you can make here (and you can often make when talking about distance with matrix/vector questions):
Firstly, once you start doing detailed calculations, you nearly always want to be thinking about distance^2, not distance. So in general, the only point you'd take the square root is at the end if you need the actual distance. But to just compare distances, find when the distance is minimized, etc. you would just work with distance^2.
Secondly, you're wanting to show that
∣Mp−Mq∣=∣p−q∣. But
Mp−Mq=M(p−q), so it's actually enough to show
∣Mp∣=∣p∣. To put it another way, you can replace
x1−x2 and
y1−y2 by
x,y (where
P=(xy)).
So the expression we care about is
∣Mp∣2=a2x2+2abxy+b2y2+c2x2+2cdxy+d2y2 (compare with the expression inside the square root above).
If we group the x^2 terms, xy terms and y^2 terms we get:
∣Mp∣2=(a2+c2)x2+2(ab+cd)xy+(b2+d2)Now note that you observed:
MTM=(a2+c2ab+cdab+cdb2+d2)So if
MTM=I, then we must have
a2+c2=b2+d2=1,ab+cd=0.
And then
∣Mp∣2=x2+y2=∣p∣2 as desired.
Note that this method *is* exactly the "viewing matrices as individual entries and evaluating everything in terms of them." approach that mqb said not to do.
Although he's right that there are better methods (that don't involve much algebra and work for matrices of arbitrary size), it's fairly clear you haven't really covered the material for them yet. And the algebra grind here isn't that bad, and I think it's pretty clear it's what you were supposed to do here.
When you feel more confident with matrix transposes, (and in particular, the result that
(AB)T=BTAT), if you want to come back to this thread we can talk about the two more conceptual approaches that don't involve explicit calculation.