You want lines which do not pass through the origin, i.e. c is non-zero.
But your result you ended up with has to be satisfies for all x. So if c is non-zero, what must m be?
Put this into the coeff of x ... what must lambda be?
for part a, I have formulated the equation : 0= -2x(m^2 + m + 0.5Lamda ) -2c ( m+1) , however I'm not sure what to do now
Question is asking for "no invariant lines through the origin".
So, suppose there is an invariant line through the origin.
To start with c will be zero.
Then what about the multiplier for x?
(It's not totally obvious, but have a think)
Note: There is a slicker method of doing this, and also the method you're using doesn't cater for the line x=0, where m would be infinite, and would thus need to be treated separately.
Edit: Since RDKGames is suggesting something different, I suggest going with his method, and you can come back to this one later if you so wish. I'll leave it spoilered for ref.
Solve for m from coeff of x.
m is not necessarily -1 since it depends on lambda for x coeff to be 0.
So c MUST be zero for an invariant line ...
Ok, so all invariant lines pass through the origin if they exist ... the only thing we can do to ensure there are none passing through origin is to ensure there are no invariant lines at all.
So ... look at m in terms of lambda and decide how you can ensure there are no invariant lines.
A reflection matrix has an infinite number of invariant lines that don't pass through the origin.
To be clear, lambda=0 gives infinitely many invariant lines which do not pass through the origin with gradient -1. One of them does (so I suppose this value of lambda can be neglected). For any other lambda c must be zero.
But I am trying to get OP to think about the case of no invariant lines at all since this is an alternative yielding no lines through origin.