What youve done is correct, but too long using the KE directly. The post impact v=5,3 solutions are rejected as they are the preimpact velocities and will trivially satisfy both the momentum and KE equations.
For the proof about restitution being conservation of KE (with momentum) the wiki link goes through it, but say you have masses m and M and velocities u, U and v, V, the momentum gives
mu + MU = mv + MV
and rearranging
m(u-v) = M(V-U)
KE gives
1/2 mu^2 + 1/2 MU^2 = 1/2 mv^2 + 1/2 MV^2
so multiplying by 2 and rearranging
m(u^2-v^2) = M(V^2-U^2)
and difference of two squares
m(u-v)(u+v) = M(V-U)(U+V)
Divide by the momentum equation (note you divide out the u=v, U=V "solution")
u+v = U+V
or
u-U = V-v
so the relative velocity before, u-U, equals the relative velocity after, V-v, and a bit of fiddling to define a positive restitution coefficient if KE is not conserved.
You have two linear simultaneous equations rather than a linear and quadratic. It sounds like they expect you to know it.