'Fraid I know nothing about this, so can't help much. From a bit of Googling.If this is just matrix multiplication then I agree with your trace of P as
2cos(θ)+cos2(θ) If this now equals
1+2cosϕ (formula for the trace of a 3D rotation matrix) where phi is the angle of rotation, then after rearranging into that form we have:
ϕ=cos−1(cosθ−21sin2θ)As to how you show it's a rotation matrix, I don't know.
If
θ=2πthen
ϕ=cos−1(−21)Which gives two possibiities, 2pi/3 and 4pi/3. Don't know the rationale for choosing one over the other.
And that's about my limit.
Going to need someone with experience of this.