Answer:
Eigenvectors may be chosen as
\(\lambda=1:\ \begin{pmatrix}1\\0\\-1\end{pmatrix},\qquad \lambda=3:\ \begin{pmatrix}0\\1\\1\end{pmatrix}.\)
The vector \(\begin{pmatrix}0\\2\\1\end{pmatrix}\) has eigenvalue \(-2\).
One valid diagonalisation is
\(D=\begin{pmatrix}-2&0&0\\0&1&0\\0&0&3\end{pmatrix},\qquad P=\begin{pmatrix}0&1&0\\2&0&1\\1&-1&1\end{pmatrix},\qquad P^{-1}=\begin{pmatrix}-1&1&-1\\1&0&0\\2&-1&2\end{pmatrix}.\)
For \(\lambda=1\), solve \((A-I)\mathbf{x}=0\):
\(A-I=\begin{pmatrix}0&0&0\\10&-8&10\\7&-5&7\end{pmatrix}.\)
If \(\mathbf{x}=\begin{pmatrix}x\\y\\z\end{pmatrix}\), the two non-zero rows give
\(10x-8y+10z=0,\qquad 7x-5y+7z=0.\)
These equations give \(y=0\) and \(x=-z\). A convenient eigenvector is therefore
\(\begin{pmatrix}1\\0\\-1\end{pmatrix}.\)
For \(\lambda=3\),
\(A-3I=\begin{pmatrix}-2&0&0\\10&-10&10\\7&-5&5\end{pmatrix}.\)
The first row gives \(x=0\), and the remaining rows give \(y=z\). Hence a corresponding eigenvector is
\(\begin{pmatrix}0\\1\\1\end{pmatrix}.\)
Now check the given eigenvector:
\(A\begin{pmatrix}0\\2\\1\end{pmatrix}=\begin{pmatrix}0\\-4\\-2\end{pmatrix}=-2\begin{pmatrix}0\\2\\1\end{pmatrix}.\)
So its eigenvalue is \(-2\).
To diagonalise, place the eigenvectors as the columns of \(P\) in the same order as the eigenvalues in \(D\):
\(P=\begin{pmatrix}0&1&0\\2&0&1\\1&-1&1\end{pmatrix},\qquad D=\begin{pmatrix}-2&0&0\\0&1&0\\0&0&3\end{pmatrix}.\)
Then
\(P^{-1}=\begin{pmatrix}-1&1&-1\\1&0&0\\2&-1&2\end{pmatrix},\)
and these matrices satisfy
\(P^{-1}AP=D.\)
