Answer: (a) The characteristic equation is \(\lambda^{3}-4\lambda^{2}-20\lambda+48=0\).

Factorising, \((\lambda-2)(\lambda+4)(\lambda-6)=0\), so the eigenvalues are \(2\), \(-4\) and \(6\).

(b) One suitable choice is

\(\mathbf{P}=\left(\begin{array}{ccc}1&1&1\\-1&0&2\\1&2&2\end{array}\right)\), \(\mathbf{D}=\left(\begin{array}{ccc}32&0&0\\0&-1024&0\\0&0&7776\end{array}\right)\).

Then \(\mathbf{A}^{5}=\mathbf{PDP}^{-1}\).

(a) The characteristic equation comes from

\(\det(\mathbf{A}-\lambda \mathbf{I})=0\), so

\(\left|\begin{array}{ccc}18-\lambda&5&-11\\8&6-\lambda&-4\\32&10&-20-\lambda\end{array}\right|=0\).

Expanding along the first row,

\(\det(\mathbf{A}-\lambda \mathbf{I})=(18-\lambda)\left|\begin{array}{cc}6-\lambda&-4\\10&-20-\lambda\end{array}\right|-5\left|\begin{array}{cc}8&-4\\32&-20-\lambda\end{array}\right|-11\left|\begin{array}{cc}8&6-\lambda\\32&10\end{array}\right|\).

Now

\(\left|\begin{array}{cc}6-\lambda&-4\\10&-20-\lambda\end{array}\right|=(6-\lambda)(-20-\lambda)+40=\lambda^2+14\lambda-80\),

\(\left|\begin{array}{cc}8&-4\\32&-20-\lambda\end{array}\right|=8(-20-\lambda)+128=-32-8\lambda\),

\(\left|\begin{array}{cc}8&6-\lambda\\32&10\end{array}\right|=80-32(6-\lambda)=32\lambda-112\).

So

\(\det(\mathbf{A}-\lambda \mathbf{I})=(18-\lambda)(\lambda^2+14\lambda-80)-5(-32-8\lambda)-11(32\lambda-112)\).

Expanding and simplifying gives

\(\det(\mathbf{A}-\lambda \mathbf{I})=-\lambda^3+4\lambda^2+20\lambda-48\).

Hence the characteristic equation is

\(\lambda^3-4\lambda^2-20\lambda+48=0\).

Factorising,

\((\lambda-2)(\lambda+4)(\lambda-6)=0\),

so the eigenvalues are \(2\), \(-4\) and \(6\).

(b) We find eigenvectors corresponding to these eigenvalues.

For \(\lambda=2\), solving \((\mathbf{A}-2\mathbf{I})\mathbf{x}=\mathbf{0}\) gives an eigenvector

\(\mathbf{v}_1=\left(\begin{array}{c}1\\-1\\1\end{array}\right)\).

For \(\lambda=-4\), solving \((\mathbf{A}+4\mathbf{I})\mathbf{x}=\mathbf{0}\) gives an eigenvector

\(\mathbf{v}_2=\left(\begin{array}{c}1\\0\\2\end{array}\right)\).

For \(\lambda=6\), solving \((\mathbf{A}-6\mathbf{I})\mathbf{x}=\mathbf{0}\) gives an eigenvector

\(\mathbf{v}_3=\left(\begin{array}{c}1\\2\\2\end{array}\right)\).

Take these as the columns of \(\mathbf{P}\):

\(\mathbf{P}=\left(\begin{array}{ccc}1&1&1\\-1&0&2\\1&2&2\end{array}\right)\).

Since \(\mathbf{A}^5\) has eigenvalues \(2^5,(-4)^5,6^5\), we have

\(\mathbf{D}=\left(\begin{array}{ccc}32&0&0\\0&-1024&0\\0&0&7776\end{array}\right)\).

Therefore

\(\mathbf{A}^{5}=\mathbf{PDP}^{-1}\).