Answer: (a) One suitable choice is

\(\mathbf{P}=\begin{pmatrix}1&2&2\\0&3&0\\0&0&-1\end{pmatrix}\), \(\mathbf{D}=\begin{pmatrix}25&0&0\\0&36&0\\0&0&1\end{pmatrix}\),

so \(\mathbf{A}^2=\mathbf{PDP}^{-1}\).

(b) \(\mathbf{A}^3=31\mathbf{A}-30\mathbf{I}=\begin{pmatrix}125&-\frac{682}{3}&248\\0&-216&0\\0&0&1\end{pmatrix}.\)

(a) Since \(\mathbf{A}\) is upper triangular, its eigenvalues are the diagonal entries:

\(\lambda=5,-6,1\).

To diagonalise \(\mathbf{A}^2\), use the eigenvectors of \(\mathbf{A}\). If \(\mathbf{A}\mathbf{v}=\lambda\mathbf{v}\), then \(\mathbf{A}^2\mathbf{v}=\lambda^2\mathbf{v}\).

For \(\lambda=5\):

\(\mathbf{A}-5\mathbf{I}=\begin{pmatrix}0&-\frac{22}{3}&8\\0&-11&0\\0&0&-4\end{pmatrix}\).

So \(-11y=0\Rightarrow y=0\) and \(-4z=0\Rightarrow z=0\). Thus an eigenvector is \(\begin{pmatrix}1\\0\\0\end{pmatrix}\).

For \(\lambda=-6\):

\(\mathbf{A}+6\mathbf{I}=\begin{pmatrix}11&-\frac{22}{3}&8\\0&0&0\\0&0&7\end{pmatrix}\).

Then \(7z=0\Rightarrow z=0\), and \(11x-\frac{22}{3}y=0\), so \(33x=22y\), hence \(3x=2y\). Taking \(x=2, y=3\), an eigenvector is \(\begin{pmatrix}2\\3\\0\end{pmatrix}\).

For \(\lambda=1\):

\(\mathbf{A}-\mathbf{I}=\begin{pmatrix}4&-\frac{22}{3}&8\\0&-7&0\\0&0&0\end{pmatrix}\).

Then \(-7y=0\Rightarrow y=0\), and \(4x+8z=0\Rightarrow x=-2z\). Taking \(z=-1\), an eigenvector is \(\begin{pmatrix}2\\0\\-1\end{pmatrix}\).

Using these as columns, one suitable matrix is

\(\mathbf{P}=\begin{pmatrix}1&2&2\\0&3&0\\0&0&-1\end{pmatrix}.\)

For \(\mathbf{A}^2\), the corresponding eigenvalues are \(25,36,1\), so

\(\mathbf{D}=\begin{pmatrix}25&0&0\\0&36&0\\0&0&1\end{pmatrix}.\)

Hence \(\mathbf{A}^2=\mathbf{PDP}^{-1}\).

(b) The characteristic equation is

\((\lambda-5)(\lambda+6)(\lambda-1)=0.\)

Expanding gives

\(\lambda^3-31\lambda+30=0.\)

By Cayley-Hamilton,

\(\mathbf{A}^3-31\mathbf{A}+30\mathbf{I}=\mathbf{0}\),

so

\(\mathbf{A}^3=31\mathbf{A}-30\mathbf{I}.\)

Now

\(31\mathbf{A}=\begin{pmatrix}155&-\frac{682}{3}&248\\0&-186&0\\0&0&31\end{pmatrix}\), \(30\mathbf{I}=\begin{pmatrix}30&0&0\\0&30&0\\0&0&30\end{pmatrix}\).

Therefore

\(\mathbf{A}^3=\begin{pmatrix}125&-\frac{682}{3}&248\\0&-216&0\\0&0&1\end{pmatrix}.\)