Answer: (a) One suitable choice is

\(\mathbf{P}=\begin{pmatrix}1&1&2\\0&2&1\\0&0&2\end{pmatrix}\), \(\mathbf{D}=\begin{pmatrix}-\frac16&0&0\\0&-\frac12&0\\0&0&\frac18\end{pmatrix}\),

so that \(\mathbf{A}^{-1}=\mathbf{PDP}^{-1}\).

(b) \(\mathbf{A}^{-1}=\begin{pmatrix}-\frac16&-\frac16&\frac38\\0&-\frac12&\frac5{16}\\0&0&\frac18\end{pmatrix}\).

(a) Since \(\mathbf{A}\) is upper triangular, its eigenvalues are the diagonal entries:

\(\lambda=-6,\,-2,\,8\).

To diagonalise \(\mathbf{A}^{-1}\), we use the same eigenvectors as for \(\mathbf{A}\), with reciprocal eigenvalues.

For \(\lambda=-6\), solve \((\mathbf{A}+6\mathbf{I})\mathbf{x}=\mathbf{0}\):

\(\begin{pmatrix}0&2&13\\0&4&5\\0&0&14\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}0\\0\\0\end{pmatrix}.\)

This gives \(z=0\), then \(y=0\), so an eigenvector is \(\begin{pmatrix}1\\0\\0\end{pmatrix}\).

For \(\lambda=-2\), solve \((\mathbf{A}+2\mathbf{I})\mathbf{x}=\mathbf{0}\):

\(\begin{pmatrix}-4&2&13\\0&0&5\\0&0&10\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}0\\0\\0\end{pmatrix}.\)

This gives \(z=0\) and then \(-4x+2y=0\), so \(y=2x\). One eigenvector is \(\begin{pmatrix}1\\2\\0\end{pmatrix}\).

For \(\lambda=8\), solve \((\mathbf{A}-8\mathbf{I})\mathbf{x}=\mathbf{0}\):

\(\begin{pmatrix}-14&2&13\\0&-10&5\\0&0&0\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}0\\0\\0\end{pmatrix}.\)

From \(-10y+5z=0\), we get \(z=2y\). Substituting into the first equation:

\(-14x+2y+13(2y)=0\Rightarrow -14x+28y=0\Rightarrow x=2y\).

Taking \(y=1\), an eigenvector is \(\begin{pmatrix}2\\1\\2\end{pmatrix}\).

Hence we may take

\(\mathbf{P}=\begin{pmatrix}1&1&2\\0&2&1\\0&0&2\end{pmatrix}.\)

The corresponding eigenvalues of \(\mathbf{A}^{-1}\) are \(-\frac16\), \(-\frac12\), \(\frac18\), so

\(\mathbf{D}=\begin{pmatrix}-\frac16&0&0\\0&-\frac12&0\\0&0&\frac18\end{pmatrix}.\)

Therefore \(\mathbf{A}^{-1}=\mathbf{PDP}^{-1}\).

(b) The characteristic equation of \(\mathbf{A}\) is

\((\lambda+6)(\lambda+2)(\lambda-8)=0.\)

Expanding:

\((\lambda+6)(\lambda+2)=\lambda^2+8\lambda+12\), so

\((\lambda^2+8\lambda+12)(\lambda-8)=\lambda^3-52\lambda-96.\)

Hence, by Cayley–Hamilton,

\(\mathbf{A}^3-52\mathbf{A}-96\mathbf{I}=\mathbf{0}.\)

Since \(\mathbf{A}\) is invertible, multiply by \(\mathbf{A}^{-1}\):

\(\mathbf{A}^2-52\mathbf{I}-96\mathbf{A}^{-1}=\mathbf{0},\)

so

\(96\mathbf{A}^{-1}=\mathbf{A}^2-52\mathbf{I}.\)

Now

\(\mathbf{A}^2=\begin{pmatrix}36&-16&36\\0&4&30\\0&0&64\end{pmatrix}.\)

Therefore

\(\mathbf{A}^2-52\mathbf{I}=\begin{pmatrix}-16&-16&36\\0&-48&30\\0&0&12\end{pmatrix}.\)

So

\(\mathbf{A}^{-1}=\frac1{96}\begin{pmatrix}-16&-16&36\\0&-48&30\\0&0&12\end{pmatrix}=\begin{pmatrix}-\frac16&-\frac16&\frac38\\0&-\frac12&\frac5{16}\\0&0&\frac18\end{pmatrix}.\)