Answer: (a) The system does not have a unique solution when

\(a=\frac{3}{16}(1\pm \sqrt{17})\).

(b) \(\mathbf{B}^2=p\mathbf{I}+q\mathbf{A}\) with

\(p=\frac{7}{9},\quad q=-\frac{2}{9}.\)

So

\(\mathbf{B}^2=\frac{7}{9}\mathbf{I}-\frac{2}{9}\mathbf{A}.\)

(c) One suitable choice is

\(\mathbf{P}=\begin{pmatrix}1&2&2\\0&1&1\\0&0&-1\end{pmatrix},\quad \mathbf{D}=\begin{pmatrix}\frac{2}{3}&0&0\\0&\frac{1}{3}&0\\0&0&-1\end{pmatrix}.\)

Then \(\mathbf{A}^{-1}=\mathbf{PDP}^{-1}\).

(a) A system of three linear equations does not have a unique solution when the determinant of its coefficient matrix is zero.

The coefficient matrix is

\(\begin{pmatrix}\frac32&3&8\\a&3&4\\0&a&-1\end{pmatrix}.\)

So

\(\det\begin{pmatrix}\frac32&3&8\\a&3&4\\0&a&-1\end{pmatrix}=0.\)

Expanding along the first row,

\(\det=\frac32\begin{vmatrix}3&4\\a&-1\end{vmatrix}-3\begin{vmatrix}a&4\\0&-1\end{vmatrix}+8\begin{vmatrix}a&3\\0&a\end{vmatrix}.\)

This gives

\(\det=\frac32(-3-4a)-3(-a)+8a^2.\)

Hence

\(\det=-\frac92-6a+3a+8a^2=8a^2-3a-\frac92.\)

Therefore

\(8a^2-3a-\frac92=0.\)

Multiplying by 2,

\(16a^2-6a-9=0.\)

Using the quadratic formula,

\(a=\frac{6\pm \sqrt{36+576}}{32}=\frac{6\pm 6\sqrt{17}}{32}=\frac{3}{16}(1\pm \sqrt{17}).\)

(b) Since \(\mathbf{A}\) is upper triangular, its eigenvalues are \(\frac32,\ 3,\ -1\).

So the characteristic equation is

\((\lambda-\frac32)(\lambda-3)(\lambda+1)=0.\)

Expanding,

\(\lambda^3-\frac72\lambda^2+\frac92=0.\)

By the Cayley–Hamilton theorem,

\(\mathbf{A}^3-\frac72\mathbf{A}^2+\frac92\mathbf{I}=\mathbf{0}.\)

So

\(\frac92\mathbf{I}=\frac72\mathbf{A}^2-\mathbf{A}^3.\)

Multiply by \(\mathbf{A}^{-2}=\mathbf{B}^2\):

\(\frac92\mathbf{B}^2=\frac72\mathbf{I}-\mathbf{A}.\)

Hence

\(\mathbf{B}^2=\frac79\mathbf{I}-\frac29\mathbf{A}.\)

Therefore \(p=\frac79\) and \(q=-\frac29\).

(c) To diagonalise \(\mathbf{A}^{-1}\), use eigenvectors of \(\mathbf{A}\).

For \(\lambda=\frac32\):

\((\mathbf{A}-\frac32\mathbf{I})\mathbf{x}=\mathbf{0}\) gives

\(\begin{pmatrix}0&3&8\\0&\frac32&4\\0&0&-\frac52\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\mathbf{0}.\)

From the third row, \(z=0\); then from the second row, \(y=0\). So one eigenvector is

\(\begin{pmatrix}1\\0\\0\end{pmatrix}.\)

For \(\lambda=3\):

\((\mathbf{A}-3\mathbf{I})\mathbf{x}=\mathbf{0}\) gives

\(\begin{pmatrix}-\frac32&3&8\\0&0&4\\0&0&-4\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\mathbf{0}.\)

So \(z=0\), and then \(-\frac32x+3y=0\Rightarrow y=\frac12x\). Take \(x=2\), giving the eigenvector

\(\begin{pmatrix}2\\1\\0\end{pmatrix}.\)

For \(\lambda=-1\):

\((\mathbf{A}+\mathbf{I})\mathbf{x}=\mathbf{0}\) gives

\(\begin{pmatrix}\frac52&3&8\\0&4&4\\0&0&0\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\mathbf{0}.\)

From the second row, \(y=-z\). Choose \(z=-1\), so \(y=1\). Then the first row gives

\(\frac52x+3-8=0\Rightarrow \frac52x=5\Rightarrow x=2.\)

So one eigenvector is

\(\begin{pmatrix}2\\1\\-1\end{pmatrix}.\)

Hence take

\(\mathbf{P}=\begin{pmatrix}1&2&2\\0&1&1\\0&0&-1\end{pmatrix}.\)

The eigenvalues of \(\mathbf{A}^{-1}\) are the reciprocals of those of \(\mathbf{A}\), namely

\(\frac23,\ \frac13,\ -1.\)

So

\(\mathbf{D}=\begin{pmatrix}\frac23&0&0\\0&\frac13&0\\0&0&-1\end{pmatrix}.\)

Therefore

\(\mathbf{A}^{-1}=\mathbf{PDP}^{-1}.\)