Answer: (a) The system does not have a unique solution when the determinant of its coefficient matrix is zero, so \(a=-\frac{3}{5}\).

(b) One suitable diagonalisation of \(\mathbf{A}^2\) is

\(\mathbf{P}=\left(\begin{array}{ccc}4&0&0\\5&-1&0\\19&1&1\end{array}\right),\qquad \mathbf{D}=\left(\begin{array}{ccc}9&0&0\\0&1&0\\0&0&4\end{array}\right),\)

so \(\mathbf{A}^2=\mathbf{PDP}^{-1}\).

(c) The integer is \(b=-4\), and

\((\mathbf{A}+6\mathbf{I})^2=\mathbf{A}^4(\mathbf{A}-4\mathbf{I})^2.\)

(a) The coefficient matrix of the system is

\(\left(\begin{array}{ccc}3&a&0\\5&-1&0\\1&3&2\end{array}\right).\)

For the system not to have a unique solution, this matrix must be singular, so

\(\det\left(\begin{array}{ccc}3&a&0\\5&-1&0\\1&3&2\end{array}\right)=0.\)

Expanding along the third column,

\(2\det\left(\begin{array}{cc}3&a\\5&-1\end{array}\right)=0.\)

So

\(2(3(-1)-5a)=0\)

\(-6-10a=0\)

\(a=-\frac{3}{5}.\)

(b) Since \(\mathbf{A}\) is lower triangular, its eigenvalues are its diagonal entries:

\(3,-1,2.\)

Hence the eigenvalues of \(\mathbf{A}^2\) are

\(9,1,4.\)

We find eigenvectors of \(\mathbf{A}\), which are also eigenvectors of \(\mathbf{A}^2\).

For \(\lambda=3\):

\((\mathbf{A}-3\mathbf{I})\mathbf{x}=\mathbf{0}\), so

\(\left(\begin{array}{ccc}0&0&0\\5&-4&0\\1&3&-1\end{array}\right)\left(\begin{array}{c}x\\y\\z\end{array}\right)=\left(\begin{array}{c}0\\0\\0\end{array}\right).\)

This gives

\(5x-4y=0\Rightarrow y=\frac54x,\)

\(x+3y-z=0\Rightarrow z=x+3y=\frac{19}{4}x.\)

Choose \(x=4\). Then an eigenvector is

\(\left(\begin{array}{c}4\\5\\19\end{array}\right).\)

For \(\lambda=-1\):

\((\mathbf{A}+\mathbf{I})\mathbf{x}=\mathbf{0}\), so

\(\left(\begin{array}{ccc}4&0&0\\5&0&0\\1&3&3\end{array}\right)\left(\begin{array}{c}x\\y\\z\end{array}\right)=\left(\begin{array}{c}0\\0\\0\end{array}\right).\)

From the first equation, \(x=0\). Then

\(3y+3z=0\Rightarrow y=-z.\)

Choose \(z=1\). Then an eigenvector is

\(\left(\begin{array}{c}0\\-1\\1\end{array}\right).\)

For \(\lambda=2\):

\((\mathbf{A}-2\mathbf{I})\mathbf{x}=\mathbf{0}\), so

\(\left(\begin{array}{ccc}1&0&0\\5&-3&0\\1&3&0\end{array}\right)\left(\begin{array}{c}x\\y\\z\end{array}\right)=\left(\begin{array}{c}0\\0\\0\end{array}\right).\)

The first equation gives \(x=0\), and then the second gives \(y=0\). Choose \(z=1\). Then an eigenvector is

\(\left(\begin{array}{c}0\\0\\1\end{array}\right).\)

So we can take

\(\mathbf{P}=\left(\begin{array}{ccc}4&0&0\\5&-1&0\\19&1&1\end{array}\right).\)

Matching this order of eigenvectors,

\(\mathbf{D}=\left(\begin{array}{ccc}9&0&0\\0&1&0\\0&0&4\end{array}\right).\)

Hence

\(\mathbf{A}^2=\mathbf{PDP}^{-1}.\)

(c) The characteristic equation of \(\mathbf{A}\) is

\((\lambda-3)(\lambda+1)(\lambda-2)=0.\)

Expanding,

\(\lambda^3-4\lambda^2+\lambda+6=0.\)

By Cayley-Hamilton,

\(\mathbf{A}^3-4\mathbf{A}^2+\mathbf{A}+6\mathbf{I}=\mathbf{0}.\)

So

\(\mathbf{A}+6\mathbf{I}=-\mathbf{A}^3+4\mathbf{A}^2.\)

Now square both sides:

\((\mathbf{A}+6\mathbf{I})^2=\left(-\mathbf{A}^3+4\mathbf{A}^2\right)^2.\)

Factorise the right-hand side:

\(-\mathbf{A}^3+4\mathbf{A}^2=\mathbf{A}^2(-\mathbf{A}+4\mathbf{I})=-\mathbf{A}^2(\mathbf{A}-4\mathbf{I}).\)

Therefore

\((\mathbf{A}+6\mathbf{I})^2=\left(-\mathbf{A}^2(\mathbf{A}-4\mathbf{I})\right)^2=\mathbf{A}^4(\mathbf{A}-4\mathbf{I})^2.\)

Comparing with \(\mathbf{A}^4(\mathbf{A}+b\mathbf{I})^2\), we obtain

\(b=-4.\)