Answer: (a) The coefficient matrix is \(\begin{pmatrix}1&-1&2\\1&-1&-3\\1&-1&7\end{pmatrix}\). Its determinant is

\(\begin{vmatrix}1&-1&2\\1&-1&-3\\1&-1&7\end{vmatrix}=1\begin{vmatrix}-1&-3\\-1&7\end{vmatrix}-(-1)\begin{vmatrix}1&-3\\1&7\end{vmatrix}+2\begin{vmatrix}1&-1\\1&-1\end{vmatrix}=-10+10+0=0.\)

So the system does not have a unique solution.

(b) When \(a=-5\), solving gives \(z=\frac{9}{5}\) and \(x-y=\frac{2}{5}\). Hence there are infinitely many solutions, given by \(x-y=\frac{2}{5}\) and \(z=\frac{9}{5}\).

Geometrically, the three planes all contain the same line.

(c) If \(a\ne -5\), the first and third equations still give \(z=\frac{9}{5}\) and then \(x-y=\frac{2}{5}\). Substituting into the second equation gives \(a=\frac{2}{5}-3\cdot\frac{9}{5}=-5\), which contradicts \(a\ne -5\).

So the system is inconsistent.

Geometrically, the three planes have no common point; each pair meets in a line, and these lines are parallel, so the planes form a triangular prism arrangement.

(a) A system of three linear equations has a unique solution only if the determinant of its coefficient matrix is non-zero.

Here the coefficient matrix is

\(A=\begin{pmatrix}1&-1&2\\1&-1&-3\\1&-1&7\end{pmatrix}.\)

Now calculate \(\det(A)\):

\(\det(A)=\begin{vmatrix}1&-1&2\\1&-1&-3\\1&-1&7\end{vmatrix}.\)

Expanding along the first row,

\(\det(A)=1\begin{vmatrix}-1&-3\\-1&7\end{vmatrix}-(-1)\begin{vmatrix}1&-3\\1&7\end{vmatrix}+2\begin{vmatrix}1&-1\\1&-1\end{vmatrix}.\)

So

\(\det(A)=1((-1)(7)-(-3)(-1))+1((1)(7)-(-3)(1))+2((1)(-1)-(-1)(1)).\)

That is

\(\det(A)=(-7-3)+(7+3)+0=-10+10=0.\)

Therefore the determinant is zero, so the system does not have a unique solution.

(b) Take \(a=-5\). The system becomes

\(x-y+2z=4\)

\(x-y-3z=-5\)

\(x-y+7z=13.\)

Subtract the second equation from the first:

\((x-y+2z)-(x-y-3z)=4-(-5)\)

\(5z=9\)

\(z=\frac{9}{5}.\)

Substitute into the first equation:

\(x-y+2\left(\frac{9}{5}\right)=4\)

\(x-y+\frac{18}{5}=\frac{20}{5}\)

\(x-y=\frac{2}{5}.\)

Check in the third equation:

\(x-y+7z=\frac{2}{5}+7\left(\frac{9}{5}\right)=\frac{2}{5}+\frac{63}{5}=\frac{65}{5}=13,\)

so the third equation is also satisfied.

Hence the system is consistent.

The common solutions satisfy

\(x-y=\frac{2}{5},\qquad z=\frac{9}{5}.\)

There is one free variable, so there are infinitely many solutions. For example, if \(y=t\), then

\(x=t+\frac{2}{5},\qquad y=t,\qquad z=\frac{9}{5}.\)

Geometrically, each equation represents a plane, and all three planes share the same line.

(c) Now suppose \(a\ne -5\). Use the first and third equations:

\((x-y+7z)-(x-y+2z)=13-4\)

\(5z=9\)

\(z=\frac{9}{5}.\)

Then from the first equation,

\(x-y=4-2z=4-2\left(\frac{9}{5}\right)=\frac{20}{5}-\frac{18}{5}=\frac{2}{5}.\)

If the second equation is also to be satisfied, then

\(a=x-y-3z=\frac{2}{5}-3\left(\frac{9}{5}\right)=\frac{2}{5}-\frac{27}{5}=-5.\)

But this contradicts \(a\ne -5\).

Therefore the system is inconsistent when \(a\ne -5\).

Geometrically, the three planes do not have a common point. Each pair intersects in a line, and these three lines are parallel, so the planes form a triangular prism arrangement.