Answer: The system has no unique solution when the determinant of the coefficient matrix is zero.

The coefficient matrix is \(\begin{pmatrix}1 & -1 & 2\\ 1 & a & -3\\ 1 & -1 & 7\end{pmatrix}\).

Subtract row 1 from rows 2 and 3:

\(\begin{pmatrix}1 & -1 & 2\\ 0 & a+1 & -5\\ 0 & 0 & 5\end{pmatrix}\)

so \(\det = 1\cdot (a+1)\cdot 5 = 5(a+1)\).

For no unique solution, \(5(a+1)=0\), hence \(a=-1\).

(i) With \(a=-1\) and \(b=-5\), the system is

\(x-y+2z=4\), \(x-y-3z=-5\), \(x-y+7z=13\).

Subtract the first equation from the second:

\(-5z=-9\), so \(z=\frac95\).

Then from \(x-y+2z=4\),

\(x-y+\frac{18}{5}=4\Rightarrow x-y=\frac25\).

Let \(y=t\). Then \(x=t+\frac25\), \(z=\frac95\).

So the general solution is

\(\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}\frac25\\0\\\frac95\end{pmatrix}+t\begin{pmatrix}1\\1\\0\end{pmatrix}\).

(ii) If \(b\neq -5\), there is no common point of intersection. Geometrically, the three planes form a prism.

For the system to fail to have a unique solution, the coefficient matrix must be singular.

\(A=\begin{pmatrix}1 & -1 & 2\\ 1 & a & -3\\ 1 & -1 & 7\end{pmatrix}.\)

Compute \(\det A\) by row reduction:

\(R_2\leftarrow R_2-R_1\), \(R_3\leftarrow R_3-R_1\), giving

\(\begin{pmatrix}1 & -1 & 2\\ 0 & a+1 & -5\\ 0 & 0 & 5\end{pmatrix}.\)

Hence

\(\det A = 1\cdot (a+1)\cdot 5 = 5(a+1).\)

So

\(5(a+1)=0\Rightarrow a=-1.\)

(i) General solution when \(b=-5\)

Substitute \(a=-1\), \(b=-5\):

\(\begin{aligned}x-y+2z&=4,\\x-y-3z&=-5,\\x-y+7z&=13.\end{aligned}\)

Subtract the first equation from the second:

\(-5z=-9\Rightarrow z=\frac95.\)

Then use the first equation:

\(x-y+2\left(\frac95\right)=4\Rightarrow x-y=\frac25.\)

Let \(y=t\). Then

\(x=t+\frac25,\qquad z=\frac95.\)

Therefore

\(\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}\frac25\\0\\\frac95\end{pmatrix}+t\begin{pmatrix}1\\1\\0\end{pmatrix},\qquad t\in\mathbb R.\)

(ii) Geometrical interpretation when \(b\neq -5\)

With \(a=-1\), the first and second planes have the same normal vector, since their left-hand sides are both \(x-y+\text{(multiple of }z\text{)}\) with the same \(x\) and \(y\) coefficients structure leading to no common simultaneous solution when the constants differ. For \(b\neq -5\), the three planes do not all meet in a single point.

Geometrically, the three planes form a prism.