Answer: The system has a unique solution when the determinant of the coefficient matrix is non-zero.

For the coefficient matrix \(\begin{pmatrix}1&5&6\\k&2&2\\-3&4&8\end{pmatrix}\),

\(\begin{aligned} \det &= \begin{vmatrix}1&5&6\\k&2&2\\-3&4&8\end{vmatrix} \\ &= 1\begin{vmatrix}2&2\\4&8\end{vmatrix}-5\begin{vmatrix}k&2\\-3&8\end{vmatrix}+6\begin{vmatrix}k&2\\-3&4\end{vmatrix} \\ &= 8-5(8k+6)+6(4k+6) \\ &= 14-16k. \end{aligned}\)

So a unique solution exists when \(14-16k \ne 0\), that is,

\(\boxed{k \ne \frac{7}{8}}\).

Geometrically, the three equations represent three planes, and for \(k \ne \frac{7}{8}\) these three planes intersect in exactly one point.

Write the system as

\(\begin{pmatrix}1&5&6\\k&2&2\\-3&4&8\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}1\\2\\3\end{pmatrix}.\)

A linear system has a unique solution exactly when the coefficient matrix has non-zero determinant.

So we calculate

\(\begin{aligned} \det\begin{pmatrix}1&5&6\\k&2&2\\-3&4&8\end{pmatrix} &=1\begin{vmatrix}2&2\\4&8\end{vmatrix}-5\begin{vmatrix}k&2\\-3&8\end{vmatrix}+6\begin{vmatrix}k&2\\-3&4\end{vmatrix} \\ &=1(16-8)-5(8k+6)+6(4k+6) \\ &=8-40k-30+24k+36 \\ &=14-16k. \end{aligned}\)

For a unique solution,

\(14-16k\ne 0\).

Hence

\(16k\ne 14\), so \(\boxed{k\ne \frac{7}{8}}\).

Geometrically, each equation represents a plane in three-dimensional space. When \(k\ne \frac{7}{8}\), the three planes intersect at exactly one point, giving a single solution \((x,y,z)\).