(a) The matrix \(M\) is a product of two matrices: \(\begin{pmatrix} 1 & k \\ 0 & 1 \end{pmatrix}\) represents a shear in the x-direction, and \(\begin{pmatrix} \frac{1}{2}\sqrt{2} & -\frac{1}{2}\sqrt{2} \\ \frac{1}{2} & \frac{1}{2}\sqrt{2} \end{pmatrix}\) represents a rotation anticlockwise about the origin through 45°.

(b) To find \(M^{-1}\), we calculate the inverse of the product of the two matrices: \(M^{-1} = \begin{pmatrix} \frac{k+1}{\sqrt{2}} & \frac{k-1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{-1}{\sqrt{2}} \end{pmatrix}\).

(c) For no invariant lines through the origin, the characteristic equation of \(M\) must have no real roots. The characteristic equation is derived from \(\det(M - \lambda I) = 0\). Solving gives \(k^2 + 4k - 4 < 0\), leading to the solution \(2(-1-\sqrt{2}) < k < 2(-1+\sqrt{2})\).