(a) The matrix \(\mathbf{M} = \begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}\) represents a rotation followed by a shear. The rotation is anticlockwise, centered at the origin, through an angle \(\theta\), with the x-axis fixed and \((0,1)\) mapped to \((2,1)\).

(b) The combined matrix is \(\mathbf{M} = \begin{pmatrix} \cos \theta + 2\sin \theta & 2\cos \theta - \sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}\).

\(To find the line of invariant points, solve:\)

\(\begin{pmatrix} \cos \theta + 2\sin \theta & 2\cos \theta - \sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x \\ y \end{pmatrix}\)

This gives the equations:

\(x\sin \theta + y\cos \theta = y\)

\(x\cos \theta + 2x\sin \theta - y\sin \theta + 2y\cos \theta = x\)

Rearranging and simplifying, we find:

\(x\cos \theta + 2x\sin \theta + \frac{x\sin \theta}{1 - \cos \theta}(2\cos \theta - \sin \theta) = x\)

\((\cos \theta + 2\sin \theta)(1 - \cos \theta) + 2\sin \theta\cos \theta - \sin^2 \theta = 1 - \cos \theta\)

\(\cos^2 \theta - \cos^2 \theta + 2\sin \theta - \sin^2 \theta = 1 - \cos \theta\)

\(\Rightarrow \sin \theta + \cos \theta = 1\)

Solving gives \(\theta = \frac{1}{2} \pi\).