(a) The matrix \(\mathbf{M}\) can be decomposed into two transformations: a rotation and a shear. The rotation is represented by \(\begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}\), which is an anticlockwise rotation about the origin through an angle \(\theta\). The shear is represented by \(\begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix}\), which fixes the x-axis and maps \((0,1)\) to \((2,1)\).

(b) The matrix \(\mathbf{M}\) is given by:

\(\mathbf{M} = \begin{pmatrix} \cos \theta + 2 \sin \theta & 2 \cos \theta - \sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}\)

Applying \(\mathbf{M}\) to a point \((x, y)\) gives:

\(\begin{pmatrix} \cos \theta + 2 \sin \theta & 2 \cos \theta - \sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x \cos \theta + 2x \sin \theta - y \sin \theta + 2y \cos \theta \\ x \sin \theta + y \cos \theta \end{pmatrix}\)

\(For a line of invariant points, we require:\)

\(x \sin \theta + y \cos \theta = y\) and \(x \cos \theta + 2x \sin \theta - y \sin \theta + 2y \cos \theta = x\).

Solving these equations gives:

\(x \cos \theta + 2x \sin \theta + \frac{x \sin \theta}{1 - \cos \theta} (2 \cos \theta - \sin \theta) = x\)

Simplifying, we find:

\((\cos \theta + 2 \sin \theta)(1 - \cos \theta) + 2 \sin \theta - \sin^2 \theta = 1 - \cos \theta\)

\(\cos \theta - \cos^2 \theta + 2 \sin^2 \theta - \sin^2 \theta = 1 - \cos \theta \Rightarrow \sin \theta + \cos \theta = 1\)

Thus, \(\theta = \frac{1}{2} \pi\).