(a) For \(\mathbf{M}\) to represent a rotation, it must be of the form \(\begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}\). Comparing with \(\mathbf{M} = \begin{pmatrix} a & b^2 \\ c^2 & a \end{pmatrix}\), we have \(b^2 = -c^2\), which is impossible since \(b\) and \(c\) are real and \(b \neq 0\).

(b) The transformation of \(\begin{pmatrix} x \\ y \end{pmatrix}\) by \(\mathbf{M}\) gives \(\begin{pmatrix} ax + b^2y \\ c^2x + ay \end{pmatrix}\). For invariant lines through the origin, \(y = mx\) gives \(c^2x + amx = m(ax + b^2y)\). Simplifying, \(c^2 + am = ma + b^2m^2\) leads to \(c^2 = b^2m^2\), giving \(y = \frac{c}{b} x\) and \(y = -\frac{c}{b} x\).

(c) The enlargement matrix is \(\begin{pmatrix} 5 & 0 \\ 0 & 5 \end{pmatrix}\) and the shear matrix is \(\begin{pmatrix} 1 & 5 \\ 0 & 1 \end{pmatrix}\). Thus, \(\mathbf{M} = \begin{pmatrix} 1 & 5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 5 & 0 \\ 0 & 5 \end{pmatrix} = \begin{pmatrix} 5 & 5^2 \\ 0 & 5 \end{pmatrix}\).

(d) The area of \(PQR\) is \(12 \times \det(\mathbf{M})\). \(\det(\mathbf{M}) = 5 \times 5 - 0 \times 25 = 25\). Therefore, the area is \(12 \times 25 = 300 \text{ cm}^2\).