(a) The transformation is a stretch parallel to the x-axis with scale factor k, represented by \(\begin{pmatrix} k & 0 \\ 0 & 1 \end{pmatrix}\), followed by a shear with (0, 1) mapped to (k, 1), represented by \(\begin{pmatrix} 1 & k \\ 0 & 1 \end{pmatrix}\). The combined transformation matrix M is:

\(\begin{pmatrix} 1 & k \\ 0 & 1 \end{pmatrix} \begin{pmatrix} k & 0 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} k & k \\ 0 & 1 \end{pmatrix}\)

(b) The line of invariant points satisfies \(\begin{pmatrix} k & k \\ 0 & 1 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x \\ y \end{pmatrix}\). This gives:

\(kx + ky = x\) and \(y = y\)

Solving \(kx + ky = x\) gives \(y = \frac{1-k}{k}x\).

(c) The inverse of M is:

\(\text{If } \text{det}(\textbf{M}) = k, \text{ then } \textbf{M}^{-1} = \frac{1}{k} \begin{pmatrix} 1 & -k \\ 0 & k \end{pmatrix}\)

(d) Given \(|k| = 3k^2\), solving gives \(k = \pm \frac{1}{3}\).