Answer: (i) \rank\,\mathbf{M}=2.

(ii) A basis for the null space is \(\left\{\begin{pmatrix}-2\\3\\3\\0\end{pmatrix},\begin{pmatrix}1\\-3\\0\\3\end{pmatrix}\right\}\).

(iii) The general solution of \(\mathbf{M}\mathbf{x}=\begin{pmatrix}2\\5\\8\\-2\end{pmatrix}\) is \(\mathbf{x}=\begin{pmatrix}0\\1\\0\\0\end{pmatrix}+\lambda\begin{pmatrix}-2\\3\\3\\0\end{pmatrix}+\mu\begin{pmatrix}1\\-3\\0\\3\end{pmatrix}\), where \(\lambda,\mu\in\mathbb{R}\).

(i) We row-reduce \(\mathbf{M}\):

\(\begin{pmatrix}3&2&0&1\\6&5&-1&3\\9&8&-2&5\\-3&-2&0&-1\end{pmatrix}\)

Use the first row to eliminate entries below the leading 3:

\(R_2\to R_2-2R_1,\; R_3\to R_3-3R_1,\; R_4\to R_4+R_1\), giving

\(\begin{pmatrix}3&2&0&1\\0&1&-1&1\\0&2&-2&2\\0&0&0&0\end{pmatrix}\).

Now \(R_3\to R_3-2R_2\):

\(\begin{pmatrix}3&2&0&1\\0&1&-1&1\\0&0&0&0\\0&0&0&0\end{pmatrix}\).

There are 2 non-zero rows, so \(\operatorname{rank}(\mathbf{M})=2\).

(ii) Let \(\mathbf{x}=(x,y,z,t)^T\). For the null space, solve \(\mathbf{M}\mathbf{x}=\mathbf{0}\), which after row reduction gives

\(3x+2y+t=0,\qquad y-z+t=0\).

Take \(z=\lambda\) and \(t=\mu\). Then from \(y-z+t=0\),

\(y=\lambda-\mu\).

Substitute into \(3x+2y+t=0\):

\(3x+2(\lambda-\mu)+\mu=0\), so \(3x=-2\lambda+\mu\), hence

\(x=-\frac{2}{3}\lambda+\frac{1}{3}\mu\).

Therefore

\(\mathbf{x}=\lambda\begin{pmatrix}-\frac23\\1\\1\\0\end{pmatrix}+\mu\begin{pmatrix}\frac13\\-1\\0\\1\end{pmatrix}\).

Multiplying each vector by 3 gives a simpler basis:

\(K=\operatorname{span}\left\{\begin{pmatrix}-2\\3\\3\\0\end{pmatrix},\begin{pmatrix}1\\-3\\0\\3\end{pmatrix}\right\}.\)

(iii) We look for one particular solution of \(\mathbf{M}\mathbf{x}=\begin{pmatrix}2\\5\\8\\-2\end{pmatrix}\). Try \(\mathbf{x}=\begin{pmatrix}0\\1\\0\\0\end{pmatrix}\):

\(\mathbf{M}\begin{pmatrix}0\\1\\0\\0\end{pmatrix}=\begin{pmatrix}2\\5\\8\\-2\end{pmatrix}\).

So \(\begin{pmatrix}0\\1\\0\\0\end{pmatrix}\) is a particular solution. The general solution is this particular solution plus any vector from the null space:

\(\mathbf{x}=\begin{pmatrix}0\\1\\0\\0\end{pmatrix}+\lambda\begin{pmatrix}-2\\3\\3\\0\end{pmatrix}+\mu\begin{pmatrix}1\\-3\\0\\3\end{pmatrix},\quad \lambda,\mu\in\mathbb{R}.\)