Answer: The rank of \(\mathbf{M}\) is \(2\), and a basis for the null space \(K\) is \(\{(-5,-2,1,0)^T,(7,3,0,1)^T\}\).
Also, \(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\), so every solution of \(\mathbf{Mx}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\) has the form \(\mathbf{x}=\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}+\lambda \mathbf{e}_1+\mu \mathbf{e}_2\), where \(\{\mathbf{e}_1,\mathbf{e}_2\}\) is any basis for \(K\).
We row-reduce \(\mathbf{M}\):
\(\begin{pmatrix}1&-3&-1&2\\4&-10&0&2\\1&-1&3&-4\\5&-12&1&1\end{pmatrix}\)
Use the first row as a pivot row.
\(R_2\leftarrow R_2-4R_1\), \(R_3\leftarrow R_3-R_1\), \(R_4\leftarrow R_4-5R_1\):
\(\begin{pmatrix}1&-3&-1&2\\0&2&4&-6\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\)
Now divide the second row by \(2\):
\(\begin{pmatrix}1&-3&-1&2\\0&1&2&-3\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\)
Then eliminate below the pivot in column 2:
\(R_3\leftarrow R_3-2R_2\), \(R_4\leftarrow R_4-3R_2\), giving
\(\begin{pmatrix}1&-3&-1&2\\0&1&2&-3\\0&0&0&0\\0&0&0&0\end{pmatrix}\)
There are two non-zero rows, so \(\operatorname{rank}(\mathbf{M})=2\).
To find the null space, solve \(\mathbf{Mx}=\mathbf{0}\). From the echelon form:
\(x-3y-z+2t=0\)
\(y+2z-3t=0\)
Let \(z=\lambda\) and \(t=\mu\). Then
\(y=3\mu-2\lambda\)
and
\(x=3y+z-2t=3(3\mu-2\lambda)+\lambda-2\mu=7\mu-5\lambda\).
So
\(\mathbf{x}=\lambda\begin{pmatrix}-5\\-2\\1\\0\end{pmatrix}+\mu\begin{pmatrix}7\\3\\0\\1\end{pmatrix}.\)
Hence a basis for \(K\) is \(\{(-5,-2,1,0)^T,(7,3,0,1)^T\}\).
Now evaluate the given product:
\(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}1&-3&-1&2\\4&-10&0&2\\1&-1&3&-4\\5&-12&1&1\end{pmatrix}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}.\)
Therefore the vector \(\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}\) is one particular solution of \(\mathbf{Mx}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\).
If \(\mathbf{x}\) is any other solution, then
\(\mathbf{M}\left(\mathbf{x}-\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}\right)=\mathbf{Mx}-\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\mathbf{0},\)
so \(\mathbf{x}-\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}\in K\).
Since \(K\) has basis \(\{\mathbf{e}_1,\mathbf{e}_2\}\), every vector in \(K\) can be written as \(\lambda \mathbf{e}_1+\mu \mathbf{e}_2\). Hence every solution has the form
\(\mathbf{x}=\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}+\lambda \mathbf{e}_1+\mu \mathbf{e}_2.\)