Answer: The rank of \(\mathbf{M}\) is \(2\).

A basis for the null space \(K\) is \(\left\{\begin{pmatrix}-5\\-2\\1\\0\end{pmatrix},\begin{pmatrix}7\\3\\0\\1\end{pmatrix}\right\}\).

Also, \(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\).

Hence every solution of \(\mathbf{M}\mathbf{x}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\) has the form \(\mathbf{x}=\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}+\lambda \mathbf{e}_1+\mu \mathbf{e}_2\), where \(\{\mathbf{e}_1,\mathbf{e}_2\}\) is a basis for \(K\).

We row-reduce \(\mathbf{M}\):

\(\mathbf{M}=\begin{pmatrix}1&-3&-1&2\\4&-10&0&2\\1&-1&3&-4\\5&-12&1&1\end{pmatrix}\).

Subtract suitable multiples of the first row from rows 2, 3 and 4:

\(\begin{pmatrix}1&-3&-1&2\\4&-10&0&2\\1&-1&3&-4\\5&-12&1&1\end{pmatrix}\to\begin{pmatrix}1&-3&-1&2\\0&2&4&-6\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\).

Now divide the second row by \(2\):

\(\begin{pmatrix}1&-3&-1&2\\0&2&4&-6\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\to\begin{pmatrix}1&-3&-1&2\\0&1&2&-3\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\).

Then eliminate below the pivot in column 2:

\(\begin{pmatrix}1&-3&-1&2\\0&1&2&-3\\0&2&4&-6\\0&3&6&-9\end{pmatrix}\to\begin{pmatrix}1&-3&-1&2\\0&1&2&-3\\0&0&0&0\\0&0&0&0\end{pmatrix}\).

There are two non-zero rows, so \(\operatorname{rank}(\mathbf{M})=2\).

To find the null space, solve \(\mathbf{M}\mathbf{x}=\mathbf{0}\). From the echelon form, if \(\mathbf{x}=\begin{pmatrix}x\\y\\z\\t\end{pmatrix}\), then

\(x-3y-z+2t=0\), \(y+2z-3t=0\).

Let \(z=\lambda\) and \(t=\mu\). Then from \(y+2z-3t=0\),

\(y=3\mu-2\lambda\).

Substitute into \(x-3y-z+2t=0\):

\(x=3y+z-2t=3(3\mu-2\lambda)+\lambda-2\mu=7\mu-5\lambda\).

So

\(\mathbf{x}=\lambda\begin{pmatrix}-5\\-2\\1\\0\end{pmatrix}+\mu\begin{pmatrix}7\\3\\0\\1\end{pmatrix}\).

Therefore a basis for \(K\) is \(\left\{\begin{pmatrix}-5\\-2\\1\\0\end{pmatrix},\begin{pmatrix}7\\3\\0\\1\end{pmatrix}\right\}\).

Now evaluate \(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}\):

First component: \(1-3(-2)-(-3)+2(-4)=1+6+3-8=2\).

Second component: \(4-10(-2)+0(-3)+2(-4)=4+20-8=16\).

Third component: \(1-(-2)+3(-3)-4(-4)=1+2-9+16=10\).

Fourth component: \(5-12(-2)+(-3)+(-4)=5+24-3-4=22\).

Hence

\(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\).

Since \(\mathbf{M}\mathbf{x}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\) and \(\mathbf{M}\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}=\begin{pmatrix}2\\16\\10\\22\end{pmatrix}\), the difference between any solution \(\mathbf{x}\) and \(\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}\) lies in the null space \(K\). Therefore every solution can be written as

\(\mathbf{x}=\begin{pmatrix}1\\-2\\-3\\-4\end{pmatrix}+\lambda\mathbf{e}_1+\mu\mathbf{e}_2\),

where \(\{\mathbf{e}_1,\mathbf{e}_2\}\) is any basis for \(K\).