Answer:

Row-reducing \(M\) gives

\(\begin{pmatrix}1&-2&-3&1\\3&-5&-7&7\\5&-9&-13&9\\7&-13&-19&11\end{pmatrix}\sim\begin{pmatrix}1&-2&-3&1\\0&1&2&4\\0&0&0&0\\0&0&0&0\end{pmatrix}.\)

There are two non-zero rows, so

\(\operatorname{rank}(M)=2.\)

For the null space, let \(\mathbf{x}=\begin{pmatrix}x\\y\\z\\t\end{pmatrix}\). The reduced system is

\(x-2y-3z+t=0,\qquad y+2z+4t=0.\)

Let \(z=\lambda\) and \(t=\mu\). Then

\(y=-2\lambda-4\mu,\qquad x=-\lambda-9\mu.\)

Thus

\(\mathbf{x}=\lambda\begin{pmatrix}-1\\-2\\1\\0\end{pmatrix}+\mu\begin{pmatrix}-9\\-4\\0\\1\end{pmatrix}.\)

Hence the two displayed vectors form a basis for the null space.

Since \(M\mathbf{x}=M\mathbf{e}\), we need \(\mathbf{x}-\mathbf{e}\) to be in the null space. Therefore

\(\mathbf{x}=\mathbf{e}+\lambda\begin{pmatrix}-1\\-2\\1\\0\end{pmatrix}+\mu\begin{pmatrix}-9\\-4\\0\\1\end{pmatrix}.\)

With \(\mathbf{e}=\begin{pmatrix}1\\2\\3\\4\end{pmatrix}\), require the third and fourth entries to be \(-1\):

\(3+\lambda=-1\Rightarrow \lambda=-4,\qquad 4+\mu=-1\Rightarrow \mu=-5.\)

So

\(a=1-(-4)-9(-5)=50,\qquad b=2-2(-4)-4(-5)=30.\)