Answer: (i) \(\operatorname{rank}(A)=2\), and a basis for the null space is \(\left\{\begin{pmatrix}2\\2\\-1\\0\end{pmatrix},\begin{pmatrix}1\\3\\0\\1\end{pmatrix}\right\}\).
(ii) \(\mathbf{x}=\begin{pmatrix}p+2\lambda+\mu\\q+2\lambda+3\mu\\-\lambda\\\mu\end{pmatrix}\).
(iii) \(p=2\), \(q=-1\).
(iv) \(\mathbf{x}=\begin{pmatrix}4\\9\\1\\4\end{pmatrix}\).
Row-reducing \(A\) gives an echelon form with two non-zero rows, for example
\(\begin{pmatrix}1&-1&0&2\\0&1&2&-3\\0&0&0&0\\0&0&0&0\end{pmatrix}\).
Therefore \(\operatorname{rank}(A)=2\).
To find the null space, solve \(A\mathbf{x}=\mathbf{0}\). From the echelon form, with \(\mathbf{x}=(x,y,z,t)^T\),
\(x-y+2t=0\), and \(y+2z-3t=0\).
Let \(z=\lambda\) and \(t=\mu\). Then
\(y=-2\lambda+3\mu\), and \(x=y-2\mu=-2\lambda+\mu\).
Thus
\(\mathbf{x}=\lambda\begin{pmatrix}-2\\-2\\1\\0\end{pmatrix}+\mu\begin{pmatrix}1\\3\\0\\1\end{pmatrix}\).
Replacing the first vector by its negative gives the equivalent basis
\(\left\{\begin{pmatrix}2\\2\\-1\\0\end{pmatrix},\begin{pmatrix}1\\3\\0\\1\end{pmatrix}\right\}\).
The first two columns of \(A\) are \(\begin{pmatrix}1\\3\\5\\-2\end{pmatrix}\) and \(\begin{pmatrix}-1\\-1\\-8\\3\end{pmatrix}\). Hence
\(A\begin{pmatrix}p\\q\\0\\0\end{pmatrix}=p\begin{pmatrix}1\\3\\5\\-2\end{pmatrix}+q\begin{pmatrix}-1\\-1\\-8\\3\end{pmatrix}\).
Therefore the general solution is a particular solution plus the null space:
\(\mathbf{x}=\begin{pmatrix}p\\q\\0\\0\end{pmatrix}+\lambda\begin{pmatrix}2\\2\\-1\\0\end{pmatrix}+\mu\begin{pmatrix}1\\3\\0\\1\end{pmatrix}\).
So
\(\mathbf{x}=\begin{pmatrix}p+2\lambda+\mu\\q+2\lambda+3\mu\\-\lambda\\\mu\end{pmatrix}\).
Now solve
\(p\begin{pmatrix}1\\3\\5\\-2\end{pmatrix}+q\begin{pmatrix}-1\\-1\\-8\\3\end{pmatrix}=\begin{pmatrix}3\\7\\18\\-7\end{pmatrix}\).
From the first two components,
\(p-q=3\), and \(3p-q=7\).
Subtracting gives \(2p=4\), so \(p=2\). Then \(2-q=3\), so \(q=-1\).
For \(A\mathbf{x}=\begin{pmatrix}3\\7\\18\\-7\end{pmatrix}\), use \(p=2\) and \(q=-1\). Then
\(\mathbf{x}=\begin{pmatrix}2+2\lambda+\mu\\-1+2\lambda+3\mu\\-\lambda\\\mu\end{pmatrix}\).
We need the first two components to be \(4\) and \(9\), so
\(2\lambda+\mu=2\), and \(2\lambda+3\mu=10\).
Subtracting gives \(2\mu=8\), so \(\mu=4\). Then \(2\lambda+4=2\), so \(\lambda=-1\).
Therefore
\(\mathbf{x}=\begin{pmatrix}4\\9\\1\\4\end{pmatrix}\).