(a) To find the cosine of angle BAC, we first find the vectors \(\overrightarrow{AB}\) and \(\overrightarrow{AC}\):

\(\overrightarrow{AB} = \overrightarrow{OB} - \overrightarrow{OA} = \begin{pmatrix} 3 \\ 1 \\ 2 \end{pmatrix} - \begin{pmatrix} 1 \\ 3 \\ 1 \end{pmatrix} = \begin{pmatrix} 2 \\ -2 \\ 1 \end{pmatrix}\)

\(\overrightarrow{AC} = \overrightarrow{OC} - \overrightarrow{OA} = \begin{pmatrix} 5 \\ 3 \\ -2 \end{pmatrix} - \begin{pmatrix} 1 \\ 3 \\ 1 \end{pmatrix} = \begin{pmatrix} 4 \\ 0 \\ -3 \end{pmatrix}\)

The scalar product \(\overrightarrow{AB} \cdot \overrightarrow{AC} = 2 \times 4 + (-2) \times 0 + 1 \times (-3) = 8 - 3 = 5\).

The magnitudes are \(|\overrightarrow{AB}| = \sqrt{2^2 + (-2)^2 + 1^2} = \sqrt{9} = 3\) and \(|\overrightarrow{AC}| = \sqrt{4^2 + 0^2 + (-3)^2} = \sqrt{25} = 5\).

Thus, \(\cos \angle BAC = \frac{\overrightarrow{AB} \cdot \overrightarrow{AC}}{|\overrightarrow{AB}| |\overrightarrow{AC}|} = \frac{5}{3 \times 5} = \frac{1}{3}\).

(b) To find the area of triangle ABC, we use the formula \(\frac{1}{2} |\overrightarrow{AB} \times \overrightarrow{AC}|\).

The cross product \(\overrightarrow{AB} \times \overrightarrow{AC} = \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ 2 & -2 & 1 \\ 4 & 0 & -3 \end{vmatrix} = \mathbf{i}((-2)(-3) - 1(0)) - \mathbf{j}(2(-3) - 1(4)) + \mathbf{k}(2(0) - (-2)(4))\).

\(= \mathbf{i}(6) - \mathbf{j}(-6 - 4) + \mathbf{k}(8) = 6\mathbf{i} + 10\mathbf{j} + 8\mathbf{k}\).

The magnitude \(|\overrightarrow{AB} \times \overrightarrow{AC}| = \sqrt{6^2 + 10^2 + 8^2} = \sqrt{36 + 100 + 64} = \sqrt{200} = 10\sqrt{2}\).

Thus, the area is \(\frac{1}{2} \times 10\sqrt{2} = 5\sqrt{2}\).