(i) To find the cosine of angle \(AOB\), we use the dot product formula:

\(\overrightarrow{OA} \cdot \overrightarrow{OB} = 3 \times 6 + 0 \times (-3) + (-4) \times 2 = 18 - 8 = 10\).

The magnitudes are \(|\overrightarrow{OA}| = \sqrt{3^2 + 0^2 + (-4)^2} = 5\) and \(|\overrightarrow{OB}| = \sqrt{6^2 + (-3)^2 + 2^2} = 7\).

Thus, \(\cos \angle AOB = \frac{\overrightarrow{OA} \cdot \overrightarrow{OB}}{|\overrightarrow{OA}| |\overrightarrow{OB}|} = \frac{10}{35} = \frac{2}{7}\).

(ii) The vector \(\overrightarrow{AB} = \overrightarrow{OB} - \overrightarrow{OA} = \begin{pmatrix} 6 \\ -3 \\ 2 \end{pmatrix} - \begin{pmatrix} 3 \\ 0 \\ -4 \end{pmatrix} = \begin{pmatrix} 3 \\ -3 \\ 6 \end{pmatrix}\).

The magnitude of \(\overrightarrow{AB}\) is \(\sqrt{3^2 + (-3)^2 + 6^2} = \sqrt{9 + 9 + 36} = 7\).

Given \(|\overrightarrow{AB}| = |\overrightarrow{OC}|\), we have:

\(\sqrt{k^2 + (-2k)^2 + (2k - 3)^2} = 7\).

Squaring both sides, \(k^2 + 4k^2 + (2k - 3)^2 = 49\).

\(k^2 + 4k^2 + 4k^2 - 12k + 9 = 49\).

\(9k^2 - 12k - 40 = 0\).

Solving the quadratic equation, \(k = 3\) or \(k = -\frac{5}{3}\).

(i) To find when \(\overrightarrow{OA}\) is perpendicular to \(\overrightarrow{OB}\), their dot product must be zero:

\((1)(-7) + (3)(1-p) + (p)(p) = 0\)

\(-7 + 3 - 3p + p^2 = 0\)

\(p^2 - 3p - 4 = 0\)

Factoring gives \((p+1)(p-4) = 0\), so \(p = -1\) or \(p = 4\).

(ii) The magnitudes are \(a = \sqrt{1^2 + 3^2 + p^2}\) and \(b = \sqrt{(-7)^2 + (1-p)^2 + p^2}\).

Given \(b^2 = 2a^2\):

\(49 + (1-p)^2 + p^2 = 2(1 + 9 + p^2)\)

\(49 + 1 - 2p + p^2 + p^2 = 2(10 + p^2)\)

\(50 - 2p + 2p^2 = 20 + 2p^2\)

\(50 - 2p = 20\)

\(2p = 30\)

\(p = 15\).

(iii) When \(p = -8\), \(\overrightarrow{AB} = \overrightarrow{OB} - \overrightarrow{OA} = (-7\mathbf{i} + 9\mathbf{j} + (-8)\mathbf{k}) - (\mathbf{i} + 3\mathbf{j} - 8\mathbf{k})\).

\(\overrightarrow{AB} = -8\mathbf{i} + 6\mathbf{j}\).

The magnitude \(|\overrightarrow{AB}| = \sqrt{(-8)^2 + 6^2} = 10\).

The unit vector is \(\frac{1}{10}(-8\mathbf{i} + 6\mathbf{j})\).

(i) To show that angle OAB is a right angle, we need to show that vectors OA and AB are perpendicular. The position vector of A is 3i + 2j - k and the position vector of B is 7i - 3j + k.

The vector AB is given by:

\(\mathbf{AB} = \mathbf{B} - \mathbf{A} = (7\mathbf{i} - 3\mathbf{j} + \mathbf{k}) - (3\mathbf{i} + 2\mathbf{j} - \mathbf{k}) = 4\mathbf{i} - 5\mathbf{j} + 2\mathbf{k}\)

The dot product \(\mathbf{AO} \cdot \mathbf{AB}\) is:

\((3\mathbf{i} + 2\mathbf{j} - \mathbf{k}) \cdot (4\mathbf{i} - 5\mathbf{j} + 2\mathbf{k}) = 3 \times 4 + 2 \times (-5) + (-1) \times 2 = 12 - 10 - 2 = 0\)

Since the dot product is zero, angle OAB is a right angle.

(ii) To find the area of triangle OAB, we use the formula for the area of a triangle given by vectors:

\(\text{Area} = \frac{1}{2} \times |\mathbf{OA}| \times |\mathbf{AB}|\)

First, calculate the magnitudes:

\(|\mathbf{OA}| = \sqrt{3^2 + 2^2 + (-1)^2} = \sqrt{9 + 4 + 1} = \sqrt{14}\)

\(|\mathbf{AB}| = \sqrt{4^2 + (-5)^2 + 2^2} = \sqrt{16 + 25 + 4} = \sqrt{45}\)

Thus, the area is:

\(\text{Area} = \frac{1}{2} \times \sqrt{14} \times \sqrt{45} = \frac{1}{2} \times \sqrt{630} = 12.5\)

(i) To find angle \(BAC\), we need vectors \(\overrightarrow{AB}\) and \(\overrightarrow{AC}\).

\(\overrightarrow{AB} = \overrightarrow{OB} - \overrightarrow{OA} = \begin{pmatrix} 6 \\ -1 \\ 7 \end{pmatrix} - \begin{pmatrix} 2 \\ 1 \\ 3 \end{pmatrix} = \begin{pmatrix} 4 \\ -2 \\ 4 \end{pmatrix}\)

\(\overrightarrow{AC} = \overrightarrow{OC} - \overrightarrow{OA} = \begin{pmatrix} 2 \\ 4 \\ 7 \end{pmatrix} - \begin{pmatrix} 2 \\ 1 \\ 3 \end{pmatrix} = \begin{pmatrix} 0 \\ 3 \\ 4 \end{pmatrix}\)

The dot product \(\overrightarrow{AB} \cdot \overrightarrow{AC} = 4 \times 0 + (-2) \times 3 + 4 \times 4 = 0 - 6 + 16 = 10\)

The magnitudes are \(|\overrightarrow{AB}| = \sqrt{4^2 + (-2)^2 + 4^2} = \sqrt{36} = 6\)

\(|\overrightarrow{AC}| = \sqrt{0^2 + 3^2 + 4^2} = \sqrt{25} = 5\)

Using the cosine formula: \(\cos BAC = \frac{\overrightarrow{AB} \cdot \overrightarrow{AC}}{|\overrightarrow{AB}| \times |\overrightarrow{AC}|} = \frac{10}{6 \times 5} = \frac{1}{3}\)

Thus, \(BAC = \cos^{-1}\left(\frac{1}{3}\right)\).

(ii) To find the area of triangle \(ABC\), use \(\sin BAC = \sqrt{1 - \left(\frac{1}{3}\right)^2} = \sqrt{\frac{8}{9}} = \frac{\sqrt{8}}{3}\)

Area = \(\frac{1}{2} \times |\overrightarrow{AB}| \times |\overrightarrow{AC}| \times \sin BAC = \frac{1}{2} \times 6 \times 5 \times \frac{\sqrt{8}}{3} = 5\sqrt{8}\)

(i) To find the values of \(p\) for which angle \(AOB\) is 90°, we need \(\overrightarrow{OA} \cdot \overrightarrow{OB} = 0\).

Calculate the dot product:

\(\overrightarrow{OA} \cdot \overrightarrow{OB} = (3p)(-p) + (4)(-1) + (p^2)(p^2) = -3p^2 - 4 + p^4\).

Set the dot product to zero:

\(-3p^2 - 4 + p^4 = 0\).

Rearrange to form a quadratic in \(p^2\):

\(p^4 - 3p^2 - 4 = 0\).

Factorize:

\((p^2 + 1)(p^2 - 4) = 0\).

Thus, \(p^2 = -1\) (no real solution) or \(p^2 = 4\).

Therefore, \(p = \pm 2\).

(ii) For \(p = 3\), find \(\overrightarrow{BA}\):

\(\overrightarrow{BA} = \overrightarrow{OA} - \overrightarrow{OB} = \begin{pmatrix} 3(3) \\ 4 \\ 3^2 \end{pmatrix} - \begin{pmatrix} -3 \\ -1 \\ 9 \end{pmatrix} = \begin{pmatrix} 9 \\ 4 \\ 9 \end{pmatrix} - \begin{pmatrix} -3 \\ -1 \\ 9 \end{pmatrix} = \begin{pmatrix} 12 \\ 5 \\ 0 \end{pmatrix}\).

Calculate the magnitude of \(\overrightarrow{BA}\):

\(|\overrightarrow{BA}| = \sqrt{12^2 + 5^2 + 0^2} = \sqrt{144 + 25} = \sqrt{169} = 13\).

The unit vector in the direction of \(\overrightarrow{BA}\) is:

\(\frac{1}{13} \begin{pmatrix} 12 \\ 5 \\ 0 \end{pmatrix}\).