The position vectors give the coordinates \(A(-1,1,2)\), \(B(-2,-1,0)\) and \(C(2,0,2)\).
(a) Two direction vectors in the plane \(ABC\) are
\(\overrightarrow{AB}=B-A=(-2,-1,0)-(-1,1,2)=(-1,-2,-2)\),
\(\overrightarrow{AC}=C-A=(2,0,2)-(-1,1,2)=(3,-1,0)\).
A normal vector to the plane is given by the cross product:
\(\overrightarrow{AB}\times \overrightarrow{AC}=(-1,-2,-2)\times(3,-1,0)=(-2,-6,7)\).
So the equation of the plane has the form \(-2x-6y+7z=d\). Substituting \(A(-1,1,2)\),
\(-2(-1)-6(1)+7(2)=2-6+14=10\).
Hence the equation of the plane is \(-2x-6y+7z=10\).
(b) The distance from the origin to the plane \(-2x-6y+7z=10\) is
\(\dfrac{|0-10|}{\sqrt{(-2)^2+(-6)^2+7^2}}=\dfrac{10}{\sqrt{4+36+49}}=\dfrac{10}{\sqrt{89}}\).
(c) The plane \(OAB\) contains the vectors \(\overrightarrow{OA}=(-1,1,2)\) and \(\overrightarrow{OB}=(-2,-1,0)\). A normal vector to this plane is
\(\overrightarrow{OA}\times \overrightarrow{OB}=(-1,1,2)\times(-2,-1,0)=(2,-4,3)\).
A normal vector to the plane \(ABC\) is \((-2,-6,7)\). The angle between two planes is the angle between their normal vectors, taking the acute angle.
Using the scalar product,
\((-2,-6,7)\cdot(2,-4,3)=(-2)(2)+(-6)(-4)+7(3)=-4+24+21=41\).
The magnitudes are \(|(-2,-6,7)|=\sqrt{89}\) and \(|(2,-4,3)|=\sqrt{29}\).
Therefore \(\cos\theta=\dfrac{41}{\sqrt{89}\sqrt{29}}\), so
\(\theta=\cos^{-1}\left(\dfrac{41}{\sqrt{89}\sqrt{29}}\right)=36.2^\circ\) to 3 significant figures.
