Answer: \(\overrightarrow{OX}=\frac14\mathbf{a}+\frac34\mathbf{b}\), \(\overrightarrow{AC}=2\mathbf{b}-\mathbf{a}\), \(h=\frac85\), \(m=\frac35\).

(a) Since \(AX:XB=3:1\), point \(X\) is three quarters of the way from \(A\) to \(B\). Therefore

\(\overrightarrow{OX} =\mathbf{a}+\frac34\overrightarrow{AB}.\)

But

\(\overrightarrow{AB}=\mathbf{b}-\mathbf{a}.\)

So

\(\overrightarrow{OX} =\mathbf{a}+\frac34(\mathbf{b}-\mathbf{a}) =\frac14\mathbf{a}+\frac34\mathbf{b}.\)

(b) Since \(B\) is the midpoint of \(OC\),

\(\overrightarrow{OC}=2\mathbf{b}.\)

Thus

\(\overrightarrow{AC} =\overrightarrow{OC}-\overrightarrow{OA} =2\mathbf{b}-\mathbf{a}.\)

(c) Given that \(\overrightarrow{OY}=h\overrightarrow{OX}\),

\(\overrightarrow{OY} =h\left(\frac14\mathbf{a}+\frac34\mathbf{b}\right).\)

Therefore

\(\overrightarrow{AY} =\overrightarrow{OY}-\overrightarrow{OA} =-\mathbf{a}+h\left(\frac14\mathbf{a}+\frac34\mathbf{b}\right).\)

(d) Since

\(\overrightarrow{AY}=m\overrightarrow{AC},\)

we have

\(-\mathbf{a}+h\left(\frac14\mathbf{a}+\frac34\mathbf{b}\right) =m(2\mathbf{b}-\mathbf{a}).\)

Equating coefficients of \(\mathbf{a}\) and \(\mathbf{b}\),

\(-1+\frac{h}{4}=-m\)

and

\(\frac{3h}{4}=2m.\)

From the first equation,

\(m=1-\frac{h}{4}.\)

Substitute this into the second equation:

\(\frac{3h}{4}=2\left(1-\frac{h}{4}\right).\)

So

\(3h=8-2h,\)

and hence

\(h=\frac85.\)

Then

\(m=1-\frac{1}{4}\cdot\frac85=\frac35.\)