Answer: \(\cos 5\theta\equiv \cos\theta\left(16\sin^4\theta-12\sin^2\theta+1\right)\), and

\(\sin^2\left(\frac{\pi}{10}\right)=\frac{3-\sqrt{5}}{8}.\)

Let \(c=\cos\theta\) and \(s=\sin\theta\).

By de Moivre’s theorem,

\((c+is)^5=\cos 5\theta+i\sin 5\theta\).

Expand using the binomial theorem:

\((c+is)^5=c^5+5c^4(is)+10c^3(is)^2+10c^2(is)^3+5c(is)^4+(is)^5\).

Taking real parts,

\(\cos 5\theta=c^5-10c^3s^2+5cs^4\).

Factor out \(c\):

\(\cos 5\theta=c\left(c^4-10c^2s^2+5s^4\right)\).

Now use \(c^2=1-s^2\):

\(\cos 5\theta=c\left((1-s^2)^2-10s^2(1-s^2)+5s^4\right)\).

Expanding the bracket gives

\(\cos 5\theta=c\left(1-2s^2+s^4-10s^2+10s^4+5s^4\right)\)

so

\(\cos 5\theta=c\left(16s^4-12s^2+1\right)\).

Hence

\(\cos 5\theta\equiv \cos\theta\left(16\sin^4\theta-12\sin^2\theta+1\right)\).

Now consider \(\cos 5\theta=0\). Then

\(5\theta=\frac{\pi}{2},\frac{3\pi}{2},\frac{5\pi}{2},\frac{7\pi}{2},\frac{9\pi}{2}\),

so

\(\theta=\frac{\pi}{10},\frac{3\pi}{10},\frac{\pi}{2},\frac{7\pi}{10},\frac{9\pi}{10}\).

From the identity,

\(\cos\theta\left(16\sin^4\theta-12\sin^2\theta+1\right)=0\).

For \(\theta=\frac{\pi}{10}\), \(\cos\theta\neq 0\), so

\(16\sin^4\theta-12\sin^2\theta+1=0\).

Let \(x=\sin^2\theta\). Then

\(16x^2-12x+1=0\).

Solving,

\(x=\frac{12\pm\sqrt{144-64}}{32}=\frac{12\pm\sqrt{80}}{32}=\frac{12\pm 4\sqrt5}{32}=\frac{3\pm\sqrt5}{8}\).

These correspond to the two distinct values of \(\sin^2\theta\) from the roots. Since \(\sin^2\left(\frac{\pi}{10}\right)\lt \sin^2\left(\frac{3\pi}{10}\right)\), the smaller root is needed:

\(\sin^2\left(\frac{\pi}{10}\right)=\frac{3-\sqrt5}{8}.\)