Answer: Using de Moivre’s theorem, let \(c=\cos\theta\) and \(s=\sin\theta\). Then

\((c+i s)^5=\cos 5\theta+i\sin 5\theta\).

Expanding and taking real parts,

\(\cos 5\theta=c^5-10c^3s^2+5cs^4\).

Factor out \(c\):

\(\cos 5\theta=c\left(c^4-10c^2s^2+5s^4\right)\).

Now use \(c^2=1-s^2\):

\(\cos 5\theta=c\left((1-s^2)^2-10s^2(1-s^2)+5s^4\right)\)

\(=c\left(1-2s^2+s^4-10s^2+10s^4+5s^4\right)\)

\(=c\left(16s^4-12s^2+1\right)\).

Hence

\(\cos 5\theta\equiv \cos\theta\left(16\sin^4\theta-12\sin^2\theta+1\right)\).

Now take \(\theta=\frac{\pi}{10}\). Then \(5\theta=\frac{\pi}{2}\), so \(\cos 5\theta=0\). Also \(\cos\frac{\pi}{10}\neq 0\), so

\(16\sin^4\left(\frac{\pi}{10}\right)-12\sin^2\left(\frac{\pi}{10}\right)+1=0\).

Let \(x=\sin^2\left(\frac{\pi}{10}\right)\). Then

\(16x^2-12x+1=0\),

so

\(x=\frac{12\pm\sqrt{144-64}}{32}=\frac{12\pm4\sqrt5}{32}=\frac{3\pm\sqrt5}{8}\).

Since \(\sin^2\left(\frac{\pi}{10}\right)\) is the smaller value,

\(\sin^2\left(\frac{\pi}{10}\right)=\frac{3-\sqrt5}{8}\).

Let \(c=\cos\theta\) and \(s=\sin\theta\). By de Moivre’s theorem,

\((c+is)^5=\cos 5\theta+i\sin 5\theta\).

Expand using the binomial theorem:

\((c+is)^5=c^5+5c^4is+10c^3(is)^2+10c^2(is)^3+5c(is)^4+(is)^5\).

The real part is therefore

\(\cos 5\theta=c^5-10c^3s^2+5cs^4\).

Factor out \(c\):

\(\cos 5\theta=c\left(c^4-10c^2s^2+5s^4\right)\).

Now substitute \(c^2=1-s^2\):

\(\cos 5\theta=c\left((1-s^2)^2-10s^2(1-s^2)+5s^4\right)\).

Expand and simplify:

\(\cos 5\theta=c\left(1-2s^2+s^4-10s^2+10s^4+5s^4\right)\)

\(=c\left(16s^4-12s^2+1\right)\).

So

\(\cos 5\theta=\cos\theta\left(16\sin^4\theta-12\sin^2\theta+1\right)\),

as required.

Now consider \(\cos 5\theta=0\). The solutions in \(0\le \theta \le \pi\) are

\(\theta=\frac{\pi}{10},\frac{3\pi}{10},\frac{\pi}{2},\frac{7\pi}{10},\frac{9\pi}{10}\).

For \(\theta=\frac{\pi}{10}\), we have \(\cos\theta\neq 0\), so from the identity

\(16\sin^4\theta-12\sin^2\theta+1=0\).

Let \(x=\sin^2\left(\frac{\pi}{10}\right)\). Then

\(16x^2-12x+1=0\).

Using the quadratic formula,

\(x=\frac{12\pm\sqrt{12^2-4\cdot16\cdot1}}{32}=\frac{12\pm\sqrt{80}}{32}=\frac{3\pm\sqrt5}{8}\).

Since \(\sin^2\left(\frac{\pi}{10}\right)\lt \sin^2\left(\frac{3\pi}{10}\right)\), the value at \(\theta=\frac{\pi}{10}\) is the smaller root. Hence

\(\sin^2\left(\frac{\pi}{10}\right)=\frac{3-\sqrt5}{8}\).