Answer: \(I_4=\frac{\pi}{32}\).

Let \(I_n=\int_0^{\pi/2}\cos^n x\sin^2 x\,dx\).

Consider \(\cos^{n-1}x\sin^3x\). Differentiating gives

\(\frac{d}{dx}(\cos^{n-1}x\sin^3x)=-(n-1)\cos^{n-2}x\sin^4x+3\cos^n x\sin^2x\).

Now integrate this derivative from \(0\) to \(\pi/2\):

\(\left[\cos^{n-1}x\sin^3x\right]_0^{\pi/2}=\int_0^{\pi/2}\left(-(n-1)\cos^{n-2}x\sin^4x+3\cos^n x\sin^2x\right)dx\).

The boundary term is zero, since \(\sin 0=0\) and \(\cos(\pi/2)=0\). Also, \(\sin^4x=\sin^2x(1-\cos^2x)\), so

\(0=-(n-1)\int_0^{\pi/2}\cos^{n-2}x\sin^2x(1-\cos^2x)\,dx+3I_n\).

Expanding the bracket gives

\(0=-(n-1)\int_0^{\pi/2}\cos^{n-2}x\sin^2x\,dx+(n-1)\int_0^{\pi/2}\cos^n x\sin^2x\,dx+3I_n\).

So

\(0=-(n-1)I_{n-2}+(n-1)I_n+3I_n\),

hence

\((n+2)I_n=(n-1)I_{n-2}\) for \(n\ge 2\).

For \(I_4\), use the recurrence twice:

\(6I_4=3I_2\), so \(I_4=\frac12 I_2\); and \(4I_2=I_0\), so \(I_2=\frac14 I_0\). Therefore

\(I_4=\frac18 I_0\).

Now compute \(I_0\):

\(I_0=\int_0^{\pi/2}\sin^2x\,dx=\int_0^{\pi/2}\frac{1-\cos2x}{2}\,dx=\left[\frac{x}{2}-\frac{\sin2x}{4}\right]_0^{\pi/2}=\frac{\pi}{4}\).

So

\(I_4=\frac18\cdot\frac{\pi}{4}=\frac{\pi}{32}\).