Answer: (i) \((2n+1)I_n=\sqrt2-2nI_{n-1}\).

(ii) \(I_n=\int_0^{\pi/4}\tan^{2n+1}\theta\sec\theta\,d\theta\).

(iii) \(\int_0^{\pi/4}\frac{\sin^7\theta}{\cos^8\theta}\,d\theta=\frac{16-9\sqrt2}{35}\).

For \(I_n=\int_1^{\sqrt2}(x^2-1)^n\,dx\), integrate by parts using \(u=(x^2-1)^n\) and \(dv=dx\). Then \(du=2nx(x^2-1)^{n-1}\,dx\) and \(v=x\).

So

\(I_n=\left[x(x^2-1)^n\right]_1^{\sqrt2}-2n\int_1^{\sqrt2}x^2(x^2-1)^{n-1}\,dx\).

The boundary term is \(\sqrt2\). Also, \(x^2=(x^2-1)+1\), so

\(\int_1^{\sqrt2}x^2(x^2-1)^{n-1}\,dx=I_n+I_{n-1}\).

Therefore

\(I_n=\sqrt2-2nI_n-2nI_{n-1}\), which gives \((2n+1)I_n=\sqrt2-2nI_{n-1}\).

Now use \(x=\sec\theta\). Then \(dx=\sec\theta\tan\theta\,d\theta\). When \(x=1\), \(\theta=0\), and when \(x=\sqrt2\), \(\theta=\frac{\pi}{4}\).

Also \(x^2-1=\sec^2\theta-1=\tan^2\theta\). Hence

\(I_n=\int_0^{\pi/4}(\tan^2\theta)^n\sec\theta\tan\theta\,d\theta=\int_0^{\pi/4}\tan^{2n+1}\theta\sec\theta\,d\theta\).

Since \(\frac{\sin^7\theta}{\cos^8\theta}=\tan^7\theta\sec\theta\), the required integral is \(I_3\).

First, \(I_0=\int_1^{\sqrt2}1\,dx=\sqrt2-1\).

Using the reduction formula with \(n=1\),

\(3I_1=\sqrt2-2I_0=\sqrt2-2(\sqrt2-1)=2-\sqrt2\), so \(I_1=\frac{2-\sqrt2}{3}\).

With \(n=2\),

\(5I_2=\sqrt2-4I_1=\sqrt2-\frac{4(2-\sqrt2)}{3}=\frac{7\sqrt2-8}{3}\), so \(I_2=\frac{7\sqrt2-8}{15}\).

With \(n=3\),

\(7I_3=\sqrt2-6I_2=\sqrt2-\frac{6(7\sqrt2-8)}{15}=\frac{16-9\sqrt2}{5}\).

Therefore

\(I_3=\frac{16-9\sqrt2}{35}\), and the required integral is \(\frac{16-9\sqrt2}{35}\).