Answer: (a) \((5n+1)I_n=2^n+5nI_{n-1}\).

(b) \(I_3=\frac{323}{176}\).

(a) Consider \(\frac{\mathrm d}{\mathrm dx}\big(x(1+x^5)^n\big)\).

Using the product rule,

\(\frac{\mathrm d}{\mathrm dx}\big(x(1+x^5)^n\big)=(1+x^5)^n+x\cdot n(1+x^5)^{n-1}\cdot 5x^4=(1+x^5)^n+5nx^5(1+x^5)^{n-1}.\)

Now write \(x^5=(1+x^5)-1\). Then

\(5nx^5(1+x^5)^{n-1}=5n\big((1+x^5)-1\big)(1+x^5)^{n-1}=5n(1+x^5)^n-5n(1+x^5)^{n-1}.\)

So

\(\frac{\mathrm d}{\mathrm dx}\big(x(1+x^5)^n\big)=(5n+1)(1+x^5)^n-5n(1+x^5)^{n-1}.\)

Integrating from \(0\) to \(1\),

\(\left[x(1+x^5)^n\right]_0^1=(5n+1)\int_0^1(1+x^5)^n\,\mathrm dx-5n\int_0^1(1+x^5)^{n-1}\,\mathrm dx.\)

Hence

\(\left[x(1+x^5)^n\right]_0^1=(5n+1)I_n-5nI_{n-1}.\)

Evaluating the left-hand side,

\(\left[x(1+x^5)^n\right]_0^1=1\cdot(1+1^5)^n-0=2^n.\)

Therefore

\(2^n=(5n+1)I_n-5nI_{n-1},\)

so

\((5n+1)I_n=2^n+5nI_{n-1}.\)

(b) First find a starting value:

\(I_1=\int_0^1(1+x^5)\,\mathrm dx=\left[x+\frac{x^6}{6}\right]_0^1=\frac76.\)

Now use the recurrence.

For \(n=2\):

\(11I_2=2^2+10I_1=4+10\cdot\frac76=4+\frac{35}{3}=\frac{47}{3},\)

so

\(I_2=\frac{47}{33}.\)

For \(n=3\):

\(16I_3=2^3+15I_2=8+15\cdot\frac{47}{33}=8+\frac{235}{11}=\frac{323}{11},\)

hence

\(I_3=\frac{323}{176}.\)