Answer: (a) An appropriate integrating factor is \(\frac14 x+\frac14\sqrt{x^2+16}\).

(b) The required solution is

\(y\left(x+\sqrt{x^2+16}\right)=\frac13x^3+\frac13(x^2+16)^{3/2}-\frac83\),

so equivalently

\(y=\frac{\frac13x^3+\frac13(x^2+16)^{3/2}-\frac83}{x+\sqrt{x^2+16}}\).

(a) First divide the differential equation by \(\sqrt{x^2+16}\):

\(\frac{\mathrm dy}{\mathrm dx}+\frac{1}{\sqrt{x^2+16}}y=x.\)

This is a linear differential equation of the form \(\frac{\mathrm dy}{\mathrm dx}+P(x)y=Q(x)\), where \(P(x)=\frac{1}{\sqrt{x^2+16}}\).

So an integrating factor is

\(\mathrm e^{\int \frac{1}{\sqrt{x^2+16}}\,\mathrm dx}.\)

Using \(\int \frac{1}{\sqrt{x^2+a^2}}\,\mathrm dx=\sinh^{-1}\left(\frac{x}{a}\right)\) with \(a=4\),

\(\int \frac{1}{\sqrt{x^2+16}}\,\mathrm dx=\sinh^{-1}\left(\frac{x}{4}\right).\)

Hence the integrating factor is

\(\mathrm e^{\sinh^{-1}(x/4)}.\)

Let \(u=\sinh^{-1}(x/4)\). Then \(\sinh u=\frac{x}{4}\), and so

\(\cosh u=\sqrt{1+\sinh^2u}=\sqrt{1+\frac{x^2}{16}}=\frac{\sqrt{x^2+16}}{4}.\)

Therefore

\(\mathrm e^u=\sinh u+\cosh u=\frac{x}{4}+\frac{\sqrt{x^2+16}}{4}.\)

So an appropriate integrating factor is \(\frac14x+\frac14\sqrt{x^2+16}\).

(b) A constant multiple of an integrating factor is also an integrating factor, so take

\(\mu(x)=x+\sqrt{x^2+16}.\)

Multiplying \(\frac{\mathrm dy}{\mathrm dx}+\frac{1}{\sqrt{x^2+16}}y=x\) by \(\mu(x)\) gives

\(\frac{\mathrm d}{\mathrm dx}\left(y\left(x+\sqrt{x^2+16}\right)\right)=x\left(x+\sqrt{x^2+16}\right)=x^2+x\sqrt{x^2+16}.\)

Integrating,

\(y\left(x+\sqrt{x^2+16}\right)=\int x^2\,\mathrm dx+\int x\sqrt{x^2+16}\,\mathrm dx+C.\)

Now

\(\int x^2\,\mathrm dx=\frac13x^3,\)

and with \(u=x^2+16\), \(\mathrm du=2x\,\mathrm dx\),

\(\int x\sqrt{x^2+16}\,\mathrm dx=\frac12\int u^{1/2}\,\mathrm du=\frac13u^{3/2}=\frac13(x^2+16)^{3/2}.\)

Hence

\(y\left(x+\sqrt{x^2+16}\right)=\frac13x^3+\frac13(x^2+16)^{3/2}+C.\)

Using \(y=6\) when \(x=3\):

\(6(3+\sqrt{25})=\frac{27}{3}+\frac{25}{3}\sqrt{25}+C.\)

So

\(48=9+\frac{125}{3}+C=\frac{152}{3}+C,\)

giving

\(C=-\frac83.\)

Therefore

\(y\left(x+\sqrt{x^2+16}\right)=\frac13x^3+\frac13(x^2+16)^{3/2}-\frac83,\)

or equivalently

\(y=\frac{\frac13x^3+\frac13(x^2+16)^{3/2}-\frac83}{x+\sqrt{x^2+16}}.\)