Answer: (a) An appropriate integrating factor is \(x+\sqrt{x^{2}-1}\).

(b) \(y=\dfrac{\sqrt{x^{2}-1}+\frac{5}{4}}{x+\sqrt{x^{2}-1}}\).

(a) Divide through by \(\sqrt{x^{2}-1}\):

\(\dfrac{\mathrm{d}y}{\mathrm{d}x}+\dfrac{1}{\sqrt{x^{2}-1}}y=\dfrac{x^{2}-x\sqrt{x^{2}-1}}{\sqrt{x^{2}-1}}\).

So an integrating factor is

\(\mathrm{e}^{\int \frac{1}{\sqrt{x^{2}-1}}\,\mathrm{d}x}=\mathrm{e}^{\cosh^{-1}x}.\)

Using \(\cosh^{-1}x=\ln\!\bigl(x+\sqrt{x^{2}-1}\bigr)\), this becomes

\(\mathrm{e}^{\cosh^{-1}x}=x+\sqrt{x^{2}-1}.\)

Hence an appropriate integrating factor is \(x+\sqrt{x^{2}-1}\).

(b) Multiply the linear equation by \(x+\sqrt{x^{2}-1}\):

\(\dfrac{\mathrm{d}}{\mathrm{d}x}\left(y\left(x+\sqrt{x^{2}-1}\right)\right)=\dfrac{(x+\sqrt{x^{2}-1})(x^{2}-x\sqrt{x^{2}-1})}{\sqrt{x^{2}-1}}.\)

Since

\((x+\sqrt{x^{2}-1})(x^{2}-x\sqrt{x^{2}-1})=x(x+\sqrt{x^{2}-1})(x-\sqrt{x^{2}-1})=x,\)

the equation becomes

\(\dfrac{\mathrm{d}}{\mathrm{d}x}\left(y\left(x+\sqrt{x^{2}-1}\right)\right)=\dfrac{x}{\sqrt{x^{2}-1}}.\)

Integrating,

\(y\left(x+\sqrt{x^{2}-1}\right)=\sqrt{x^{2}-1}+C.\)

Now use \(y=1\) when \(x=\frac{5}{4}\). Then

\(\sqrt{x^{2}-1}=\sqrt{\frac{25}{16}-1}=\frac{3}{4},\)

so

\(1\left(\frac{5}{4}+\frac{3}{4}\right)=\frac{3}{4}+C\)

which gives \(2=\frac{3}{4}+C\), hence \(C=\frac{5}{4}.\)

Therefore

\(y=\dfrac{\sqrt{x^{2}-1}+\frac{5}{4}}{x+\sqrt{x^{2}-1}}.\)