Answer: The general solution is \(y=\dfrac{1}{x}\left(Ae^{-3x}+Be^{x}+2e^{2x}\right)\), where \(A\) and \(B\) are arbitrary constants.
Let \(v=xy\). Then
\(\dfrac{dv}{dx}=y+x\dfrac{dy}{dx}\).
Differentiating again,
\(\dfrac{d^2v}{dx^2}=\dfrac{dy}{dx}+\dfrac{d}{dx}\left(x\dfrac{dy}{dx}\right)=\dfrac{dy}{dx}+\dfrac{dy}{dx}+x\dfrac{d^2y}{dx^2}=2\dfrac{dy}{dx}+x\dfrac{d^2y}{dx^2}.\)
Now start with
\(x\dfrac{d^2y}{dx^2}+(2x+2)\dfrac{dy}{dx}+(2-3x)y=10e^{2x}.\)
Rewrite the left-hand side as
\(\left(x\dfrac{d^2y}{dx^2}+2\dfrac{dy}{dx}\right)+2\left(y+x\dfrac{dy}{dx}\right)-3xy.\)
Using \(v''=2\dfrac{dy}{dx}+x\dfrac{d^2y}{dx^2}\), \(v'=y+x\dfrac{dy}{dx}\) and \(v=xy\), this becomes
\(\dfrac{d^2v}{dx^2}+2\dfrac{dv}{dx}-3v=10e^{2x}.\)
To solve this, first find the complementary function from
\(m^2+2m-3=0.\)
So
\((m+3)(m-1)=0\), giving \(m=-3\) and \(m=1\).
Hence the complementary function is
\(v_c=Ae^{-3x}+Be^x.\)
For a particular integral, try \(v_p=ke^{2x}\).
Then
\(v_p'=2ke^{2x}, \qquad v_p''=4ke^{2x}.\)
Substituting into \(v''+2v'-3v=10e^{2x}\),
\(4ke^{2x}+2(2ke^{2x})-3ke^{2x}=10e^{2x}.\)
So
\(5ke^{2x}=10e^{2x}\), hence \(k=2\).
Therefore
\(v=Ae^{-3x}+Be^x+2e^{2x}.\)
Since \(v=xy\),
\(y=\dfrac{v}{x}=\dfrac{1}{x}\left(Ae^{-3x}+Be^x+2e^{2x}\right).\)