Answer: \(y=\mathrm{e}^{-x/3}\left(-2\cos\left(\frac{\sqrt{2}}{3}x\right)+5\sqrt{2}\sin\left(\frac{\sqrt{2}}{3}x\right)\right)+x^2-4x+2\).

We solve \(3\dfrac{\mathrm{d}^2y}{\mathrm{d}x^2}+2\dfrac{\mathrm{d}y}{\mathrm{d}x}+y=x^2\), with \(y(0)=0\) and \(y'(0)=0\).

First solve the homogeneous equation:

\(3y''+2y'+y=0\).

The auxiliary equation is

\(3m^2+2m+1=0\).

So

\(m=\dfrac{-2\pm\sqrt{4-12}}{6}=\dfrac{-2\pm 2\sqrt{2}\,i}{6}=-\dfrac13\pm\dfrac{\sqrt2}{3}i\).

Hence the complementary function is

\(y_c=\mathrm{e}^{-x/3}\left(A\cos\left(\frac{\sqrt2}{3}x\right)+B\sin\left(\frac{\sqrt2}{3}x\right)\right)\).

For a particular integral, try

\(y_p=px^2+qx+r\).

Then

\(y_p'=2px+q,\qquad y_p''=2p\).

Substituting into the differential equation gives

\(3(2p)+2(2px+q)+(px^2+qx+r)=x^2\),

so

\(px^2+(4p+q)x+(6p+2q+r)=x^2\).

Equating coefficients:

\(p=1,\qquad 4p+q=0,\qquad 6p+2q+r=0\).

Thus

\(q=-4,\qquad r=2\).

So

\(y_p=x^2-4x+2\).

The general solution is therefore

\(y=\mathrm{e}^{-x/3}\left(A\cos\left(\frac{\sqrt2}{3}x\right)+B\sin\left(\frac{\sqrt2}{3}x\right)\right)+x^2-4x+2\).

Now use the conditions.

From \(y(0)=0\):

\(0=A+2\), so \(A=-2\).

Differentiate:

\(y'=\mathrm{e}^{-x/3}\left(-\frac{\sqrt2}{3}A\sin\left(\frac{\sqrt2}{3}x\right)+\frac{\sqrt2}{3}B\cos\left(\frac{\sqrt2}{3}x\right)\right)-\frac13\mathrm{e}^{-x/3}\left(A\cos\left(\frac{\sqrt2}{3}x\right)+B\sin\left(\frac{\sqrt2}{3}x\right)\right)+2x-4\).

From \(y'(0)=0\):

\(0=\frac{\sqrt2}{3}B-\frac13A-4\).

Substitute \(A=-2\):

\(0=\frac{\sqrt2}{3}B+\frac23-4=\frac{\sqrt2}{3}B-\frac{10}{3}\).

So

\(\sqrt2 B=10\), hence \(B=5\sqrt2\).

Therefore the required particular solution is

\(y=\mathrm{e}^{-x/3}\left(-2\cos\left(\frac{\sqrt{2}}{3}x\right)+5\sqrt{2}\sin\left(\frac{\sqrt{2}}{3}x\right)\right)+x^2-4x+2\).