Answer: \(y=t^2+1-\frac14e^{-3t/5}\left(4\cos\frac{4t}{5}+3\sin\frac{4t}{5}\right).\)
Divide the differential equation by \(5\):
\(\frac{\mathrm d^2y}{\mathrm dt^2}+\frac65\frac{\mathrm dy}{\mathrm dt}+y=3+\frac{12}{5}t+t^2.\)
The auxiliary equation for the complementary function is
\(m^2+\frac65m+1=0.\)
So
\(m=-\frac35\pm\frac45i.\)
Hence
\(y_c=e^{-3t/5}\left(A\cos\frac{4t}{5}+B\sin\frac{4t}{5}\right).\)
Try a particular integral \(y_p=at^2+bt+c\). Substitution gives
\(a=1,\qquad b=0,\qquad c=1,\)
so
\(y_p=t^2+1.\)
Thus
\(y=e^{-3t/5}\left(A\cos\frac{4t}{5}+B\sin\frac{4t}{5}\right)+t^2+1.\)
Use \(y(0)=0\):
\(A+1=0,\qquad A=-1.\)
Differentiate and use \(y'(0)=0\):
\(-\frac35A+\frac45B=0.\)
With \(A=-1\), this gives
\(\frac35+\frac45B=0,\qquad B=-\frac34.\)
Therefore
\(y=t^2+1-e^{-3t/5}\left(\cos\frac{4t}{5}+\frac34\sin\frac{4t}{5}\right).\)
Equivalently,
\(y=t^2+1-\frac14e^{-3t/5}\left(4\cos\frac{4t}{5}+3\sin\frac{4t}{5}\right).\)
