Answer: \(\alpha^2+\beta^2+\gamma^2=31\).

The determinant of \(\begin{pmatrix}1&\alpha&\beta\\ \alpha&1&\gamma\\ \beta&\gamma&1\end{pmatrix}\) is \(0\), so the matrix is singular.

Let the roots of \(x^3+5x^2-3x-15=0\) be \(\alpha\), \(\beta\) and \(\gamma\).

By Vieta's formulae,

\(\alpha+\beta+\gamma=-5\),

\(\alpha\beta+\beta\gamma+\gamma\alpha=-3\),

and

\(\alpha\beta\gamma=15\).

Use the identity

\(\alpha^2+\beta^2+\gamma^2=(\alpha+\beta+\gamma)^2-2(\alpha\beta+\beta\gamma+\gamma\alpha)\).

Therefore

\(\alpha^2+\beta^2+\gamma^2=(-5)^2-2(-3)=25+6=31\).

Now consider the determinant

\(\det\begin{pmatrix}1&\alpha&\beta\\ \alpha&1&\gamma\\ \beta&\gamma&1\end{pmatrix}\).

Expanding gives

\(\det=1-(\alpha^2+\beta^2+\gamma^2)+2\alpha\beta\gamma\).

Substitute the values found above:

\(\det=1-31+2(15)=0\).

Since the determinant is zero, the matrix is singular.