(a) By Vieta's formulas, we have:

\(\alpha + \beta + \gamma = -5\)

\(\alpha\beta + \beta\gamma + \gamma\alpha = 10\)

\(\alpha\beta\gamma = 2\)

We use the identity:

\(\alpha^2 + \beta^2 + \gamma^2 = (\alpha + \beta + \gamma)^2 - 2(\alpha\beta + \beta\gamma + \gamma\alpha)\)

\(\alpha^2 + \beta^2 + \gamma^2 = (-5)^2 - 2(10) = 25 - 20 = 5\)

(b) To show the matrix is singular, we find its determinant:

\(\begin{vmatrix} 1 & \alpha & \beta \\ \alpha & 1 & \gamma \\ \beta & \gamma & 1 \end{vmatrix} = 1 - \alpha^2 - \beta^2 - \gamma^2 + 2\alpha\beta\gamma\)

Substituting the known values:

\(1 - 5 + 2(2) = 0\)

Thus, the determinant is zero, and the matrix is singular.