(a) Using the identity for the sum of squares of roots: \(\alpha^2 + \beta^2 + \gamma^2 = (\alpha + \beta + \gamma)^2 - 2(\alpha\beta + \beta\gamma + \gamma\alpha)\).

From the cubic equation, \(\alpha + \beta + \gamma = -4\) and \(\alpha\beta + \beta\gamma + \gamma\alpha = 6\).

Thus, \(\alpha^2 + \beta^2 + \gamma^2 = (-4)^2 - 2(6) = 16 - 12 = 4\).

(b) Expand \((\alpha + r)^2 = \alpha^2 + 2\alpha r + r^2\).

Therefore, \(\sum_{r=1}^{n} ((\alpha + r)^2 + (\beta + r)^2 + (\gamma + r)^2) = \sum_{r=1}^{n} (\alpha^2 + 2\alpha r + r^2 + \beta^2 + 2\beta r + r^2 + \gamma^2 + 2\gamma r + r^2)\).

Using \(\alpha + \beta + \gamma = -4\) and \(\alpha^2 + \beta^2 + \gamma^2 = 4\), we have:

\(\sum_{r=1}^{n} (3r^2 + 2(-4)r + 4) = \sum_{r=1}^{n} (3r^2 - 8r + 4)\).

Using standard summation formulas:

\(\sum_{r=1}^{n} r^2 = \frac{n(n+1)(2n+1)}{6}\) and \(\sum_{r=1}^{n} r = \frac{n(n+1)}{2}\).

Thus, \(3 \sum_{r=1}^{n} r^2 - 8 \sum_{r=1}^{n} r + 4n = 3 \cdot \frac{n(n+1)(2n+1)}{6} - 8 \cdot \frac{n(n+1)}{2} + 4n\).

Simplifying gives:

\(\frac{n(n+1)(2n+1)}{2} - 4n(n+1) + 4n\).

\(= n(n^2 - \frac{5}{2}n + \frac{1}{2})\).

Thus, \(a = -\frac{5}{2}, b = \frac{1}{2}\).