Answer: (a) \(V=\frac{\sqrt3}{9}(h^3-24h^2+144h)\), so \(a=1\), \(b=-24\), \(c=144\); (b) \(h=4\).

Answer: (a) \(V=\frac{\sqrt3}{9}(h^3-24h^2+144h)\), so \(a=1\), \(b=-24\), \(c=144\); (b) \(h=4\).

(a) Since pyramids \(OABC\) and \(OPQR\) are similar, the side length \(PQ\) of the cross-section is proportional to the distance from \(O\).

The distance from \(O\) down to \(PQR\) is \(12-h\), while the full height of the pyramid is \(12\). Therefore

\(\frac{PQ}{8}=\frac{12-h}{12}.\)

So

\(PQ=\frac{8(12-h)}{12}=\frac{2}{3}(12-h).\)

The triangular cross-section \(PQR\) is equilateral, so its area is

\(\frac{\sqrt3}{4}PQ^2.\)

Thus

The prism has height \(h\), so

\(V=\frac{\sqrt3}{9}h(12-h)^2.\)

Expand:

\(V=\frac{\sqrt3}{9}h(h^2-24h+144).\)

Therefore

\(V=\frac{\sqrt3}{9}(h^3-24h^2+144h).\)

So

\(a=1,\quad b=-24,\quad c=144.\)

(b) Differentiate \(V\) with respect to \(h\):

\(\frac{dV}{dh} =\frac{\sqrt3}{9}(3h^2-48h+144).\)

At a maximum or minimum, \(\frac{dV}{dh}=0\), so

\(3h^2-48h+144=0.\)

Divide by \(3\):

\(h^2-16h+48=0.\)

Factorise:

\((h-4)(h-12)=0.\)

So \(h=4\) or \(h=12\). When \(h=12\), the cross-section has shrunk to zero, so this is not the maximum volume.

Therefore the maximum volume occurs when

\(h=4.\)