Answer:

(a)

\(\bar{x}=\frac{h^2-6ah+11a^2}{3(3a-h)}, \qquad \bar{y}=\frac{2a(5a-2h)}{3(3a-h)}.\)

(b)

\(0\lt h\le a \quad\text{or}\quad h=2a.\)

If degenerate cases are included, \(h=0\) is also allowed.

(a) The square \(ABCD\) has area \(4a^2\) and centre of mass at distance \(a\) from both \(AD\) and \(AB\).

The triangle \(AED\) has area

\(\frac12(a)(2a)=a^2.\)

Its centre of mass is at distance \(\frac13a\) from \(AD\), and at distance \(\frac23a\) from \(AB\).

The triangle \(BCF\) has area

\(\frac12(h)(2a)=ah.\)

Its centre of mass is at distance

\(2a-\frac13h\)

from \(AD\), and at distance

\(\frac43a\)

from \(AB\).

The remaining lamina \(BFDE\) has area

\(4a^2-a^2-ah=3a^2-ah.\)

Taking moments about \(AD\),

\((3a^2-ah)\bar{x} =4a^2(a)-a^2\left(\frac13a\right)-ah\left(2a-\frac13h\right).\)

So

\((3a^2-ah)\bar{x} =4a^3-\frac13a^3-2a^2h+\frac13ah^2.\)

Hence

\((3a^2-ah)\bar{x} =\frac13a\left(11a^2-6ah+h^2\right).\)

Therefore

\(\bar{x} =\frac{\frac13a(h^2-6ah+11a^2)}{a(3a-h)} =\frac{h^2-6ah+11a^2}{3(3a-h)}.\)

Now take moments about \(AB\):

\((3a^2-ah)\bar{y} =4a^2(a)-a^2\left(\frac23a\right)-ah\left(\frac43a\right).\)

Thus

\((3a^2-ah)\bar{y} =4a^3-\frac23a^3-\frac43a^2h.\)

So

\((3a^2-ah)\bar{y} =\frac23a^2(5a-2h).\)

Therefore

\(\bar{y}=\frac{2a(5a-2h)}{3(3a-h)}.\)

(b) The lamina rests on the edge \(EB\). Since \(AE=a\), the support edge \(EB\) runs from \(x=a\) to \(x=2a\).

For equilibrium, the vertical line through the centre of mass must pass through the support edge. The relevant condition is

\(\bar{x}\ge a.\)

Using the expression for \(\bar{x}\),

\(\frac{h^2-6ah+11a^2}{3(3a-h)}\ge a.\)

For \(0\le h\le2a\), the denominator \(3(3a-h)\) is positive, so

\(h^2-6ah+11a^2\ge 3a(3a-h).\)

Hence

\(h^2-6ah+11a^2\ge9a^2-3ah.\)

So

\(h^2-3ah+2a^2\ge0.\)

Factorising,

\((h-a)(h-2a)\ge0.\)

With \(0\le h\le2a\), this gives

\(0\le h\le a \quad\text{or}\quad h=2a.\)

If \(h\) is required to be positive, the set is

\(0\lt h\le a\quad\text{or}\quad h=2a.\)