Answer: \(e=\dfrac{k-1}{k+1}\), and then \(\boxed{k=3}\) and \(\boxed{e=\dfrac12}\).

Take the original direction of motion of sphere \(A\) as positive. Then initially

\[u_A=u,\qquad u_B=-ku.\]

After the collision, sphere \(B\) is brought to rest. Let the final velocity of \(A\) be \(v_A\).

(i) Conservation of momentum gives

\[mu+m(-ku)=mv_A.\]

So

\[v_A=(1-k)u.\]

Newton's law of restitution gives

\[\text{speed of separation}=e\times\text{speed of approach}.\]

Thus

\[0-v_A=e\{u-(-ku)\}.\]

Substitute \(v_A=(1-k)u\):

\[-(1-k)u=e(1+k)u.\]

Hence

\[(k-1)u=e(1+k)u,\]

and therefore

\[\boxed{e=\frac{k-1}{k+1}}.\]

(ii) The final kinetic energy is \(40\%\) of the initial kinetic energy, because \(60\%\) is lost.

The initial kinetic energy is

\[\frac12mu^2+\frac12m(ku)^2=\frac12mu^2(1+k^2).\]

The final kinetic energy is

\[\frac12m\{(1-k)u\}^2=\frac12mu^2(1-k)^2.\]

So

\[(1-k)^2=0.4(1+k^2).\]

Expand and simplify:

\[k^2-2k+1=0.4+0.4k^2,\]

so

\[0.6k^2-2k+0.6=0.\]

Multiplying by \(5\),

\[3k^2-10k+3=0.\]

Factorise:

\[(3k-1)(k-3)=0.\]

So \(k=\frac13\) or \(k=3\). But \(e=\frac{k-1}{k+1}\) must be positive, so \(k>1\). Therefore

\[\boxed{k=3}.\]

Finally,

\[e=\frac{3-1}{3+1}=\boxed{\frac12}.\]