(a) Expanding \(R\cos(\theta+\alpha)\) gives \(R\cos\alpha\cos\theta-R\sin\alpha\sin\theta\). Thus \(R\cos\alpha=5,R\sin\alpha=3\), so \[\boxed{R=\sqrt{34},\qquad\alpha=\arctan(3/5)}.\]

(b) Using (a), \[\begin{aligned}\int_0^\alpha\frac{68}{(5\cos\theta-3\sin\theta)^2}\,d\theta&=2\int_0^\alpha\sec^2(\theta+\alpha)\,d\theta\\&=2[\tan(\theta+\alpha)]_0^\alpha=2(\tan2\alpha-\tan\alpha).\end{aligned}\] Since \(\tan\alpha=3/5\), \[\tan2\alpha=\frac{2(3/5)}{1-(3/5)^2}=\frac{15}{8},\] hence the integral equals \(2(15/8-3/5)=\boxed{51/20}\).