Yup. Remove the eyecup and the Swarovski adaptor works just fine. In practice, you will find that you don't miss the eyecup at all. I don't even bother taking it in the car anymore. (Same setup as you, Softsfan: ATS80HD & CP4500.) You will also find that not being able to zoom without removing the adaptor (or at least part-removing it) is no big deal. You will be taking the adaptor of and putting it on again all the time to aquire birds and to get your focus 100%, and just to enjoy the bird, so having to part-remove it to adjust the zoom is no problem.
In any case, you'll almost always find yourself working at 20X for maximum clarity and the fastest possible shutter speed, and the in-camera zoom is ample to allow you to frame your shot. If you need more zoom than 20 * 4 = 80X, then you are probably too far away from the bird to get a decent picture anyway, unless the bird keeps very still for you, there is no wind, and you have masses of light.
I'm gradually coming round to the view that 80% of the time, the best way to improve my shots is to get closer to the bird. This means less zoom, which means more photons on the CCD, which means faster shutter speeds, which means sharper pictures.