Definitely not – some of the best clay and game shooters in this country are also deadly with a rifle when it comes to lamping foxes, rabbits and deer stalking.
True, shooting the two is very different because you deliberately take aim with a stationary rifle whereas you must keep both eyes open and ignore the shotgun as it moves, and keeps moving, when the trigger is pulled.
If I were you I would play a little safe by getting thoroughly proficient with the shotgun first before doing too much with a rifle and getting accustomed to ‘aiming’ rather than ‘pointing’.
Some rifle shooters I’ve met who come into shotguns later in life do find it more difficult to stop ‘poking’ at a moving target.
Once learnt, shotgunners, on the other hand, rarely lose the knack of swinging.