If we were having a convivial jar in our local pub, the flippant response would be: “If you can’t carry the rifle, what the heck are you going to do if you shoot a deer?”
But since we are not in the pub and since, in hill stalking, the carcass will probably be recovered for you by someone else, let me try to be a bit more helpful.
A sound moderator will almost certainly alter your rifle’s zero.
The exact amount will depend largely on the weight, ie stiffness, of the barrel and could be anything from a couple of inches to more than a foot, at 100 yards.
You won’t know until you try it.
While a sound moderator certainly reduces muzzle blast and recoil, it does absolutely nothing, as you have discovered, for balance and ease of handling.
Life was ever a compromise.