Probably not. While imperfect bullets are obviously not going to fly quite as true, unless the damage is severe the difference is undetectable at normal hunting ranges and under field conditions.
Long range target shooting is another matter but our damn-fool legislation doesn?t allow the use of soft-point bullets for that.
However, there are a few points arising from your question.
First, I hope the rounds are only in your pocket while you are actually out with the rifle otherwise you could be in breach of Condition 4(a) on your Firearm Certificate.
Second, spare rounds are much better carried in a purpose-made pouch or ? as I usually do ? in one of those handy little plastic clips in which Federal ammo is packed.
And third, if in doubt, use damaged rounds for target practice rather than hunting.