The general recommendation is 300 metres in the direction of shooting.
Not many cartridges will project their shot that far in still air, but range can be considerably extended if you have a strong wind at your back.
The safety area is also best if it is open countryside, devoid of footpaths, and any obstructions which could obscure the presence of people, such as folks out walking dogs or, in the case of a working farm, farm employees.
People are rarely harmed by spent shot falling on them at extreme range, but it does frighten them badly – particularly if they have no knowledge of the shooting sports.
Complaints, even if they are unjustified, can be bad publicity for the shooting sports.
There is also, of course, the possibility of being sued, or members of your party accidentally injuring each other, and for these reasons, however safe and experienced you are, I suggest that you should all join one of the main shooting associations, if you are not members already