You may also want to go into settings and disable friends “quick joining” your games. If I remember correctly only “real id” friends can join, but it is possible for someone you’ve added to your friends list that you know in real life to instantly join the game with that setting enabled.