You’ll probably have to ask Microsoft these questions since it’s not Blizzard’s software. Maybe they have info on their site of possible ways your IP can become exposed.
Unless you were assigned a static IP, you may be able to change the IP on your own by keeping the modem offline for a bit. Sometimes “a bit” can be 5-10 minutes, other times it may take several hours, so try doing it overnight if the shorter period doesn’t work.
Make sure your router supports VPN setup, or you feel comfortable setting up a tunnel through a computer’s VPN connection. As far as I know, there is no direct VPN client support on the Xbox.