I am on the Xbox Series X and also have this issue. Sometimes ”zoning” whether by going through a portal or a waypoint helps the majority of the time. I can have it in the middle of a fight or in the middle of any town it doesn’t seem to matter. I had times where I am in my inventory and it’s not working but I’ll go into a trade menu and all of a sudden it is right afterwards. There doesn’t seem to be any correlation to any action that I take to cause the issue. I’ve had multiple belts on multiple characters have the issue.
I noticed that someone said changing it to quality instead of performance on the PS five stopped it from happening however I’ve always had mine set at the quality setting.
My Xbox is completely up-to-date with the newest updates. And the game is completely updated as well.