Xbox Controller Bug - X button doesn't work at Rift start

Issue: There is a 2 to 3 second delay in being able to use the X button on my Xbox controller at the very beginning of every rift I enter.

Details:
I started playing the current season of D3 on my Xbox One X last Sunday (July 24th) and I have been having issues using the skill assigned to the X button for a brief period of time as soon as I enter a Rift. It is only a brief period of time, but it can cost precious seconds and use up my cheat death when an Elite group is near the rift entrance.

It doesn’t matter if it is a Nephalem Rift or Greater Rift, it happens in both. It does not happen when teleporting to areas to do bounties and I haven’t noticed this behavior in any other part of the game.

When I first enter the rift I can immediately use any skill assigned to other buttons, but not the X button. I had Cluster Arrow assigned to X and got beat up pretty bad a few times when the elite group was close enough to the rift entrance to start attacking me right away. With advice from others I swapped Cluster Arrow to the Right Trigger and moved Sentry to the X button. Then I was unable to drop Sentries immediately upon entering a rift, but I could then use Cluster Arrow right away.

Note: It happens on my Xbox Series X and on my Xbox One X. I’m using different controllers as well, so it is not controller specific. Considering I’m using two different systems and different controllers, the issue either has to be with my character files or with the game. Any help would be appreciated.