D3 Set armor bonus don't always register with mobs

Playing season 28 right now in D3 on a monk wearing the Patterns of Justice set. I have noticed that when I play, I can wipe mobs very well in T13, but then all of a sudden I am doing like no damage to a trash mob (or champion or elite…) after I literally cleared a rift map of champion and elite mobs. I’ll get rofl stomped and one shot to death a couple times because of this, then all of a sudden I’m able to dish out the damage like normal again and resume clearing mobs. It’s like the set armor damage bonus increase randomly just quits.

Anyone else experience this? I know it’s not an affix ability as trash mobs don’t have any.