There are some… let’s call them “issues” with VoH. The first is the latest nVidia driver apparently changed some important power management settings. So just when your card needs to ramp up it decides to take a nap.
Next the VM profile for VoH seems more beefy than the base game. Definitely make sure your VM settings are optimized for D4: VoH. See this link from DTMAce for details https://us.forums.blizzard.com/en/d4/t/diablo-iv-has-run-out-of-memory/130683/103
Lastly there’s a feature on Windows 11 that doesn’t play nicely with D4 on all graphics cards. It’s called DirectStorage and turning it off seems to help some people with stability. Add “-disableds” to your D4 launch parameters to disable it.