I finally found something that worked for me:
Switched Shader Cache to “Unlimited” in NVCP.
(And Power Management in NVCP to “Maximum Performance” in NVCP for Diablo IV.exe.)
(I noticed switching Textures from Ultra to High back to Ultra fixed the issue. But since I switched Shader Cache to Unlimited I haven’t needed to switch the texture option back and forth. Been working for a couple of hours.)
Setup: 3080 ti, i7 8700k, 32 GB 3600 mhz, SSD.