Set the paging file to a fixed size. Do not use different values for minimum or maximum settings. Set both boxes to the same value, so the paging file isn’t dynamically trying to adjust when it thinks it needs to.
Exact instructions here:
-Diablo IV has run out of memory - #103 by DTMAce-1687
Now, that may not help you in your case, but it will help the system performance overall.
Here’s another possibility:
Perhaps your card is having issues when the demand is high. Have you run any stress tests on your hardware to see if there are any issues, such as Furmark?
You can use these tools to run stress/diagnostics on your GPU to see if it has issues:
https://geeks3d.com/furmark/
https://benchmark.unigine.com/heaven
https://benchmark.unigine.com/valley
https://benchmark.unigine.com/superposition
https://www.passmark.com/products/performancetest/
See if any of those show any issues.