When I had Nvidia 512.15 version drivers for my 3080 card, I had Out of memory Errors and when I tried the latest I still had Out of memory crashes with this game.
But after I used DDU uninstaller after I uninstalled all the Nvidia drivers first in control panel, and Installed the NVIDIA Studio Driver 472.12 version I no longer had game crashes in Diablo 4 beta
Make sure to install old driver with no internet because Geforce Experience will try to search fore updated drivers after completing the install.
Why not try the newest 531.29 drivers? They came out on the 14th. I didn’t have any problems with them and I’m also using a 3080 on Win11 @1440p
that was the first thing I tried and it actually was crashing me out of the game as well, I did find it strange it would do that but it did!. the older 472.12 worked!!!
I’m editing this because I forgot to mention the newest driver 531.29 was overheating my card and fans not spinning right as well and I had to use MSI afterburner to force run the fans to help lower the temps but did not help. Why do I think so that Nvidia is tweaking latest drivers for their cards to run hotter and harder to try to break them on purpose so we have to keep paying out big money 
Ahh yeah maybe. My system is on water so I don’t generally notice temp changes like that. MO-RA3 is a beast.
The driver did seem to work for me. Sucks it is causing your card to overheat.