So, I went strolling through some of the posts in the WoW Classic Support and Bug report forum and I came across this post by Aggrend - CLASSIC-Crashes When enabling VSync
I disabled Discords overlay feature and… the game runs fine in windowed mode now, no crashes. I’m not sure what the relation is between the two programs, but if anyone else is having this issue, you can try suggesting this as well.