There is some point to that, I’ve only ever done it on hard mode too. But there are things like oh say… the entire dungeon from walls boss onwards that is going to suck, even if you drop the hard-mode components.
Also, the stage makes no sense with M+ aspects in force.