That literally is the way to make it more fun imo.
While I am strongly against enemies oneshotting (and some enemy dmg is part of what needs a nerf!), 5 hits to die is not exactly quick. That is quite some time to drink a potion etc.
Are you making sure to CC enemies? And well, avoid some of the attacks.
CC will go a long way.
Certainly not a bad decision.