Bugged for god knows how long since none of the spirits even move anymore, you have to hope you can simply walk through them and be quick enough to get past and make it to the top before being caught, which I have yet been able to do.
I had the same issue. I cleared the cave and got the flame buff from the blue fire and was able to re-enter.