Jade 2 piece Haunt not working on crowds

I have Jade Harvest 2 piece, Haunt Resentful Spirits (2 spirits), and Haunting Girdle (extra spirit).

The 3500 seconds damage works great with 3 or less enemies but when cast into a crowd, all the spirits stop working. I can see everyone with a Haunt on them but casting another one doesn’t do anything. Creeping Death passive has no effect either.

What am I doing wrong?

EDIT: I’m not sure but it seems if a spirit doesn’t land on the enemy I’m targeting, none of the spirits will do the extra damage.