Actually it matters quite a lot Imo.
The damage is technically the same but if you let the Phantasms stack on a single target then you are not doing damage with your actual Spirit Barrage because you are not casting it. If the target moves away before you explode, it will not take any damage at all. On the other side you can do far more damage by setting up a big explosion in density than you can by spamming.