It’s not that simple. Sure an index is great for mapping a forest of trees and storing which trees are within x yards from each, but monsters move around a lot. You would need to update the index every single time a monster moves to check if it entered or exited a specific range of another monster; this would be very inefficient.
Since the lag increases with the monster density, it’s likely they’re not using indexing and just initiating a search tree per hit. A slightly more efficient method could be to do a search tree and index per frame that a monster takes damage from a valid area damage source. This search and index would then be used only for that one frame to apply all area damage sources from everything.
In short, that idea would make the area damage calculation faster if there are many targets getting hit on the same frame, but would make a lot of extra and unnecessary calculations if there are only a few. So while it would reduce the extreme lag in high density, it might introduce lag in low density.