Chiming in too, this is bugged for me. I’ve tried reloading, quit the quest repeatedly, tried grouping up with others, and nothing works.
I cannot interact with the dirt piles at all period.
And this is the first quest that I’ve ironically hit a bug on.