Part of it is bad luck with layouts, but part of it could be that you’re slowing yourself down. Keep moving forward and let the dust devils kill things behind you. Don’t stay around long, just keep going.
Shaved a bit off my speed runs by doing that, maybe it’ll help you too.