100% yes, absolutely.
These people do not want to group with you, and you should not force them. They are boosting because it’s the fastest method available; if you break boosting, they either quit or go to the next fastest method (which isn’t dungeon grouping, by the way.)
If taking longer to find groups makes the game ruined for you, sounds like a you problem tbh.