It’s more than one vote. When you single someone to vote against, you not only vote against them every time, you also vote for someone else. So the gap just widens.
Do this as an experiment: queue with a friend and tell that friend to always vote against you no matter what. See how often you qualify.
Three, actually. It’s a 6 people group and one is you, two can’t vote.