There is a balance that the game has to make in order to be able to match making. If they allowed for unfettered avoid as teammate, it would make match making neigh impossible and make wait times rage quitable.
However, three is simply not enough. Being able to avoid someone who is toxic, throwing, etc makes for a less toxic overall environment. Constantly unblocking someone in the hopes they dont play that night so you can block the throwing Sombra is too hit or miss.
I think having it at 5 slots would help ease this pain without harming the match making system. Right now, anything that makes the game more enjoyable (or less un-enjoyable) is a good thing for the playerbase.