The performance modifier is only a small effect, and only happens below 3000. You are compared to other players of your hero/rank/map. See How Competitive Skill Rating Works (Season 10) for a more complete explanation, especially the performance modifier section.