The reason why you think you should have more keys is because for season 1 we could farm a lot of keys before delves opened, so we had like 15 to use when the season finally opened.
Further more you can get more keys from key fragments, which can be a reward from world quests and from doing the world soul stuff.