I agree with you, I dont much care for the ageism thing but I can understand that some people might find it challenging to get to that point .
Tbh I think is less about age and more about WHO you dealt with. When someone talks about being irked around 20s ppl what I get is they dont wanna deal with a particular subset of people that might be incompatible for other reasons but that they associated with an age group.