Avoiding xuen historically was strictly a damage loss the majority of the time. If for whatever reason you really hate it hate it that much feel free to not take it!
The vast majority of players really have no idea what they are talking about and SEF is largely fine outside of very niche scenarios.