Often side quests will have decent rewards (or unusual rewards) to make up for them being otherwise less appealing than the main quest line. Unfortunately it is a bit hard to answer the question without knowing what level you are, what zone you’re talking about, and what your priorities are in the game.
If it’s a leveling zone and you’re just trying to get leveled up quickly, absolutely skip them.
If you’re taking your time or it’s your first character, you might want to run down a few side tracks.