Legendary aspects that modify a skill are fine, but generally they are a massive buff to the skill that also can change the way the skill behaves. This limits how a skill can be played and the playstyle of the skill, because you are forced into the playstyle offered by the legendary aspect.
In order to make things interesting and increase build diversity, the base skill should be strong without needing the legendary aspect. The legendary aspect should offer a playstyle change to the skill that can be specialized into, but the base skill should still be viable. Using Ball Lightning as an example, you are forced to play in melee range with the legendary aspect. The base skill is awful as a result, and you cannot play a ranged Ball Lightning build. Maybe the base skill can fire 3 balls to compensate, so you can actually play it ranged if you want, or play it melee with the aspect that makes you fire 1 ball instead of 3, with the upside that they orbit you, guaranteeing more hits.
You can still require us to find the skill modifying legendary aspect before they are available. Make it an empty slot as a 3rd choice for the skill mod in the skill tree, beside the two existing ones. Or let us spend 1 more point after the 2 choices for each skill so we can unlock the legendary aspect socket for the skill after choosing the 1 of 2 specializations.
All other non-skill aspects:
These are by and large generic power gains. I think these are fine to leave on gear and swap them around. They can provide generic upsides to your build, or specialized powers like damage based on barrier. These are fine as is, and provide an interesting way to build into your skills.
However having legendary aspects that modify a skill be part of your gear is a huge mistake in itemization IMO because it reduces build diversity. The base skill is not playable without it’s legendary aspect which is a huge loss in what is actually viable at end game.