I have a couple of ideas on how to improve the state of Shadow Priest with this talent tree.
Move to main tree/make baseline (Priest): Silence, Dispersion, Vampiric Embrace
Make ability baseline (Shadow): Mind Flay, Vampiric Touch, Devouring Plague, Mind Sear, Void Eruption
Remove/Rework/Leveling Spell: Searing Nightmare, Surrender to Madness, Damnation
The fact that so many spells that are part of the core kit can be skipped over is a little insane, pun intended. Imagine building a tree where you don’t take Devouring Plague or Searing Nightmare and have no way to spend Insanity.
As far as Searing Nightmare and Surrender to Madness go, they have always been the problem child of Shadow Priest. Ideally, they would just be added as spells you get while leveling.
Searing Nightmare could work as a toggle: When toggled, Mind Sear will drain Insanity while channeling (rather than generating it), and will apply Shadow Word: Pain to all targets, and deal increased damage against targets with Shadow Word: Pain applied to them. This could be balanced by capping how many targets it applies Shadow Word: Pain to, or reducing the damage beyond ~8 targets. The toggle would be disabled manually, or when Insanity is spent. Additionally, talents could be made to empower it to spread more than Shadow Word: Pain, and such.
As for Surrender to Madness, outside of making this only usable when a target is below a health threshold or reworking it to have a different downside besides the player dying, this seems like an ability that is better off being removed, as the playstyle that enabled it to be good, where Voidform wasn’t a damage CD, and instead the player was racing to stay “ascended” as long as possible, is no longer in the game.