This is copied from a guide on vanillagaming
Players have resistances to one or more spell-schools. In battlegrounds players may have shadow protection, mark of the wild, paladin auras or totems that increases player resistances, so having spell penetration is quite useful in PvP and should not be forgotten. However this guide will focus on spell penetration from a PvE-point of view.
The trash mobs in Molten Core will have high fire resistances or even immunity. Mages would spec Frost there and so spell penetration would only be neccesary for the bosses.
All bosses have resistances to all schools (fire, shadow, nature etc.), different numbers depending on the school and the boss. Some bosses are more resistant to Shadow, some are more resistant to Nature, etc.
This boss resistance means that a part of your damage will be reduced unless you find ways to reduce the enemys resistance.
Warlocks can cast Curse of the Elements and Curse of Shadows
When these 2 curses are on the mob (you will need 2 warlocks), Shadow, Arcane, Frost and Fire resistance on the target is reduced by 75.
It is not possible to reduce a player or monsters resistance to less than 0.
Resistance comes after Hit
When you cast on a mob, the first thing that happens is the Hit roll. Meaning that you need Chance to Hit to overcome a level difference for example. This is before resistance is added into the picture.
As a caster you will -always- have 1% chance to miss with your spells.
Vs target of your own level: 4% chance to miss. 3% chance to Hit is needed for cap. (pvp)
Vs target of your own level +1: 5% chance to miss. 4% chance to Hit is needed for cap.
Vs target of your own level +2: 6% chance to miss. 5% chance to Hit is needed for cap.
Vs target of your own level +3: 17% chance to miss. 16% chance to Hit is needed for cap.
Spell Penetration will only increase the effect of spells that already have succesfully landed on the target. However, even spells that land have a chance to do nothing at all if the mob resistance is very high. That’s what spell penetration is for.
Miss is when you don’t have enough Chance to Hit.
Resist is when you don’t have enough Spell Penetration.
That being said, our combat log doesn’t make it easy for us to see what’s going on. For some reason even when a spell misses, it will show as “resisted” in the combat log. Keep this in mind when you observe combat logs. But for the sake of understanding the mechanic we’ll just have to be aware that there is a difference.
Binary and non-binary spells
Some spells will either do their full effect or none at all. It is not possible to have 50% of your counterspell be resisted.
Binary spells will also be affected by the targets resistance, but their chance to resist will be the “average dps lost” percentage.
Casting a binary spell on a mob with 100 resistance will mean that it gets fully resisted 25% of the time.
Casting a non-binary spell on a mob with 100 resistance will mean that it has 1% chance to be fully resisted, 4% chance to deal 75% less damage, 19% chance to deal half damage, 47% chance to deal 25% less damage, and 29% chance to deal full damage.
Examples of Binary spells:
Death Coil, Fear, Howl of Terror, Banish
Frost Nova, Counterspell, Polymorph
Mind Control, Psychic Scream
Mind-Numbing Poison, Crippling Poison
Examples of Non-Binary spells:
Shadow Bolt, Searing Pain, Immolate
Fireball, Arcane Missiles, Flamestrike
Mind Blast, Holy Fire, Mind Flay
Deadly Poison, Instant Poison