Spell pen is the most valuable stat when targets have resistance that can be penetrated. The problem is that there is no way to know how much resistance a target has, it’s literally impossible. If you use spell pen and the target has no resistance to penetrate, its a completely useless stat. Since curses usually penetrate most/all resistance anyways, spell pen is considered pretty garbage stat in pve.
In PvP though you wont get curses on your targets and there are a lot of talents and items with resistance so 40-50 spell pen is pretty good to have in pvp… especially against SL warlocks.
The best tool for gauging mob resistance is finding an addon that measures your penetration and logs your hits, then attempts to estimate resistance. I had one in vanilla but ive yet to find a good one in classic. I believe such a tool was used to calculate the following info: