Yeah Inner Sight vs Conviction defense reduction depends on some factors, I think they’re fairly comparable overall though in most regular scenarios.
Say you’re doing purely unique packs in ladder/single player (monsters have lower defense in non-ladder) TZs, the highest regular monster (non-boss, non-Uber areas) defense you’ll encounter is 2675 defense from level 96 (character level 94+ TZs) scarabs (scarabs have 145 AC(H) in monstats.txt which is multiplied by 18.45 in monlvl.txt), or 5350 defense when doubled by a Stone Skin unique.
A level 94 A1 merc will have level 31 Inner Sight without any skill bonuses which is -1715 defense, bringing those scarabs down to 960 defense or roughly equivalent to a 64% defense decrease. Every additional skill level in InS is another -100 defense or roughly equivalent 4% decrease, so it’s not really that far off from level 12 Conv’s -83% defense especially if you can get some skill points on her. Again that’s assuming the highest defense monster type, InS has an even higher equivalent % against anything else.
Now like you say for higher defense (like Stone Skin where InS applies after their defense is doubled) monsters then it rapidly turns to Conviction’s favor except in the case where a monster has skill enhanced defense%, in which case Conviction’s -ED% is simply subtracted from their skill ED% and is effectively negated. There’s only a handful of monsters with skill ED% though, most notably the Ancients with Madawc’s shout and Uber Izual and Uber Baal with their Chilling Armor (fun fact, they were buffed when Blizz buffed Chilling Armor in patch 2.4
). In these niche cases InS applies before the ED% so it is highly effective against them in particular.
Having said that there’s also the consideration of the skills mechanics themselves, InS has to be casted which the A1 doesn’t always do consistently at the start of a fight or at all (higher attack speed is better here for that reason, faster attacks = more chances to cast InS over a given period) while Conv is semi-passive, but InS has a greater radius and also stays even if she dies so it can be of some use in scenarios like the Ubers where any merc will likely die. Conv also reduces enemy elemental resistances of course which is helpful for any elemental damage you may have, also makes cold damage chill duration last much longer if the monster isn’t cold immune or has 0 chill effectiveness (cold resist also reduces cold length, so lower cold res = longer cold length).
Aaaaand that’s a pretty long wall of text, so…
TL;DR: I think both are pretty comparable and will get you or your minions to a high/very high chance to hit, giving a bit of an edge to Conviction against higher defense mobs (like bosses and Stone Skin) and overall better skill mechanics being semi-passive without the needing the merc to cast it first. Inner Sight is still a great skill to reduce defense though and has a niche advantage over Conv against the Ancients and Uber Izual/Baal, might also have situational use against the other Ubers or other scenarios where a merc is likely to die and thus not be able to constantly supply Conviction.
I was admittedly somewhat tunnel visioned writing this so I didn’t really go into more detail about the opportunity cost from not having another free aura that comes with an A2 merc or stuff, just comparing InS and Conviction in a vacuum.