When your statistical methodology is wrong, I would say a theory based on logic is more likely to be correct. I took the liberty of doing some proper analysis.
We know that there is a suppression on crit auras, so a proper comparison would look at the additional suppression above 3% vs. the amount of aura crit, and then fit a linear regression to that to arrive at three possible conclusions.
-
There is an intercept and zero (or close enough) coefficient on the aura crit, which means aura crit is suppressed by a flat amount
-
There is an intercept and a coefficient on the aura crit, which means aura crit is suppressed a flat amount plus some percentage of aura crit
-
There is no intercept and a coefficient on the aura crit, which means aura crit is suppressed by a percentage of aura crit (What I believe is most likely).
I fit two regression lines, one with an intercept, and one without, the results are as follows:
-
With intercept: Suppression = 1.58% + 0.0228 * aura_crit (95% CI of ±0.003% on the intercept, the slope is not statistically significant)
-
Without intercept: Suppression = 0.1770 * aura_crit (95% CI of ±0.0062 on the slope)
I can provide the Python notebook for review if you’d like, but what this tells us is that we don’t have nearly enough data to determine which is accurate, as you might think it’s 60k data points, but in fact is it only 16.
2 Likes
Unfortunately there was very limited time to do further tests, and we were only able to get a couple of caster logs, one from shaman Ayle and one from a druid. The shaman log appears to show suppression but the druid one doesn’t. Don’t read anything into that, could just be sample size so we just don’t know yet.
Ok I’ll try one more time cause this is very basic. Let’s go back to probability 101 and roll a dice 1 million times, then roll a dice again 1 million more times. You are arguing the sample size of that experiment is 2. But of course in reality the sample size is 2 million. I grant there are some subtleties in aggregating WoW combat data, but saying our sample size is “16” is pretty obviously incorrect. I could take one of those logs and split it in half, and by your logic I’d have 17 samples now. I think you can see the flawed logic there.
As for the data itself, check the suppression column in our data and you won’t see an increase, which there would be if it were percentage based. E.g. our test with 17% crit aura showed a crit aura suppression of 1.91%. Is there a chance the suppression is actually 10% for 17% crit aura? Sure but the chances are remote in the extreme. Scroll down to see the graph and as you can see, it’s pretty flat.
If you need more help in seeing where your code went wrong, post it on our github and we’ll have a look.
Better yet, you can test this yourself on the next stress test. Get 5/5 cruelty and go to town on a +3. Respec and do another test with only 1/5. Post your combat log to our github and we’ll happily review it.
5 Likes
Dice are IID, these crit observations are not, as such you can’t just lump all together and say “these are directly comparable”. You have to treat each situation (different crit values) as individual observations. Yes, if you split one log into two, you’d have two “observations”, but your data collection would be flawed in that situation.
By your logic, I could take 2 million additional swings with a 3% crit rate (and zero aura crit), see 3% crit suppression, fit a model, and by virtue of there being so many more “data points” that fit that criteria, the result would trend down toward 0% additional crit suppression.
Additionally, your theory of a flat 1.8% is not supported by the data as observed (1.58% is, but I’m still skeptical). But like I said before, there is not enough data to reach a proper conclusion. Once raiding starts and logs are more plentiful I’m sure we can fill that gap.
And I’m not too familiar with github, but it doesn’t look like I can upload or modify anything.
Also, you’re being unnecessarily rude and hostile.
4 Likes