It doesn’t matter at all for most people. It mattered to me because I read a post on reddit where a guy talked about his pc building experience, was able to recognize he was having over heating issues, and this guy proceeded to curse me out for 2 hours straight on his twitch stream. For the great crime of giving the guy some advice, he defamed and slandered me in front of 200 people. So now I have a personal stake in making sure everyone understands the x3d CPU variants are not the best purchase. Hopefully we can educate some young gamers on the proper way to analyze market trends (using data, from multiple sources, and questioning the biases of said sources) as well as treating celebrity endorsements as if they have a negative correlation with quality (because they generally do).
Generally the way I analyze data from multiple sources is to rank them based on bias displayed from 1 to 10. If their methodology is flawed, they rank lower. If they display favoritism, I rank them lower. I then average their scores by weighting their bias. Let’s say we have 3 sources and one scores a 1 and the other a 5 and the other a 9. Let’s say the first says the 5900x3d is a 10, the next an 8, and the last a 6. We first add up their bias scores and divide each bias by the total. That’s 1+5+9=15. The weights are then 1/15, 5/15, 9/15. We multiply these weights by their reported scores for the CPUs, and average it. that’s 1/15*10=0.67, 5/15*5=1.67, 9/15=3.6. The average is the sum of these numbers: 5.94.
If we averaged this normally, we’d come to a score of 8. That’s because we didn’t weight the results to account for bias. This test simply gives more weight to the data sources that display less bias. How do you score their bias? You do a questionaire like this:
- How many times do they employ preferential reasoning for a conclusion?
- Is the sample size large and robust?
- Are there flaws in their methodology?
- Do they have a motivation to be biased?
- Do they disclose their bias and take measures to mitigate their bias (blinding etc)?
For every instance where they display bias, it counts as a 0 and as a 1 otherwise. You then add up these scores for each data source, normalize all the scores to a 1-10 scale. You can do this by subtracting the minimum score from all scores, then dividing by the range in scores (max score minus min score) then multiplying all of them by 10.
Let’s say a data source displays bias three times. Let’s say they used a small sample. Let’s say their methodology was flawed in 2 ways. Let’s say they admit they are biased, and that they make an effort to minimize it. That’s 0+0+0 + 0 + 0+0 + 1 + 1 = 2. Now let’s say we did the same thing for 2 other sources and they scored 4 and 7. To normalize this, we’d simply subtract the smallest score from all scores so that they become 0, 2, 5, then divide everything by the largest to get 0, 0.4, and 1, which we multiply by 10 to get 0, 4, and 10. Next, we account for our own bias by adding 1 to the lowest score and subtracting 1 from the highest to get 1, 4, 9. Then we perform the test as detailed above.
What’s brilliant about this method is that the sources that display obvious bias will count for less in the final score. If someone says 10 out of 10 or 0 out of 10, it won’t matter if their bias score is 1 out of 10 – in a group of bias scores 1, 9, 9, the 1 would contribute to 5% of the average instead of 33%. So what you’d be saying if you applied this test is that you are going to assign more importance to the less biased sources and less importance to the more biased sources.
Let’s do this for one of the sources provided by upatree.
- Starts the videos with sponsors. Indicates a financial bias.
- Immediately find a flaw in his methodology. He says the best way to test CPU performance is by keeping the GPU constant across all tests (randomized hardware assortment is a better indication).
- He’s testing with different RAM types for each, an obvious flaw in methodology.
- He doesn’t test multiple CPUs for each test (could be a bad CPU).
- He doesn’t test on multiple operating systems.
- He does test across a broad range of games.
- He doesn’t test a varying rendering resolutions.
- He mentions the power consumption is higher for the i9, makes no effort to account for this as a source of bias.
- No indication on if the same CPU cooler is used.
- He gives a budget recommendation as an alternative to the 5900x3d.
- He makes no attempt to measure the standard mean error.
Total score: 0+0+0+0+0+0+1+0+0+0+1+0= 2. A very low objectivity score.
Let’s do the same for passmark:
- No sponsors.
- GPU is randomized across tests.
- RAM is randomized across tests.
- CPU cooler is randomized across tests.
- Multiple CPUs are tested.
- Multiple operating systems are tested.
- A broad range of tests are used.
- A broad range of resolutions are used.
- Passmark makes a range of recommendations for varying budgets.
- Passmark measures the standard mean error.
1+1+1+1+1+1+1+1+1+1=10.
That’s how you measure bias. You do the process as normal from here.
