First, Happy Valentine’s Day.
It makes sense that the dataset had limitations due to its age that presumably affected the seasonal data more than non-season. If the season and non-season data were taken at the same time, it means that the season data was 2 weeks old as you stated and the patch 2.6.7a data was about 3 weeks old. We know that there is a dramatic falloff in the number of non-season players once a new season begins. This fact also contributes to the non-season data being less reliable than at the end of an era. I am glad that Blizzard is continuing to assess the data.
Necromancers were the most obvious example but even crusaders which is a popular class “dropped” in the early season 19 analysis, highlighting the difficulties in data transformations using the selected datasets. I simply hope that Blizzard continues to look at the data in rigorous ways. It is not trivial to model/scale data without introducing artifacts and to take into consideration class popularity and player biases. To be done correctly, this is a Herculean endeavor.
Needless to say, I have played around with the data myself from era 11 vs. era 12 in many ways. One thing that I interrogated was DH and WD that did not receive buffs to look at the performance of different analytical methods. The expectations is that the 5K class performance would be identical since there were no changes to these classes during this time. Some analytics/statistical methods perform better than others given the reduction of the number of top end DH and WD where there seems to be about ~80% and ~50% reduction, respectively, in players in non-season that posted their high GR clears in the non-season leaderboard.
I fully agree that people can manipulate data. In academics, we have a saying there are liars, darn liars and statisticians. One advantage of published scientific research has is that the underlying datasets are made publicly available (for publications supported by government funds or in respectable journals), is subject to scholarly peer review before publications, and the statistical methods are explained in detail. Even with all these safeguards in place, many scientific studies are not worth the paper that they are printed on and have erroneous conclusions due to limited/faulty datasets and/or statistical analyses. Given this, there often are post-publication reviews for questionable studies that are done and some publications are retracted as a result.
Needless to say, Blizzard is a company and has every right to keep their information proprietary. From a personal standpoint, it was not my math that was wrong when you responded to me. My math was correct given the dataset that I clearly stated was being analyzed. My other issue was that the numbers did not make sense intellectually. If there is analysis that seems counterintuitive, then additional peer review is in order. For example, if the seasonal data was too immature to use, then it would have been better not to post the results at all.
I agree that people will use data/manipulate data to fir their arguments. It is important to note that this goes beyond individuals but to groups of individuals and corporations. That is why it makes sense to question in a respectful manner what has been provided as information.
Confirmation bias or motivated reasoning is something that we all suffer from. I am no exception. That is why I tend to look at data using multiple metrics to see how parsimonious the conclusions are. It sounds like by taking multiple snapshots Blizzard is trying to do these same things and presumably using multiple metrics where the data can be parsed in different manners to see if the conclusions are consistent. Altogether, these steps will help reduce erroneous findings.
You likely have noticed that when I give numbers I always try to explain the source of the data and how the data was analyzed. This way the forum community can double check (if they desire) or critique as necessary.
With more mature datasets, the analysis will be better. My initial comment was that there seemed to be something off with the data/analysis. I am pleased to see that there is recognition that the data provided was meant as an example, not “snap shot of truth.” My initial post was that there seemed to be something off. To me, it makes sense given your comments. I sincerely hope that if the analysis changes with better datasets and better ways to deal with class popularity that balance changes are made with this improved information, if warranted. With D4 on the horizon, it would be nice to have a well-balanced game (perfect balance is impossible) as discussed in the blog post.