Why the reddit survey may not be accurate or trustworthy

Diablo II: Resurrected General Discussion
1

First off let me start by saying I already know this issue has been brought up by other people and I wanted to give my opinion on it.

Here is a video about the survey I am talking about, MrLlamaSc talks about his opinion on it and goes over all the questions the survey asks

Here is why I do not think the survey is accurate or trustworthy. A lot of people know where I stand on most of the changes in the survey but everything I have to say about it should at least be considered. A lot of people may say that I am only saying I don’t trust this survey because I don’t agree with the results but that is not the case, I wouldn’t waste my time doing this if it were.

A lot of people active in the forums are using the survey as a reference to how the community of Diablo players feel about the changes. That logic is skewed.

I want to mention some points:

  • The survey is not done by anyone official

It is done by a random Reddit account. Credible or not, that’s one reason not to trust it or use it as if it is evidence of how the community feels

  • Every post against changes in these forums has more likes

The D2R forums is a great way to tell how the community feels about D2R. Reddit can be accessed by anyone all over the world but only people who purchased D2R can post in these Blizzard D2R forums. In these forums, every post I have seen saying why D2R should not be changed in some way has had anywhere from 2x-4x as many likes as the people who want changes. This should be considered no matter what you feel about changes because if what I am saying is true then it does not match at all with the survey results.

  • All you need to take the survey is an email

Above I mentioned that anyone can access Reddit and therefore the Diablo 2 Resurrected Survey that was posted, despite everyone most likely being a diablo fan. Once accessing it, you need to provide an email in order to be able to take the survey. If somebody wanted they could easily take the survey multiple times

  • Participants probably didn't actually play Diablo 2

All that you have to do to prove you played diablo 2 and that you are a fan of diablo 2 is check a box that says you played it. There is no way of knowing whatsoever how many participants actually played diablo 2 and how many even played past the first act. Anyone that doesn’t play the full game at least once probably shouldn’t be doing a survey on changes.

  •   The post was made in a r/diablo forum (not diablo 2)

This was not specific to diablo 2 fans or people who pre-ordered diablo 2 resurrected, this survey was on a popular diablo forum that mentioned information on diablo 4 and diablo 3, as well as the announcement of D2R. This could have easily been a community of just as many D3 players as D2 players that all saw the survey. While it could show us how diablo players in general feel about changes in D2R, it does not mean that they have actually played D2 or understand the questions they are answering if we can’t tell who is actually taking the survey.

There are a lot of additional reasons why the survey might not be accurate but these I think are the main reasons why, in my opinion. I know that this survey wasn’t only posted on Reddit but was also posted on the blizzard forums and certain youtube channels but again that is just another reason why we don’t know if it is at all accurate or not. When you allow anyone a chance to take the survey we have to just trust they aren’t lying or messing with the data, Including the author of the survey himself.

I think people should stop pointing to the survey as if it is of any real indication of how the Diablo community feels about changes to D2R. There is a lot we don’t know and it could be horribly inaccurate.

I also think that only people who have actually played D2 should have a say on changes because anyone who hasn’t played wouldn’t understand many aspects of it.

16 Likes
2

The most liked post I could find on this sub-forum is in favor of adding a charm inventory:

5 Likes
3

Your assessment seems to be in error.

About the Reddit poll

  1. It has selection and sampling bias as any internet poll
  2. Although it is a possibility that the poll results were mis-reported to what respondents voted/stated, the overwhelming evidence suggest the results were accurately documented.
  3. Polls do not need to be done by “anyone official”. The key thing is that the person who did the poll has integrity. As discussed later, all evidence validates that the results were reported with integrity and accuracy.
  4. Dependent on the proposed change, the D2R forum shows that some changes are very popular (e.g. reopening cow level) while some are far more contentious (charm inventory and optional personal loot). The reddit poll results correlate with the D2R forum as well as the reddit diablo forums. As noted, the second most “liked” “change” thread OP was about a charm inventory, highlighting that the OP’s personal bias and inaccurate assessment. The most liked thread is about changing the amazon face.
  5. To vote, there were safeguards in place to avoid ballot stuffing. This is a good thing, not bad as OP suggests.
  6. Although self-reporting can lead to respondents lying about things, it is incredibly far-fetched to think that in a poll about D2R that respondents were not D2 players. This poll was advertised on reddit diablo/diablo 2 subreddits, this D2R forum, d2jsp, and Rhykker’s youtube channel. The idea that this poll’s result was the consequence of a mass conspiracy of D3 players is far-fetched lunacy. This is the same type of conspiratorial nonsense that some claim that the D2R forum that requires pre-purchase of D2R is being overrun by D3 players who paid $40 to troll/ruin D2R.
  7. The survey included 86 pages of comments. These were clearly written by many, many unique individuals. In this is my comment verbatim, highlighting the post did actually contain the real words of respondents. These quotes were all over the place in terms of content, writing style, topics, and game knowledge.
  8. To my knowledge, no one has provided any direct evidence whatsoever that the poll reporting is not credible in any way. Even streamers that disagree with the results relative to what they want in D2R agree that the poll is credible.

I think it represents qualitative what the D2 community wants and is incredibly helpful to cite. Irrespective, Blizzard is doing their own survey, which will yield similar results.

13 Likes
4

Of course it isnt accurate, i didnt know about it so i wasnt voting.

7 Likes
5

It has selection and sampling bias; however, there are 4 questions:

  1. For respondents, were the results accurately reported? The answer to this is apparently yes.
  2. Does it reflect the largest publicly available sampling of D2 players opinions? Yes
  3. Do the results correlate with opinions across multiple forums? Yes
  4. Does one need to survey everyone to get a sense of what the larger interrogated population want within some error limit? No
5 Likes
6

I dont agree, only 4k players voted, thats incredibly low number.

6 Likes
7

Not really. The issues of selection and sampling bias are much more relevant than the size of the sample. This survey is not a statistically valid random sample.

If it were, you can calculate the sample size to have an alpha rate = 0.05 and 80% statistical power to discriminate statistically a population that favors something by 80% versus the sample which is a smidgen difference (specifically 75%). In that case you only need 452 respondents. If you wanted 95% statistical power, only 908 respondents would be needed.

4352 respondents is a large population in terms of statistical power. The question is that sample truly representative of the population. The answer is no. It over represents those who are passionate about D2 who visit forums/youtube and were willing to spend he time to complete the survey. The results most likely overrepresent D2 purists. Also, I bet non-forum visiting D2 players in general care far less about whether some changes are introduced.

See:
Sample Size Calculator (clincalc.com)

7 Likes
8

It is a very good sample size even for scientific standards.

Here is a table that shows how many participants a survey needs to reach an error margin of ±3 %, ±5 % and ±10 %:

Population ±3 % ±5 % ±10 %
500 345 220 80
1.000 525 285 90
3.000 810 350 100
5.000 910 370 100
10.000 1.000 385 100
100.000 1.100 400 100
1.000.000 1.100 400 100
10.000.000 1.100 400 100

Even if there would be more than 10.000.000 D2 players about 1000 participants would be enough to receive very accurate and representative results.

6 Likes
9

That is much more concise and straightforward than my post.

The point is the sample size is large enough. The question is sampling/selection bias.

2 Likes
10

I guess I should have been specific about what I meant. Ive been working a ton of overtime and haven’t been able to double and triple check every single thing. However, I do still feel like my claim still has weight to it for other changes

11

Your argument that the forum posts about not changing anything have more likes is statistically completely irrelevant.

These forums are only visited by extremely devoted and hardcore fans, a large amount of which are people who are pinning for an identical game experience to what they are currently still playing. ALL video game forums are like this- they are visited by the vocal minority, and do not, under ANY circumstances, represent the whole playerbase.

A game’s playerbase can usually be segmented something along these lines:

Vast majority: silent; simply plays the game.
Very small numbers: active on external forums, like reddit or discord.
Even smaller numbers: active on the official forums.

Most players will never visit the official forum of a video game unless they need help with a problem/bug. That’s the reality of every single game that has ever been released on Steam, and if you do some Google-fu you can find the paper they published on the average percentage of any given game’s playerbase that actually engages in the official discussion forum for that game. Spoiler: it’s tiny.

So to backtrack to what I started my reply with: it means absolutely nothing that, on this forum, there are more people who are against change than there are who are for changes. Because the playerbase liable to come to these forums is biased towards one direction.

The reddit poll is more accurate than a poll run on these forums would be, as a very simple result of this fact.

Whether the reddit poll is more accurate than an official poll conducted by Blizzard would depend on things like where it’s conducted, for how long, and what qualifications you need to answer the poll.

But yeah. Sorry, but your base argument is 100% false.

4 Likes
12

I apologize I don’t have as much time as you but if there’s anything I can count on its you showing me where I could have been wrong in my thinking even if I think you go to extreme levels to disagree with me.

Just because it seems crazy to think something doesn’t mean that it shouldnt be considered. With 99% of participants saying they have played Diablo 2 but then MrLlamaSc looking at the results and saying that some people are insane to vote for a certain change suggested (or something close to that) it is exactly how I felt reading through them. I went and looked at the survey and took it and I could take it multiple times. I could say I played Diablo 2 even if I did or didn’t.

I am not saying it had to be an official survey, I am not saying it had to be perfect. But with how easily it was for me to singlehandedly go in and manipulate the data it suggests that it may not be accurate at all. Especially when a ton of people on YouTube channels and on reddit are saying they agree with MrLlamaSc

To me it simply doesn’t add up and though it seems far fetched there are still a lot of reasons why the survey wouldn’t be accurate

13

The key here is really pointing out that people should not have voted which havent completely played the game. If this was a complete new game, forget what im saying, cause everyone’s opinion counts. But for a remaster, honestly, and sorry to say that, but not every opinion counts ! Because otherwise we would make a complete different game out of it.
Now that beeing said, there is at least one particular question in the whole survey which proves that at least 43% (and i bet its way more) of the voterd have not fully played the game to know what they are voting for. It is question 21 “should there be an additional way to respec”. There is already an additional and infinite way to respec in the game, the answer here can only be no !!

So yes as this is a remaster and not a remake or new game, this survey is pretty useless in my eyes

1 Like
14

I for one 100% agree with MrLlamaSc in his video. to many buffs to make D2 easy just like D3. If you like D3, go play D3. Some QOL changes i like , 10x10 inventory , Maybe stacking gems and ruins, colorizing drops to better see ruins gems and charms, infinit cow levels. the qol changes i say yes to.

But the buffs to the game play like second inventory for charms, bonus XP, and MODS being made part of the game is a big NOOOOO!!!

5 Likes
15

Thank you! I know some of what I said could be wrong or maybe it could be a little far fetched in some cases but overall I found the survey really easy to manipulate and I do believe that MrLlamaSc really is a perfect example of how a lot of D2 players feel.

2 Likes
16

You know he’s arguing against the most extreme positions, that the survey questions didn’t even ask. Minor buffs are not new meta making buffs, why do people infer this from questions that isn’t even what it was asking. It’s almost as if people want it to be that because that’s the only reason argue against or just makes their arguments easier. In fact you and others are manipulating the results to look like your worst nightmare, because you didn’t like the results.

Another thing he then in another video explained his positions in detail, which I actual found that he agreed with survey results. Except personal loot.

Its a long video though but here it is.

1 Like
17

I know there’s obviously a lot of angry people in these forums who want to point fingers and assume but I honestly don’t believe the survey results because I myself could easily manipulate them.

It’s not that I am trying to make it seem like my worst nightmare, it’s simply that there is undeniable reasons why the reddit survey very well could be false and everyone keeps acting like it is fact. People are quoting the survey saying stuff like “well 64% of players actually want this…” or “99% of participants played D2, and they voted for this change”

Just a click of a button and you can represent the D2 community even if you didn’t actually play it, just another click and suddenly you are voting for personal loot.

Consider this scenario… a D3 player really wants personal loot and people in these forums argue against it and that player then goes to the survey and says they played D2 and votes for personal loot, representing D2 players.

Maybe that scenario isn’t very likely but it only takes one guy who’s angry enough or just wants to mess with the results to create multiple emails and take the survey over and over again.

My little sister has only played D3 and even she wanted to take the survey.

Perhaps all my points aren’t perfectly spot on, perhaps I have failed to completely convey my opinion in the best way possible. I am honestly working 67 hour weeks and barely have time to keep up. But if I can just go in and manipulate the results of the survey how hard is it to at least consider why it may not be accurate?

2 Likes
18

Doesnt have to be the case. Respec was added in later version of games and a lot of people is using plugy, i dont think so many people know about cubing essesnces as you have to hunt all act bosses for it which i dont think SP are doing. They simply use respec from plugy and they dont realize essesnces are there to do it also.

I for example never used that repice.

Or maybe people just want unlimited respec as in the plugy without hunting for essesnces.

19

so then i’m still right, no? these players you are talking about did not “fully” play the game and that is essentially my point. These Respec Tokens are existing for decades now, so the argument that someone didnt know about it doesnt really count, they simply didnt fully play the game

1 Like
20

I agree, if people don’t know aspects of the game that have been out for over 10 years then they really have no right to say they “played the game” and then advocate for changes and represent D2 players that have

1 Like