Testing Everything Together: An Experiment Methodology?

WoW Classic Cataclysm Classic Discussion
1

So let’s see, the plan is to have a week to test HvH BGs, where Alliance also get extra rewards, and players also can’t queue for BGs in groups of more than 5.

Did I get everything or is there some other change? I honestly can’t recall.

More to the point, what data does Blizzard intend to gather about the impacts of individual changes, or is the intent only to gather data about how this set of changes plays out together?

Furthermore, given that the Blue post expressed recognition of player concern for the long term population of the Alliance faction, and cited this as the primary reason for testing out additional rewards to incentivize Alliance participation in PvP, how will this week long experiment gather meaningful data about the long term movement of players from Horde to Alliance?

Further still, consider the same question again about measuring player migration from Horde to Alliance, in light of the fact that the players are aware that these changes are a temporary one week experiment, and that the final changes, if any, could look very different.

What meaningful data then, could Blizzard ever hope to acquire from this experiment, and how can it really inform future decisions in a way that is useful?

2 Likes
2

Honestly its almost like they are not trying to find if a solution would work and just repeat a previous experiment so they could claim the good from boosting honor and adding incentives to pvp is due to same faction bg and breaking up premades without providing ranked/rated bgs.

3

My personal reading of the situation is that it is more about managing the appearance of doing something & the appearance of “fairness” to both factions.

Because there is no way they could really hope to draw meaningful insights from such short-term, non-controlled studies.

Another possibility is that they want to create a week for Horde who really want to finish their honor gear to do so, without implementing the change long term.

1 Like
4

Blizzard has no reason to prioritize one faction over another - we are all the same 15$ a month to the company. I can call out Blizz’s BS as much as the next guy, but they are profit oriented company.

The meaningful insights are already there in Blizz’s internal server metrics. It’s clear from the blue post they are looking at

  1. Queue times (hence HvH reintroduction and nerfing of AvA)
  2. Ally BG participation (hence the need for an incentive).
  3. BG winrate balancing (hence nerfing premades).

If the change(s) affects these metrics in a positive manner, then they will stick. Or atleast introduce for as long as is needed to solve the problem (They may remove HvH if the queue times are acceptable, for instance)

5

except no not really good tests would require good control and while doing any of the three separately would be a good way to measure its impact lumping the three together is the exact opposite of good scientific control and leaves too much to variables when buffing honor in general might be worth just testing it alone.

6

The control is the data they have previously w/o any changes. I don’t think the community would go over well if some servers had ally incentives/HvH and others didn’t.

7

I agree. If HvH goes permanent, for example, I don’t see it as prioritizing the Horde knowing that Alliance population will likely dwindle. I instead see it as trying to solve a problem, and lacking the foresight to see the other damage they would solve.

As for managing the appearance of favoritism – the actual favoritism need not exist for the perception to exist. We see accusations of favoritism frequently on this forum, however unfounded they may be.

I speculate that Blizzard could be doing this to manage that perception.

Or perhaps Alliance really do participate in lower numbers (likely because they got their grind done already with faster queues, moreso than any conjectures psychological difference between the factions)

8

Oh I 100% agree with this, specially after the 1st HvH test followup blue post, where they made sure to tell everyone that the BG winrates for ally were same as horde.

As long as the racial imbalances exist in TBC bewteen Horde v Alliance, the perception of favortism will continue. I would love to see racial changes, as they would make rerolling to ally more viable and encourage ally to pvp.

It appears to me that Blizzard is treating the symptoms rather than the cause, and the treatment, no matter what it is, will always look bad to alliance because of the faction’s power imbalance.

9

Not so much faster cues as fewer players. As for faction favoritism there was the issue of both giving horde paladins the dot seal and using the 1.12 version of av above the tbc version of av or the old av. Despite that version being universally hated by alliance it was ignored to the point where only 1/2 av battlegrounds where active at a time.

If what I know of battlegrounds is correct it is most likely a prerelease of buffing honor gains which they will release once there is enough tbc raids to not raid log.

As for racials prob best to wait for wrath if you want that sort of thing.

10

I find it a bit hilarious that they are doing the same thing like retail but with materials this time… the only “solution” they have is to bribe the alliance players with some sort of carrot and hope it’s good enough to keep chasing it.

We will have to see how this plays out. It didnt do well for retail mercenary mode made pvp much worse in general so now we’re playing a version where factions no longer matter for unrated instanced pvp content.
I have the feeling that they are also taking numbers for retail in a weird way as well.

11

There are other changes, also.

You find it hilarious that they are solving the same problems with the same solutions, again?

What about that do you find hilarious?

12

Blizzard has 2 sets of data at the moment:

  1. BGs when there’s no changes. (Might’ve been collected this past week as a control data set).

  2. BGs when same factions are queued.

Now this 3rd set of data will be added which has BGs with same faction queued, as well as 2 other changes (1 to remove premades, 2nd to incentive alliance).

This is actually a reasonable way to go about testing; those 2 changes are different so no reason not to test more changes at once.

13

Used “hilarious” as a friendly tos word.

it’s following retail mistakes anyone can see it.

1 Like
14

Fair point! :heart:

1 Like
15

IMO this is better than Merc Mode, but far from perfect. But w/o changes to the racials, players tend to merge in one faction (this time horde).

Lets see how it goes this week.

Thats the base of any rpg.

16

If you don’t care about understanding the longer term shift of players from Alliance to Horde…

Horde has higher population, therefore it’s easier to find guilds, arena teammates, and other groups.

BGs queues were the only counterbalancing force to keep competitive minded players from moving, and that has been obliterated first at the server level with crossrealm BGs, and now it’s being removed at the level of the entire region with HvH BGs.

The more competitive players will therefore move to the faction where they can find a larger pool of like minded players to form teams with, which only increases the pressure that other players will feel to also make that move – in other words, it’s an accelerating problem.

This concern can’t be realistically studied by these tests, because the tests are short duration and known by the players to be temporary. The permanent fix, if one is implemented, could look like any combination of these tested solutions or like something else entirely.

For players who are considering making a faction change, whether Horde going Alliance for faster queues or Alliance going to Horde for a larger pool of teammates, there is little incentive to decide before you know what that will ultimately look like.

17

Why would BG queues matter that much though? Serious question.

Like I’m not a competitive player and yet I already farmed all my offpieces, a couple blue pieces and have 50k honor stored and 90+ of each mark.

I mean, we already put up with 1-3 hour queues in classic vanilla; I don’t get why anybody who’s competitive would care about that.

Perhaps there’s some small subset of players that went alliance for BG queues explicitly…, but I really don’t know anybody for whom that was a serious consideration.

Sweaty players already put in 8-10 hours a day, so BG queue time shouldn’t be a big deal.

Especially since BG queue times have actually been fairly reasonable for horde (<30 minutes and queuing 3 simultaneously works pretty well).

Horde HPH is actually not bad right now. I earned around 20k honor over the AB weekend with AB having around 15-20 min queues.

18

I would say ask the many Horde players who have been very vocal about this issue and they will be happy to explain their reasoning.

I don’t necessarily agree with them either. I’ve played WoW around the time BGs were first introduced the game and I’ve dealt with BG queues in excess of 30 minutes at various times. I would solve it by farming out in the world while waiting on queue, or else queuing multiple BGs at the same time which leads to maintaining a good position in each queue at all times.

Some players have an expectation that the game should be more like a lobby game, where you sit in a capital city and queue for whatever activity and are instantly teleported there. Why have a world at all then?

The thesis is not that Alliance players chose Alliance for faster queues. Rather they chose the faction that they like. A lot of players surely don’t give it much further thought, but some do.

Personally, the vast majority of my network of PvP friends all chose Horde. For that specific sub-population of competitive PvP players, the choice was relatively less about aesthetics and relatively more about racials as well as the self-fulfilling prophecy that Horde is where the competitive PvP community is.

I went Alliance anyway, because the one friend I play with most often strongly prefers Alliance. I’ve played both over the years, and I like the Alliance zones and appearances a bit better, but don’t have preferences that are as strong. I’d like to have WotF as we are queuing Rogue/Druid in 2s and it’s nowhere near as good a comp without it. But my friend really insists on being a Nelf, so here we are.

And we played in classic on Alliance, having fun experiencing classic but also talking all along about how TBC is likely coming and what is our plan to be prepared. During late classic phases we played a bit of TBC PvP on a pserver in order to re-learn the classes and meta (it’s been a long time after all) and we played on both factions. And my friend isn’t stupid, he knows that our composition is objectively stronger on Horde side, but he prefers Alliance anyway.

Our guild discussed the possibility of switching Horde for TBC, and decided against it because it would suck for our Scarab Lords and Grand Marshals and Alliance gets Seal of Blood now. But if we had gone Horde as a guild, still my friend would like to remain Alliance and find a new guild.

Mind you, this whole time I have been planning since well before TBC classic launch to level Horde as well at some point, on whatever server most of my PvP friends from past retail expansions landed on. I planned this in order to have an alt that I can queue with other friends.

Fast forward to today and the perception of better Horde PvP racials, bigger PvP community, and boosts making it easy to switch, have created a situation where Horde do outnumber Alliance, for the rated Arena community in particular. Even our other PvP friends in our current Alliance guild are talking again about at least boosting Horde alts if not switching more permanently, now that the queue times for Horde are potentially removed and there truly is no remaining downside to the faction. If more players do switch, we definitely don’t want to be the only ones left. Even my one friend that really, really loves his Nelf Druid, is saying “well maybe I could get a 2nd account in order to have a Tauren too”.

So would I say that any of us chose Alliance specifically because of faster queues? No, not at all. But have we considered the removal of that downside when considering whether we might want to move to Horde for all of the upsides it offers? Absolutely.

19

BG queues matter because you have folks who get home from work and have a fixed schedule of play.

If queue times are long they get to play less. This may cause them to unsub.

So it’s better for blizzard to throw them a bone.

Still wish it was merc mode like in retail.

20

The problem is that they are doing all 3 at the same time. The data gathered won’t give sufficient evidence as to what change gave what specific outcomes if they are all implemented at the same time. At the very least, each of the proposed changes should be done separately.

1 Like