After weeks of this being a top priority for our encounter and QA teams, we’ve solved this and fixed it.
Some background
Starting on June 10, in a portion of the Halondrus encounter, multiple abilities began disappearing or rendering inconsistently. While we rolled out some partial fixes a few days later, a number of issues have lingered for the past two weeks, understandably frustrating players attempting the Halondrus encounter.
It’s unusual for a bug like this to take so long to overcome. The first step we take when any bug is reported is to have our QA team investigate and reproduce the bug. Figuring out steps to reproduce a bug is essential, both for understanding what’s causing it and to make it so that we can correctly test the eventual fix. With the Halondrus issues, we ran into an immediate challenge: none of our QA testers or engineers were able to reproduce the bug following the same steps that were causing visuals to misbehave on live realms. That’s extremely unusual, and we spent some time trying to gather more information and test different theories, while applying some quick partial fixes.
Over the last week, guided by the curious case of a single engineer who was able to reliably reproduce the issue (when no one else could), we tracked down the underlying cause as a very niche and previously undiscovered bug in our hotfix system itself. This bug made an unrelated hotfix unintentionally cause player clients to read an erroneous visual “occluder” (an invisible volume usually used to tell the engine that it doesn’t need to render whatever is on the other side of it, to optimize performance) in the Halondrus playspace. The reason internal tests didn’t easily reproduce the issue was-- the actual game data was correct.
The Current Situation
We have now developed a process to both clean up the bad hotfix data and fix the issue that made this possible in the first place. That fix was deployed earlier today.
We sincerely apologize for how long this took to fix. It was a top priority for the team the entire time, but turned out to be almost unique in coming from an underlying cause that took far longer to track down than we ever expected.