So as a few of you have noted, we’ve done a few restarts today and made a few changes, and while performance looked a little better for a while, it eventually got back into a degraded state. The tricky thing here is it hasn’t been one single problem, but a collection of things causing degraded server perf and increased memory usage on the world servers. We’ve fixed a few things, and then we find a few more, etc.
We’re still plugging away at this but we think we found a new problem. FISH! We’ve made a lot of improvements to NPC behavior, having them react to players and the environment in a more natural way, as opposed to just a general “wander around” creature flag. And that’s largely working fine in many places, but we have a lot of coastline in Midnight and that means a lot of wildlife like fish get spawned. And that would probably be fine, but once enough players get close enough to the fish this advanced NPC behavior turned on for them and, well…. something smells fishy.
Again, this might not be a magic bullet, or “The Thing” causing all of the problems, but we’re going to do some mad science tomorrow to scale things a little differently. And if that doesn’t help, we’re going to have to do some more fishing. Either way, I’m hooked!