“Why did my server shut down?”

When testing Workshop scripts, the answer to the question “why did my server shut down?” is almost always due to what we call “server load”, which is a measurement we take to determine how much processing power a given game instance is consuming. In order to accommodate a large number of instances, we must shut down individual instances that have started to use too much of the available processing power.

So why does this happen? The short answer is that too many Workshop actions are executing. Sometimes it’s because a high number of actions have occurred over several seconds. Sometimes it’s because an extremely high number of actions happen on a single frame.

Not all actions impact server load the same amount, however. Creating and destroying dummy bots, for example, is very costly, while modifying variables is relatively inexpensive (especially starting in patch 1.45). The values you provide actions will affect server load as well. For example, the ray casting values are very expensive while values such as Event Player are practically free.

So how can server load be reduced? The best way is by adding this rule (also available in the “Server Load” preset) to your Workshop script and testing your mode to see when the numbers go up:

rule("Display server performance characteristics")
{
    event
    {
        Ongoing - Global;
    }
 
    actions
    {
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load", Null, Null, Null), String("{0}%", Server Load,
            Null, Null), Null), Null, Null, Left, 0, White, White, White, Visible To and String, Default Visibility);
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load Average", Null, Null, Null), String("{0}%",
            Server Load Average, Null, Null), Null), Null, Null, Left, 1, White, White, White, Visible To and String, Default Visibility);
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load Peak", Null, Null, Null), String("{0}%",
            Server Load Peak, Null, Null), Null), Null, Null, Left, 2, White, White, White, Visible To and String, Default Visibility);
    }
}

As a general rule of thumb, you want the numbers to stay below 100 as much as possible. The higher the numbers are above 100, the more risk you run of the server load threshold triggering and your instance being shut down. Note that this measurement includes the base cost of running the game as well, so in heavy fire fights, the numbers can get up to 50 or higher even with no Workshop logic executing.

Another way server load can be reduced is by using conditions whenever possible instead of actions. For example, this…

rule("Kill players that leave the circle")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    conditions
    {
        Distance Between(Event Player, Global Variable(A)) > 5;
    }
 
    actions
    {
        Kill(Event Player, Null);
    }
}

…is much less expensive than this…

rule("Kill players that leave the circle")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    actions
    {
        Wait(0.016, Ignore Condition);
        Loop If(Compare(Distance Between(Event Player, Global Variable(A)), <=, 5));
        Kill(Event Player, Null);
    }
}

Another useful trick is to take a loop that causes too much server load on a single frame and spread its work over multiple frames:

rule("Spawn 100 effects in 10 frames")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    conditions
    {
        Is Button Held(Event Player, Interact) == True;
    }
 
    actions
    {
        For Global Variable(I, 0, 100, 1);
            Play Effect(All Players(All Teams), Good Explosion, White, Vector(Global Variable(I), 0, 0), 1);
            If(Compare(Modulo(Global Variable(I), 10), ==, 0));
                Wait(0.016, Ignore Condition);                  // Wait for a frame once every 10th effect
            End;
        End;
    }
}

It can be challenging to realize your vision given a limited performance budget, but there are often ways to optimize your scripts. As you create, it might be worth asking yourself as you go “Wait, can I do this using fewer actions?” and “What happens when 12 players all try to do this at the same time?”

One thing that will absolutely shut down your instance every time is something that’s possible starting in the 1.45 patch: infinite loops.

So what is an infinite loop? Simply put, it’s when Workshop is given an endless series of actions to execute on a single frame. Because it can never finish, it eventually gives up. By that time, it has consumed too much of the available processing power, and the server must be shut down.

What does an infinite loop look like? It can take several forms, but the simplest is:

actions
{
    While(True);
    End;
}

Because the condition of the While action always passes and causes execution to continue down to the End action, and because the End action simply causes execution to return to the While action above it, Workshop continues to execute these two actions until enough real-world time has passed that the server is shut down.

Below is a more complex example that is only an infinite loop if there are more than 5 elements in the “targets” array. Can you spot the error?

variables
{
    global:
        0: index
        1: targets
}
 
actions
{
    Set Global Variable(index, 0);
    While(Compare(Global Variable(index), <, Count Of(Global Variable(targets))));               // Consider each target
        If(Compare(Global Variable(index), <, 5));
            Heal(Value In Array(Global Variable(targets), Global Variable(index)), Null, 30);    // Heal the first 5 targets by 30
            Modify Global Variable(index, Add, 1);                                               // Advance to the next target
        Else; 
            Damage(Value In Array(Global Variable(targets), Global Variable(index)), Null, 30);  // Damage all other targets by 30
        End; 
    End;
}

Finally, here’s an example that involves subroutines (another new feature in 1.45). It might seem safe enough at first, but it’s an infinite loop because both rules use the same “index” variable:

variables
{
    global:
        0: index
}
 
rule("Call the subroutine 100 times")
{
    event
    {
        Ongoing - Global;
    }
 
    actions
    {
        For Global Variable(index, 0, 100, 1);
            Call Subroutine(Sub0);                // index will always be 10 when the subroutine returns
        End;
    }
}
 
rule("The subroutine")
{
    event
    {
        Subroutine;
        Sub0;
    }
 
    actions
    {
        For Global Variable(index, 0, 10, 1);
            Modify Global Variable(C, Add, 1);
        End;
    }
}

To summarize, game instances shut down due to server load, and one way to hit the server load limit immediately is with an infinite loop. So keep an eye on those server load values and watch out for logic that loops forever. Good luck!

17 Likes

Thank you for this :slight_smile:

Out of curiosity

  1. Which of the following is less expensive?
rule("A")
{
	event
	{
		Ongoing - Each Player;
		All;
		All;
	}

	conditions
	{
		Hero Of(Event Player) == Hero(Ana);
		Team Of(Event Player) == Team 1;
		Is Game In Progress == True;
	}

	actions
	{
		Set Player Variable(Event Player, A, 42);
	}
}

or

rule("B")
{
	event
	{
		Ongoing - Each Player;
		Team 1;
		Ana;
	}

	conditions
	{
		Is Game In Progress == True;
	}

	actions
	{
		Set Player Variable(Event Player, A, 42);
	}
}
  1. Let us assume that the Game’s duration to be 10 minutes, there are always 12 players, and that all 12 players have their Player Var A set to 42 for a duration of 5 seconds every 10 seconds. Which of the following would be less expensive ?
rule("Create effects for players in all slot.")
{
	event
	{
		Ongoing - Global;
	}

	conditions
	{
		Is Game In Progress == True;
	}

	actions
	{
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(0, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(0, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(1, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(1, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(2, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(2, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(3, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(3, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(4, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(4, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(5, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(5, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(6, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(6, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(7, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(7, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(8, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(8, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(9, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(9, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(10, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(10, All Teams), 1.500, Visible To Position and Radius);
		Create Effect(Filtered Array(All Players(All Teams), Compare(Player Variable(Players In Slot(11, All Teams), A), ==, 42)),
			Good Aura, Turquoise, Players In Slot(11, All Teams), 1.500, Visible To Position and Radius);
	}
}

or

rule("Create effect if Player Var A == 42")
{
	event
	{
		Ongoing - Each Player;
		All;
		All;
	}

	conditions
	{
		Player Variable(Event Player, A) == 42;
		Entity Exists(Player Variable(Event Player, B)) == False;
		Is Game In Progress == True;
	}

	actions
	{
		Create Effect(All Players(All Teams), Good Aura, Turquoise, Event Player, 1.500, Visible To Position and Radius);
		Set Player Variable(Event Player, B, Last Created Entity);
	}
}

rule("Destory effect if Player Var A != 42")
{
	event
	{
		Ongoing - Each Player;
		All;
		All;
	}

	conditions
	{
		Player Variable(Event Player, A) != 42;
		Entity Exists(Player Variable(Event Player, B)) == True;
		Is Game In Progress == True;
	}

	actions
	{
		Destroy Effect(Player Variable(Event Player, B));
	}
}
1 Like

In the first example, restricting at the event level is much less expensive than restricting via a condition, so rule(“B”) is less expensive.

The second example is interesting. In the first version, the server does a big spike of work up front and then never does anything again, leaving all the heavy lifting to be done by the client apps (since continuously changing values on effects and HUD elements are evaluated on the client). In the second version, both the server and client do some work whenever each player’s A switches between 42 and not 42. It’s hard to say which is “better”. Obviously, if the initial spike on the server in the first version is enough to trigger the server load threshold, then the second version is your only choice. If not, and you’re struggling with server load otherwise (and you don’t mind shifting some of the cost to the client), then the first version might be what you want. Overall, I think the second version is more desirable unless A is changing so fast that it starts noticeably affecting server load. Another advantage of the second version is that you burn fewer effects (in case you’re nearing the effect limit). It’s a complex choice, though, so you might just have to try both and see for yourself. Keep in mind that people with slower machines will want to play your mode, too, so shifting the burden entirely to the client isn’t always the best idea…

7 Likes

Anecdotally
“but… it works on my machine”
“well, then we’ll ship your machine”
and that is how docker was born…

5 Likes

IMO you shouldn’t redefine what an infinite loop is.

So what is an infinite loop? Simply put, it’s when Workshop is given an endless series of actions to execute on a single frame.

According to wikipedia:

In computer programming, an infinite loop (or endless loop ) is a sequence of instructions that, as written, will continue endlessly, unless an external intervention occurs (“pull the plug”).

What you mean by infinite loops are actually waitless loops. There can be infinite loops that have waits, and they are still infinite loops. This runs the risk of people being wary of their infinite loops with wait because a dev said “infinite loops make your server crash” whereas this is only true for waitless loops.

3 Likes

Hey Dan, thank you for sharing this; it is incredibly helpful for people like me with scripts that tend to push the workshop to the limit.
I also have a few scenarios that would be helpful to learn more about:

  • Is it cheaper to have a global rule with a condition that checks if the host player is doing something, or would an ongoing - each player rule that compares against the host player be cheaper?
  • Same as above, but with a global variable that stores a player instead of using Host Player
  • What is the overhead on Waits/Loops? For example, with something that repeats every 5 seconds: Is it better to use Wait & Loop, or to have a rule with a condition that compares if (TotalTimeElapsed - cooldown > 5) and update cooldown = TotalTimeElapsed each time the rule runs, so there’s no Wait/Loop? Which would scale better when there’s many different rules with these setups?
  • Which condition is cheaper? My gut says the first one.
conditions
{
	Player Variable(Event Player, A) == True;
	Player Variable(Event Player, B) == True;
}

// or

conditions
{
	And(Player Variable(Event Player, A), Player Variable(Event Player, B)) == True;
}
  • Similar to above, are conditions evaluated logically? ie. if A is false in the first example above, does it immediately stop checking the conditions? Does this also apply to things like nested Or's?
  • Last question, I promise. For something like visibility, you spoke about reevaluation being handled on the client. Does this include filtered arrays? Is using a filtered array of all players with a player variable as the condition cheaper than appending/removing players from a global variable array based on that player variable (assuming it does not rapidly change)?

I’m very curious about the workshop internals and often wonder how it handles various situations/actions, so these are the kinds of thoughts I have while trying to optimize my larger workshop scripts that tend to cause shutdowns. It is difficult to properly benchmark them one way or the other, so any insights are greatly appreciated. Thank you.

Hi Spinky, here are some answers to your questions:

  • It’s cheaper to have a global rule with a condition that checks if the Host Player is doing something rather than using Ongoing - Each Player with a condition that compares against the Host Player. The latter is checking 12 things (one for each player) instead of just 1 thing.
  • I haven’t measured it, but it probably doesn’t make much difference whether you use a global variable or Host Player. I’d recommend the latter since you don’t have to worry about updating a variable that way.
  • If you know the duration of the Wait in advance, and if it’s fairly long (that is, more than a few frames), then using a Wait and a Loop is better than using a condition since a condition just notices that it has a continuously-changing value and, seeing that it does, performs a check every frame. However, if you are going to be checking every frame anyway (or almost every frame), then a condition is cheaper since you don’t have the overhead of executing actions in order to perform your check. (This overhead has been reduced significantly in 1.45, but it’s still present.) As for scalability, your best bet is to perform the check in a single rule and set a variable that the other rules are listening for (since this variable won’t be changing continuously, and thus the conditions listening for it are asleep until the variable changes). Of course, if each rule requires an independent timer, then the variable trick won’t work.
  • There’s almost no difference between your examples in terms of performance since both methods take advantage of short circuiting (that is, not evaluating the second condition if the first is false). I’d encourage using the first way since it’s easier to read. (Side note: There was some disagreement earlier between me and Zach regarding whether conditions have short circuiting. I’ve confirmed today that they do. Zach was right!)
  • Yes, Workshop uses short circuiting in conditions, the And value, and the Or value.
  • Reevaluation generally occurs when something in the value is changing. For values that change continuously or could change continuously (such as positions), this causes reevaluation of the entire value to occur every frame. If the array you’re filtering has anything in it that continuously changes (either the array itself or the condition you’re using) then that might get expensive. Filtering an array of all players with a player variable as the condition, however, will not be considered continuous unless the variable is continuous (which it would be if it were being chased). Such a value would cause reevaluation only when players are added or removed or when your variable changes. For that reason, you probably don’t need a global variable that maintains a filtered copy of All Players unless you have many places where you’re making the same Filtered Array check (though what counts as “many” would need to be measured – I’m just saying there’s a cutoff at some point where maintaining a global variable is cheaper).

Good luck optimizing your scripts! I forgot to mention the Disable Inspector Recording action in my post above – use it when your script starts up to gain some efficiency back, and then use Enable Inspector Recording to debug specific problem areas.

7 Likes

So… How can I help crashing when loading the workshop editor? Can I not? And as far as we know, will server load optimizations be made on the update?

Hey Dan, would it be possible to add an option to disable built-in small messages to the workshop? things like +25 on fire meter messages and “+5 objective defense” are constantly spamming the screen for my game mode and often times players ignore those messages all-together because of how much there is being thrown at the player.

Also, create beam effect with a filtered array doesn’t work for anyone except the event player, not even for the spectator.

And one last thing, the default AI Bots are constantly getting stuck in every single map, is there any fix incoming? it’s been like this for 7 months already.

Hey Dan, My game always crashes when I swap players, but it doesn’t when I don’t use swap. Is there any good optimization method? Is the “Ongoing - Each Player” event takes up too much CPU when swapping players? thx :smiley:

Yes, I have this problem so much!

Please consider this thread.

Workshop server crashes after swapping player, multiplying server load with garbage that does not go away after a player leaves

Other players are having the same issue.

After some years of a development in workshop I would add that better way is not to shutdown a gameroom but to pause it and show to its host stats and reason of the interruption. Why it is so important? Because the host can learn from stats, var pools and change it in a shortest period of a time instead of spending hours looking for a cause of the crushing. You should respect your players in this way too.

1 Like