[2020] “Why did my server shut down?”

When testing Workshop scripts, the answer to the question “why did my server shut down?” is almost always due to what we call “server load”, which is a measurement we take to determine how much processing power a given game instance is consuming. In order to accommodate a large number of instances, we must shut down individual instances that have started to use too much of the available processing power.

So why does this happen? The short answer is that too many Workshop actions are executing. Sometimes it’s because a high number of actions have occurred over several seconds. Sometimes it’s because an extremely high number of actions happen on a single frame.

Not all actions impact server load the same amount, however. Creating and destroying dummy bots, for example, is very costly, while modifying variables is relatively inexpensive (especially starting in patch 1.45). The values you provide actions will affect server load as well. For example, the ray casting values are very expensive while values such as Event Player are practically free.

So how can server load be reduced? The best way is by adding this rule (also available in the “Server Load” preset) to your Workshop script and testing your mode to see when the numbers go up:

rule("Display server performance characteristics")
{
    event
    {
        Ongoing - Global;
    }
 
    actions
    {
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load", Null, Null, Null), String("{0}%", Server Load,
            Null, Null), Null), Null, Null, Left, 0, White, White, White, Visible To and String, Default Visibility);
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load Average", Null, Null, Null), String("{0}%",
            Server Load Average, Null, Null), Null), Null, Null, Left, 1, White, White, White, Visible To and String, Default Visibility);
        Create HUD Text(All Players(All Teams), String("{0}: {1}", String("Server Load Peak", Null, Null, Null), String("{0}%",
            Server Load Peak, Null, Null), Null), Null, Null, Left, 2, White, White, White, Visible To and String, Default Visibility);
    }
}

As a general rule of thumb, you want the numbers to stay below 100 as much as possible. The higher the numbers are above 100, the more risk you run of the server load threshold triggering and your instance being shut down. Note that this measurement includes the base cost of running the game as well, so in heavy fire fights, the numbers can get up to 50 or higher even with no Workshop logic executing.

Another way server load can be reduced is by using conditions whenever possible instead of actions. For example, this…

rule("Kill players that leave the circle")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    conditions
    {
        Distance Between(Event Player, Global Variable(A)) > 5;
    }
 
    actions
    {
        Kill(Event Player, Null);
    }
}

…is much less expensive than this…

rule("Kill players that leave the circle")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    actions
    {
        Wait(0.016, Ignore Condition);
        Loop If(Compare(Distance Between(Event Player, Global Variable(A)), <=, 5));
        Kill(Event Player, Null);
    }
}

Another useful trick is to take a loop that causes too much server load on a single frame and spread its work over multiple frames:

rule("Spawn 100 effects in 10 frames")
{
    event
    {
        Ongoing - Each Player;
        All;
        All;
    }
 
    conditions
    {
        Is Button Held(Event Player, Interact) == True;
    }
 
    actions
    {
        For Global Variable(I, 0, 100, 1);
            Play Effect(All Players(All Teams), Good Explosion, White, Vector(Global Variable(I), 0, 0), 1);
            If(Compare(Modulo(Global Variable(I), 10), ==, 0));
                Wait(0.016, Ignore Condition);                  // Wait for a frame once every 10th effect
            End;
        End;
    }
}

It can be challenging to realize your vision given a limited performance budget, but there are often ways to optimize your scripts. As you create, it might be worth asking yourself as you go “Wait, can I do this using fewer actions?” and “What happens when 12 players all try to do this at the same time?”

One thing that will absolutely shut down your instance every time is something that’s possible starting in the 1.45 patch: infinite loops.

So what is an infinite loop? Simply put, it’s when Workshop is given an endless series of actions to execute on a single frame. Because it can never finish, it eventually gives up. By that time, it has consumed too much of the available processing power, and the server must be shut down.

What does an infinite loop look like? It can take several forms, but the simplest is:

actions
{
    While(True);
    End;
}

Because the condition of the While action always passes and causes execution to continue down to the End action, and because the End action simply causes execution to return to the While action above it, Workshop continues to execute these two actions until enough real-world time has passed that the server is shut down.

Below is a more complex example that is only an infinite loop if there are more than 5 elements in the “targets” array. Can you spot the error?

variables
{
    global:
        0: index
        1: targets
}
 
actions
{
    Set Global Variable(index, 0);
    While(Compare(Global Variable(index), <, Count Of(Global Variable(targets))));               // Consider each target
        If(Compare(Global Variable(index), <, 5));
            Heal(Value In Array(Global Variable(targets), Global Variable(index)), Null, 30);    // Heal the first 5 targets by 30
            Modify Global Variable(index, Add, 1);                                               // Advance to the next target
        Else; 
            Damage(Value In Array(Global Variable(targets), Global Variable(index)), Null, 30);  // Damage all other targets by 30
        End; 
    End;
}

Finally, here’s an example that involves subroutines (another new feature in 1.45). It might seem safe enough at first, but it’s an infinite loop because both rules use the same “index” variable:

variables
{
    global:
        0: index
}
 
rule("Call the subroutine 100 times")
{
    event
    {
        Ongoing - Global;
    }
 
    actions
    {
        For Global Variable(index, 0, 100, 1);
            Call Subroutine(Sub0);                // index will always be 10 when the subroutine returns
        End;
    }
}
 
rule("The subroutine")
{
    event
    {
        Subroutine;
        Sub0;
    }
 
    actions
    {
        For Global Variable(index, 0, 10, 1);
            Modify Global Variable(C, Add, 1);
        End;
    }
}

To summarize, game instances shut down due to server load, and one way to hit the server load limit immediately is with an infinite loop. So keep an eye on those server load values and watch out for logic that loops forever. Good luck!

In the first example, restricting at the event level is much less expensive than restricting via a condition, so rule(“B”) is less expensive.

The second example is interesting. In the first version, the server does a big spike of work up front and then never does anything again, leaving all the heavy lifting to be done by the client apps (since continuously changing values on effects and HUD elements are evaluated on the client). In the second version, both the server and client do some work whenever each player’s A switches between 42 and not 42. It’s hard to say which is “better”. Obviously, if the initial spike on the server in the first version is enough to trigger the server load threshold, then the second version is your only choice. If not, and you’re struggling with server load otherwise (and you don’t mind shifting some of the cost to the client), then the first version might be what you want. Overall, I think the second version is more desirable unless A is changing so fast that it starts noticeably affecting server load. Another advantage of the second version is that you burn fewer effects (in case you’re nearing the effect limit). It’s a complex choice, though, so you might just have to try both and see for yourself. Keep in mind that people with slower machines will want to play your mode, too, so shifting the burden entirely to the client isn’t always the best idea…

Hi Spinky, here are some answers to your questions:

• It’s cheaper to have a global rule with a condition that checks if the Host Player is doing something rather than using Ongoing - Each Player with a condition that compares against the Host Player. The latter is checking 12 things (one for each player) instead of just 1 thing.
• I haven’t measured it, but it probably doesn’t make much difference whether you use a global variable or Host Player. I’d recommend the latter since you don’t have to worry about updating a variable that way.
• If you know the duration of the Wait in advance, and if it’s fairly long (that is, more than a few frames), then using a Wait and a Loop is better than using a condition since a condition just notices that it has a continuously-changing value and, seeing that it does, performs a check every frame. However, if you are going to be checking every frame anyway (or almost every frame), then a condition is cheaper since you don’t have the overhead of executing actions in order to perform your check. (This overhead has been reduced significantly in 1.45, but it’s still present.) As for scalability, your best bet is to perform the check in a single rule and set a variable that the other rules are listening for (since this variable won’t be changing continuously, and thus the conditions listening for it are asleep until the variable changes). Of course, if each rule requires an independent timer, then the variable trick won’t work.
• There’s almost no difference between your examples in terms of performance since both methods take advantage of short circuiting (that is, not evaluating the second condition if the first is false). I’d encourage using the first way since it’s easier to read. (Side note: There was some disagreement earlier between me and Zach regarding whether conditions have short circuiting. I’ve confirmed today that they do. Zach was right!)
• Yes, Workshop uses short circuiting in conditions, the And value, and the Or value.
• Reevaluation generally occurs when something in the value is changing. For values that change continuously or could change continuously (such as positions), this causes reevaluation of the entire value to occur every frame. If the array you’re filtering has anything in it that continuously changes (either the array itself or the condition you’re using) then that might get expensive. Filtering an array of all players with a player variable as the condition, however, will not be considered continuous unless the variable is continuous (which it would be if it were being chased). Such a value would cause reevaluation only when players are added or removed or when your variable changes. For that reason, you probably don’t need a global variable that maintains a filtered copy of All Players unless you have many places where you’re making the same Filtered Array check (though what counts as “many” would need to be measured – I’m just saying there’s a cutoff at some point where maintaining a global variable is cheaper).

Good luck optimizing your scripts! I forgot to mention the Disable Inspector Recording action in my post above – use it when your script starts up to gain some efficiency back, and then use Enable Inspector Recording to debug specific problem areas.