[2021] March 2021 Development Update

Anecdotes from the past month of gamedev. Focused on the decisions and thinking behind the RPC rewrite.

RPC Rewrite

Unturned uses "remote procedure calls" (RPC) for all gameplay netcode. This is a fancy way of saying that when you press the "pick up item [F]" key, the client asks the server to call a PickUpItem method, and then the server responds by telling the client to call an AddItemToInventory method.

Prior to the rewrite it looked something like:

public void askChat(CSteamID senderId, byte flags, string text)
...
channel.send("askChat", ESteamCall.SERVER, ESteamPacket.UPDATE_RELIABLE_BUFFER, flags, text);

This works fine, but there are a lot of aspects to improve:

Back when most of this old code was rewritten nameof() was not available in Unity's c# version, so a typo could break the call and not be found until runtime.
Parameters were passed as params object[] args which created a bunch of boxed garbage.
Lack of type checking made it easy to send and receive different types, e.g. passing constants like send(255, 255) receiving 8 bytes rather than 2 was the cause of numerous bugs in earlier years 2014-2015.
Index of the method was looked up by name for every send.
Argument type was determined by reflection and serialized in a huge if/else chain mentioned in the previous development update blogpost.
Methods which should be static like the example used instance lookup overhead.
Method indices were per game object and had to be rebuilt when a component was added or removed.
Method indices could not be initialized until Start() which is fired the frame after Awake(), so the client made requests for data once it could be received, rather than the server sending as appropriate.
Method indices are not constant so rate limiting requires a name lookup.
SteamID sender parameter is obviously bad for cross platform.
Deserialization again used reflection and generated garbage.
Receiving used MethodInfo.Invoke which is one of the slower approaches.

All of these have been addressed in the rewrite, and every single one of the hundreds of game RPCs was adjusted! Before diving deeper here is how the same code looks now:

private static readonly ServerStaticMethod SendChatRequest = ServerStaticMethod.Get(ReceiveChatRequest);
public static void ReceiveChatRequest(in ServerInvocationContext context, byte flags, string text)
...
SendChatRequest.Invoke(ENetReliability.Reliable, flags, text);

Code Generation

Rather than doing reflection at runtime for sending and receiving I wanted to do it at compile time. Generating code at runtime e.g. Weaver would be an option, but I would prefer to keep it simple and reduce dependencies. Since the game code calls the generated code, but needs to be able to compile before it can be generated, I decided on the one-time startup dynamic linking shown above.

"Net invokable" methods have an attribute marking them for code generation, and the code generator outputs a static class with static methods for each type:

[NetInvokableGeneratedClass(typeof(ChatManager))]
public static class ChatManager_NetMethods
{
	[NetInvokableGeneratedMethod(nameof(ChatManager.ReceiveChatRequest), ENetInvokableGeneratedMethodPurpose.Read)]
	public static void ReceiveChatRequest_Read(in ServerInvocationContext context)
	{
		NetPakReader reader = context.reader;
		System.Byte flags;
#if LOG_INVOKE_READ_ERRORS
		bool flags_ReadSuccess =
#endif // LOG_INVOKE_READ_ERRORS
		reader.ReadUInt8(out flags);
#if LOG_INVOKE_READ_ERRORS
		if (!flags_ReadSuccess)
		{
			context.ReadParameterFailed(nameof(flags));
			return;
		}
#endif // LOG_INVOKE_READ_ERRORS
		System.String text;
#if LOG_INVOKE_READ_ERRORS
		bool text_ReadSuccess =
#endif // LOG_INVOKE_READ_ERRORS
		reader.ReadString(out text);
#if LOG_INVOKE_READ_ERRORS
		if (!text_ReadSuccess)
		{
			context.ReadParameterFailed(nameof(text));
			return;
		}
#endif // LOG_INVOKE_READ_ERRORS
		ChatManager.ReceiveChatRequest(context, flags, text);
	}
	[NetInvokableGeneratedMethod(nameof(ChatManager.ReceiveChatRequest), ENetInvokableGeneratedMethodPurpose.Write)]
	public static void ReceiveChatRequest_Write(NetPakWriter writer, System.Byte flags, System.String text)
	{
		writer.WriteUInt8(flags);
		writer.WriteString(text);
	}
}

During startup the game quickly gathers all of these methods for linking, and assigns delegates pointing at them. Since every method is known at this point rate limiting can be done with an index into a flat array, and the exact number of bits necessary for the index can be used.

Fast Path

In some cases especially with larger amounts of data avoiding the overhead of repeated RPCs is desirable. For that reason the signature of the generated read implementation is only the context, which allows the message router to directly invoke a game method with the same signature instead. For example the server can tightly pack every changed vehicle state update (position, rotation, speed) into a single RPC send and then unpack in the receive implementation.

Parameter Attributes

For cases that would benefit from non-default parameter packing but writing a fast path would be overkill we can use non-runtime parameter attributes. The average Vector3 can get away with 7 fractional bits - slightly better than centimeter precision, but large objects snapped together need better than that. We can then apply an attribute to override the fractional bit count in the method parameters:

[NetPakVector3(fracBitCount: 11)] Vector3 position

Instance vs Faux-Instance

Getting the handle for sending a static RPC is as easy as passing the method reference. Although instance methods are a bit trickier because we only want to look it up once during static initialization. We cannot get a reference to the method without an instance. This meant passing the type and name:

private static readonly ClientInstanceMethod SendMarkerState = ClientInstanceMethod.Get(typeof(PlayerQuests), nameof(ReceiveMarkerState));

It feels fragile considering the type and/or name could be mismatched. I strongly considered making every instance method static and passing the instance reference as a parameter in order to work around it. But, it was clunky either way so I settled on keeping them as instance methods. To hopefully catch this kind of error, we log if the method does not exist, and run a test that every method handle is only "claimed" once.

Instance Addressing

The old system had no uniform way to lookup objects; so, each game feature had its own way to reference them. This made apparently simple features like attaching bullet holes to hit objects a mess. Now we take the obvious approach of assigning objects unique numeric ids, and use those when invoking instance methods. This enables invoking methods immediately after creation, deferral of invocation if the object does not exist yet, and a neat optimization by allocating IDs in blocks: a spawned player reserves a block of IDs, sends only the first ID, and then the client associates the following IDs sequentially e.g.; +1 = inventory, +2 = movement, et cetera.

Invocation Context

Two common cases with the receiving RPC implementation is checking whether the sender is appropriate to call the method, and debug messages when something unexpected happens. Some common cases can be checked prior to calling the method e.g. is this object controlled by the calling player, but others like "is the calling player in this vehicle" benefit from making the implementation able to do these checks. Giving the context struct metadata about the method proved really helpful for debug error logging.

Filtering Recipients

Originally there were special cases for sending RPCs to a subset of connections, e.g. SendToClientsInsideNavmesh, that iterated every client to determine who should receive it. We would later make this more generic with a predicate callback. But thinking ahead, this already did not scale well in the old version. Furthermore, it would be problem for increasing player counts in the future. Instead, now we pass IEnumerable when sending the RPC, allowing the recipients to be cached as a future optimization e.g. tracking the list of clients inside a navmesh for the earlier example.

Initial State

Due to the Awake() vs Start() problems in the intro the game had "once per connection" RPCs where the client would request information after it could process it. These were a target for cheaters a few years ago because they would reply with a ton of data, and were overall pointless considering the server should be able to send the data regardless. The request RPCs have been entirely removed now. But, it did mean messing with the ordering of initialization which broke lots of the HUD. As far as I can tell it is all working fine... but some problems will probably be found after the update.

Loopback

I debated for a while whether invoking net methods should ever be executed locally. There is some overhead to this rather than directly calling the target method, but it does simplify the calling side especially as a listen server in singleplayer. It also has the benefits of reducing edge cases (rather than branching in singleplayer) and extra testing of the netcode to some degree, so I decided to keep it.

Plugin Compatibility

With such a widespread rewrite keeping backwards compatibility with old server plugins was going to be tricky. All support for the old system was removed for client to server code, but server to client had to be retained, even for server to self loopback. Hundreds of methods have been kept and marked deprecated, and every receive implementation has a "legacy name" attribute so that it can be invoked by old packets. Thanks to many plugin developers for working with me to update their code in advance of this update! Lots of new events and methods have been added as alternatives for old hacks.

Organizing Assemblies

With Unity's newish assembly definition feature it seemed like putting the generated code in a separate assembly would be an obvious compilation speed win. Unfortunately, new assemblies cannot reference legacy assemblies, and moving all the old code into an .asmdef (assembly definition file) would break plugin references. I asked Trojaner about this and he kindly suggested binding OnAssemblyResolve to redirect old references which worked great for plugin backward compatibility. But sadly, this does not seem to be called for types loaded from Unity asset bundles. So unless you know a way around this (would be greatly appreciated!), most of the old core game code is stuck in Assembly-CSharp.dll for the foreseeable future.

Physics Material Custom Data

There are several hardcoded physics materials in Unturned for grass, wood, concrete, etc. Unfortunately this limits both the materials available to new maps, and the collision effects between materials. Ideally, mods would be able to create new materials and extend existing ones (e.g., custom projectile type not wanting to use bullet hole effect), which is an issue that arose in Unturned II as well.

My plan with the recent audio improvements in Unturned is to allow arbitrary data (audio clips in this case) to be associated with materials by name. Mods will be able to create custom materials with optional fallbacks to built-in materials, and then any asset can modify the matrix of inter-material effects.

Voice

While updating and improving the voice netcode I experimented with Unity's audio clip streaming PCM reader callback. Rather than updating all voice samples every time it would request a smaller amount as-needed. In theory this was a nice win, but I found that Unity was starving us almost immediately regardless of buffer size. Might be worth revisiting in the future if the streaming buffer size is configurable somehow.

Roadmap

I decided to put together a board of the updates I am actively considering for Unturned. Ideas that I would like to work on but do not have a plan yet (car physics, item crafting revamp, official servers, etc.) are not listed yet. https://trello.com/b/gpe4zlW3/Unturned-Roadmap

The post March 2021 Development Update appeared first on SDG Blog.