[2024] Ashes of Creation Alpha Two Server Meshing Technology Preview

Transcript (by Youtube)

2s	[Music]
20s	hello everyone Welcome to our glorious
23s	server mesh technology showcase segment
27s	exciting it is I'm Margaret Cron your
30s	director of communications and of course
32s	with me as always is Stephen shareif our
35s	wonderful creative director I'm excited
38s	we have a fun showcase today I mean it's
40s	this is something that I don't think a
41s	lot of companies go into a lot of
44s	projects don't really talk about and and
46s	you know it can be a little bit on the
48s	techie side but I think this is stuff
50s	that a lot of players uh like hearing
52s	about and I know we got some we got some
54s	exciting people to chat with today what
56s	do you think and um of course we tried
59s	to make it as visually appealing
60s	possible shout out to Mike beio as you
64s	you may know him from all of our other
65s	Graphics but he's our Graphics designer
67s	and he did a wonderful job on these
69s	slides so hopefully they help portray
72s	what our wonderful engineering team is
75s	trying to to uh showcase so we will kind
78s	of introduce them we have some new
80s	people who you've never met before we'll
83s	kick it off first with Zack if you want
85s	to talk about who you are what you do at
86s	the studio and maybe a little fun fact
90s	hi guys uh I'm Zach I'm the technical
92s	director here uh at Intrepid Studios I
95s	run the networking team and the engine
98s	team
100s	um I'm generally responsible for
102s	performance and making sure that the
103s	game scales to the amount of players
106s	that we wanted to a hefty and lofty goal
110s	many of us players will have thoughts on
113s	that and of course we have Hunter with
116s	us which some of you if you hang out on
118s	our Discord maybe have interacted with
120s	before
122s	for hi yeah thanks for having me um my
125s	name is Hunter I am a game networking
127s	engineer at inet studios I've been here
129s	for almost about four years now and
133s	first initially started off as a
134s	gameplay engineer and then I slowly work
136s	my way over to networking because that's
138s	what I've come to find that I love and I
139s	love solving the hard problems and very
142s	excited to Showcase to you all what
143s	we've been working on the gameplay
145s	engineering just wasn't torture enough
147s	for Hunter so he wanted to go straight
149s	straight the be impossible
152s	problems oh my God I like I like solving
154s	problems like this hard problems I think
157s	fun problem to solve yeah um and then of
160s	course last but not least
162s	Anton hello I'm Anton uh I'm a senior
165s	networking engineer over here at
168s	Intrepid I moved straight into this
170s	position cuz uh yeah I wanted to suffer
174s	immediately I didn't want to start with
177s	a game play
180s	all right well let's kick it off Zach I
183s	think you have a little intro here for
185s	us as we move
186s	in yeah um so Zach just I'm sorry just
190s	to start a lot of people you know
192s	sometimes don't know what networking is
196s	yeah talk to talk to us about what is
198s	networking what is a network layer at at
201s	its core networking is just the concept
206s	of moving a state from the server to the
209s	clients
210s	that that at its core that is the
212s	problem we are trying to solve we are
213s	trying to do that as fast as possible
216s	with as many players as possible um
219s	since we are making a very very large
221s	game uh we have a lot of problems that
223s	most games don't usually
227s	have yeah and and you know with with
231s	regards to like an mmpg versus like a
233s	peer-to peer versus like a small
235s	dedicated server like our networking
237s	needs are intense because the types of
241s	numbers we're looking at are extreme
244s	yeah we we we every everyone on the
246s	networking team here has worked on other
248s	multiplayer games before and this is
251s	definitely the project where we have to
252s	get the most creative with how we are
254s	going to solve uh our scaling problems
257s	you know like we are we we we are
260s	working on an MMO most MMOs don't don't
262s	do what we're about to do here they
263s	don't talk about how they you know made
266s	it so they can support thousands of
267s	players per server um so we it's very
271s	much we have to you know kind of like
272s	figure out what what has been done and
274s	and do a lot of um R&D to figure out
276s	what what we need to do for our game
279s	right and kind of answering that
280s	question of what we need to do for our
282s	game starts out with well what is our
285s	game trying to achieve right yes yeah
287s	there are a lot of different types of
288s	MMOs some that heavily you know utilize
292s	instancing um some that you know maybe
295s	don't care about replicating to hundreds
297s	of players at a given time um so first
300s	it starts with a vision what is ashes of
303s	creation and then that kind of leads you
305s	down the architectural path of okay what
308s	does that mean we have to set up because
310s	this isn't out of the box Unreal Engine
312s	5 this is a custom Network layer yeah
315s	yeah this is this is this is fully
316s	Custom Tailor Made for the needs that
319s	ashes has like this is not none of this
323s	is is is off the shelf none of this is
324s	standard unreal this is all built for
326s	the kind of game that we are
328s	making right and so how do you go about
331s	kind of you know determining that that
333s	architecture what does that look like
334s	for a game like ashes and and you know
336s	we've been working on this for a number
338s	of years yeah for years um so I guess I
341s	guess the first the first step is we
342s	need to figure out the requirements like
345s	what do we actually need to like what
349s	what are Ash's requirements um you know
352s	the the first place we start is you know
355s	we're an open world PVP game so we can't
357s	really control where players go we can't
359s	do match uh typical matchmaking where we
361s	just you know spin up more servers SP
363s	put them on the new matchmaking servers
366s	and you know not have these issues we're
368s	an open world game anyone can go
370s	anywhere um you know we're a highly PVP
373s	game so we need to have low latency
375s	people can't just uh we we can't do what
378s	other games do where you just like lower
381s	your tick rates really low and
383s	then um just simulate at a really really
386s	low rate since we're you know action PVP
389s	we can't do that um one of our I guess
394s	would you call this a game design pillar
396s	Stephen like this is we don't that we
398s	don't want to do layering like that's a
399s	that's that's that's a very like
402s	absolutely game thing we wanted we
404s	wanted so like you can go anywhere in
406s	the world stand there and the people who
410s	are there are there like
412s	yeah and I was just going to say like
414s	game laring you know obviously everyone
416s	on this call is a gamer who's played
418s	MMOs and we played games that that do
420s	layering yep you know it's just a very
422s	different experience when you can switch
425s	channels you
426s	know yeah yeah it's like a different
429s	world a core like MMO memory for me is
433s	like you know old wow you would go to
436s	the same place
438s	like uh and and there would be the same
440s	person there you know dancing on top of
442s	a mailbox every single time or dancing
444s	on top of a corpse and looking at you
447s	with Hungry Eyes they were they were
449s	always they were always there and they
450s	were always and they were always there
452s	and with layering you don't get that uh
454s	you don't get that that same level of
456s	experience that we that we that we want
457s	to give uh to our
459s	players um and then I guess our last
462s	requirement which is like you know just
463s	a typical MMO requirement is that we
465s	have you know very very high player
466s	count goals um and you know typically uh
471s	High player count goals with no layering
473s	don't really go together so right that's
476s	where it comes into like you know we
477s	need to get really creative with how
478s	we're going to solve these problems some
480s	of us you know Hunter Anton we've played
483s	these games that have high player counts
485s	but a lot of ways that they get around
489s	having higher player counts is by being
491s	less server adjudicated less server
493s	authoritative and more player which has
496s	its own risks and isn't really
498s	preserving the Integrity of that player
502s	experience indeed like some some games
504s	do not um do proper physic simulations
508s	on the server side to validate movements
510s	in order to achieve that kind of thing
512s	which means like cheaters uh can have an
515s	easier time right going through walls
517s	and stuff like that or they well I guess
521s	they do layering or sharding where uh
525s	they yes they have real high player
527s	counts but technically not really CU
530s	they're kind of broken up I can't tell
532s	you how many times I've I've jumped into
534s	either um you know some session based
537s	game or I don't want to call any any of
539s	them out out but where there's just one
541s	single player that's either hacking or
544s	or or doing something and they're just
545s	running around and ruining the
547s	experience for everyone and it's because
549s	the way they've architected in such a
551s	way it's more it's easier for those
553s	types of experiences to occur always
555s	trust the client right
559s	yeah you just made everyone sweat Hunter
563s	do we want to kind of transition that
565s	into talking about the issues that we
567s	are trying to address yeah of course
572s	yeah so um I guess the issue start with
575s	it's like you know we're making MMO and
578s	unreal um the amount of entities we want
581s	on the server um it's just not really
584s	going to work with standard outof the
586s	Box unreal replication unreal
588s	replication and unreal simulation so
591s	there there are two problems here it's
592s	like okay even if we could fit all the
594s	players we want on a single server um
597s	you
598s	know we we we still can't replicate them
601s	because those are two separate problems
603s	we have the simulation and then the
604s	replication problem um so since this is
609s	is a problem and you know there there is
612s	no standard unreal solution for this we
615s	we we had to go about you know making
617s	our own custom solution to solve these
620s	problems um and this is where Intrepid
622s	net um I guess was born um and trepid
626s	net is a fully distributed networking
628s	stack it's tailor made for creation
630s	fully built in house um none of this is
633s	offthe shelf technology it's all custom
635s	built for our game
638s	um we you know we'll we'll get into
641s	these more in a bit but Intrepid net is
644s	made up of quite a few different
646s	Technologies um the the one that you
649s	know people are most interested in
650s	people are always talking about is
652s	server meshing we'll talk a lot about
653s	server meshing and how we how we
655s	implemented it for our game um we have
658s	inter server replic
660s	which is you know we have we have
663s	multiple servers and they're replicating
664s	to each other how do we deal with that
666s	problem we have our own microservice
669s	layer for servicing our
673s	um uh communication I guess between our
676s	servers um we have uh Dynamic griding
681s	which is very exciting and you guys and
683s	you guys will like a lot and then we
685s	have the replication graph um which is
687s	just more um custom replication uh I
692s	would almost call them formulas um for
694s	our game so we'll be going into all of
697s	this it's very this is very detailed
699s	very Tech heavy
701s	so buckle in yeah we need some like
703s	theme music for the Intrepid net logo
705s	there too I know they were so excited
708s	when Mike made a logo they're like so
710s	nice of a logo I'm just thinking of like
713s	the Indiana Jones theme song right
716s	now right
718s	totally the word do we start off with
721s	this yeah so I'm going to kick it over
723s	to Hunter and he'll kind of like start
724s	with the like giving some of the base
727s	knowledge for like how unreal works and
729s	then he'll start with you know talking
731s	about how some of our server technology
733s	our server meshing technology
735s	works yeah hello um so before we start
738s	diving too deep into Intrepid net um
741s	there's probably a few terminologies
742s	that we wanted to preface with just so
744s	that it helps the community and everyone
746s	else here follow along better with like
747s	the lingo and the terminology that we
749s	will
750s	using um the first concept that we
752s	wanted to touch on is an actor um in
755s	Unreal Engine you can think of an actor
757s	as like almost any physical
759s	representation of an object or an entity
762s	that's inside the
763s	world as you can see on the photo that
766s	an actor can be anything such as an NPC
769s	a mount a tint a piece of card cargo um
774s	pretty much anything that you can
775s	visually see and represent is what we
777s	call an actor and these actors they be
779s	replicated which we'll get into on the
781s	next slide and PC would be a player
784s	character for those who don't
789s	know personal
792s	Caravan so touching on replication here
795s	um replication is a term you might often
797s	hear in networking I believe Zach
799s	touched upon that a little bit briefly
800s	in his
801s	introduction um replication is the
804s	process of synchronizing some sort of
806s	State you can think of it like a value
808s	or a property that's on an actor on the
810s	server and then it's synchronizing that
812s	state from the server to all the clients
814s	that are relevant to the said actor um
818s	this is an important because of a
819s	concept called server authoritative
822s	networking um the server when you think
825s	of like networking in video games the
827s	server normally has authority over your
829s	game world and the properties they're
831s	normally updated and checked against the
832s	server which can then be shared and
834s	synchronized between your clients which
836s	is through the form of replication
840s	um this is also part of why you never
843s	want to trust the client as we said
845s	earlier um because when networking you
847s	want when the client does some action to
849s	the server for instance you always want
851s	to verify that action on the server in
853s	some way or shape or that's how you can
855s	get hackers or cheaters or something
857s	disrupting your game because they're
859s	able to just send anything they want to
860s	to the server and your server will just
862s	trust it without verifying and then
864s	that's where you can get the concept of
866s	like I said hacking Andor
868s	cheating um so as long as the server
870s	kind of holds your source of Truth the
871s	client sends the data that your server
873s	doesn't agree with you can decline the
874s	actions and or kick or ban the user
876s	depending on how severe the action is
878s	they're taking that you're not trying to
883s	allow and obviously we've seen you know
886s	a lot of games that we've played in
887s	where as um as as the latency becomes uh
893s	worse for certain players or as the load
896s	on the server becomes higher um to
899s	process
900s	that replication can can show some weird
903s	effects in games right wait a minute
906s	what's that person doing that doesn't
907s	look right yep that's when you would
910s	that's typically when you would get like
911s	rubber banding like if your serers
913s	really slow down because you would only
914s	get an update every you know X period of
917s	time yeah so as you can kind of see in
919s	the the video we showed you can kind of
922s	see like player one is going into the
923s	water this is from the perspective of
925s	the server so as player one enters the
927s	server it'll be like okay player one is
929s	swimming so now player two and three
931s	since they're visually seeing player one
933s	swimming they're going to get a state
935s	for player one saying they are swimming
938s	and then that can kind of perform like
939s	the necessary actions of like seeing
941s	them play the swim animation or whatnot
943s	and and and and like I guess to add on
945s	to what Hunter is saying this is the
947s	this is pretty much the entire problem
949s	like you're doing this all the time for
951s	everything in the game right that's
953s	replicated all the time every movement
956s	every ability every thing you click on
958s	interact with
960s	T yeah all the Goblins that are running
962s	around they're playing animations uh the
965s	players moving stuff in their inventory
967s	the the trees leaves moving subtly yeah
972s	the the trees falling being cut down
974s	yeah so not everything needs to be
977s	replicated across the server right some
980s	things can be client only like the ways
982s	the leaves might moves specifically but
984s	when the tree gets cut down that needs
986s	to get replicated across yeah well and
989s	and and you you can do interesting
991s	things to try and like reduce this
994s	problem like like if we're talking about
995s	the wind for example you could just
997s	replicate you know the wind direction
999s	and speed yes and then the client can
1001s	just apply that to all the Tres instead
1003s	of you having you know a specific
1005s	replicated you know wind and speed for
1008s	each tree for example right and that's
1010s	part of the Finesse of replication right
1012s	trying to identifing the problem exactly
1015s	that's right yeah a big part of our job
1017s	especially with working with gameplay is
1018s	trying to figure out how make that as
1020s	efficient and optimized as possible so
1021s	that our game and servers can perform to
1023s	the expectations we need speaking of
1026s	servers the server meshing what
1028s	everyone's been excited to talk about
1030s	right so now we do replication so now
1033s	originally we just talked about how
1035s	replicating a one server works so now
1038s	the first part of intrepid net is what
1039s	we like to call what we basically
1042s	normally know as as server
1044s	meshing
1046s	um in Unreal Engine um your game
1048s	normally works under a single process
1050s	server scenario that means all your
1052s	clients are connected to a singular
1054s	server that runs the entire game now for
1056s	an MMO RPG like ashes of creation um
1060s	this can't work for us like for a few
1061s	reasons Zach talked about in the
1063s	previous slide there's just with the
1065s	mass amount of entities and actors we
1067s	have simulating and processing the
1069s	replication of all that for everyone
1071s	every actor in one server that's pretty
1075s	insane um our solution first and
1077s	foremost here is basically we had to
1080s	modify Unreal Engine so that it is
1082s	capable of having many server processes
1086s	that are kind of meshed together or
1087s	quilted
1088s	together that allows us to use all these
1092s	servers together to make up your whole
1093s	entire game world or your game
1096s	room um this allows us to split up that
1099s	simulation work between many of the
1100s	server processes and each server process
1103s	has its own authoritative state of the
1105s	world that it kind of owns and controls
1109s	um this task it was not a trivial task
1112s	and it required us ripping apart deep
1114s	layers of unal engine and their
1116s	networking stack in order to really
1118s	modify it to fit the architecture that
1120s	we have really
1122s	envisioned um outside of this too
1125s	there's like a lot of proprietary
1126s	programs as well that we made that kind
1128s	of run in the background to help it
1129s	orchestrate all this functionality as
1132s	well and by using under engines API um
1137s	and doing all this we were also able to
1139s	add a lot of minimal overhead for our
1141s	Engineers When developing for our this
1143s	solution and it allows us General
1144s	control over like what how things
1147s	replicate and where they replicate that
1149s	we'll get into in the next few
1152s	slides
1154s	um so right now you see we have like the
1157s	static Dynamic world grid okay that's
1159s	great but now you're probably thinking
1161s	okay now that we stitch many service
1163s	together what is my experience like that
1165s	in regards to traversing the world will
1168s	I be able to see into the other servers
1170s	from mine or will I randomly just start
1171s	seeing actors pop in and out of
1173s	existence right in front of
1175s	me that kind of takes us to this
1177s	slide um so our answer to that of course
1180s	is we want to make this experience as
1182s	seamless as possible for our players if
1184s	you cannot tell what's going on behind
1186s	the scenes and everything that we've we
1188s	are making then that means we did our
1191s	job right if you can tell then it
1192s	probably means something's wrong and we
1194s	need to look at what we're doing here
1196s	and how we can improve and optimize it
1197s	further
1200s	um we've developed what we call here
1203s	inter server replication for this case
1205s	which means that instead of traditional
1208s	replication as you might have as we
1210s	explained was like server to clients
1213s	this is basically representing between
1215s	servers now or replication between
1217s	servers and we do this replication to
1221s	what we call a proxy
1223s	actor um as you can see in the video
1226s	here we can see there was like a
1227s	transparent White uh wall going across
1230s	the screen that represents our in-game
1232s	debug tools with our border regions
1234s	being
1235s	drawn you can see that vexent is on one
1238s	server and then Kevin is on another one
1241s	as he's on the other side of that border
1243s	so what vexent is seeing right now is
1244s	actually not Kevin but a proxy of Kevin
1247s	is what we' have deemed it is it really
1249s	Kevin yeah who knows if only could
1252s	actually duplicate Kevin Hunter expand a
1255s	little bit on you know a lot of lot of
1258s	people who obviously aren't familiar
1260s	with game development they might not
1262s	understand the the terminology of
1263s	debugging tools right so to talk a
1265s	little bit about what a debug tool is
1268s	right so when making a networking
1270s	solution as complicated as this we need
1273s	a lot of information about the state of
1275s	the world what's happening where things
1276s	are going on so we've developed a lot of
1280s	like debugging tools that can be used
1282s	inside the game and outside of the game
1284s	which these tools kind of help us
1286s	collect that information see where all
1288s	the important things are happening see
1290s	where a failure might be going on so in
1293s	this particular instance we have a debug
1295s	tool that's in game that allows our
1297s	developers to kind of draw these server
1299s	borders so that you can kind of see
1301s	while you're playing like okay there's a
1303s	border here um and then if QA is testing
1306s	something they can be like okay we
1308s	transferred across the server and then
1309s	this broke so that means something's
1311s	probably wrong with the server transfer
1312s	so that can kind of just help you like
1314s	narrow down more of the problem space
1316s	and give you that necessary information
1318s	of like what was going on at the
1320s	time um because without that if if we
1323s	have it turned off you pretty much
1326s	hopefully should not be able to tell the
1327s	difference between the proxy we talked
1329s	about or the actual
1331s	actor um you can kind of think of the
1334s	proxies here as a puppet any
1338s	actual um action that was done by the
1341s	actual actor on its own like Authority
1344s	server which is the server that owns it
1347s	it's then replicated to this proxy on
1349s	your server which then gets replicated
1351s	to yourself so that you can kind of see
1354s	the representation of Kevin as if he was
1357s	on the same server as
1359s	you um we do it in a form of like a we
1362s	call it lightweight because it's an
1364s	up-to-date copy of the original actor
1366s	but we can specifically our developers
1368s	can specifically control like what does
1370s	and doesn't get sent to this actor so
1372s	that we're more optimal in like what
1375s	we're sending how we're sending it and
1377s	when we're sending it
1381s	that was a
1382s	[Laughter]
1384s	mouthful I I think I think the other
1386s	thing you know to think about Hunter
1388s	Zach Anton is that um not a lot of
1391s	players obviously followed development
1394s	of the games that they're excited about
1396s	right they they wait until you know the
1398s	game is ready to play and and marketing
1400s	begins and you know they can join in and
1402s	and purchase or whatever obviously ashes
1405s	has followed a little bit of a different
1406s	path where we're giving people a bit of
1408s	insight into you know what it actually
1410s	takes to make these types of games and
1413s	we generally tend not to delve too
1416s	deeply into the highly technical
1418s	components of our development process
1421s	because that's where a little bit of a
1423s	cliff starts to form of player interest
1426s	even you know viewership interest um
1429s	because uh it's nice to have the magic
1432s	work in the background and you don't
1434s	necessarily care or want to know how or
1436s	why it's working just that it feels good
1439s	that the you know it's responsive and
1441s	that you know it accomplishes from a
1443s	gameplay Loop perspective what you're
1445s	expecting it to but you know because of
1448s	the level of complexity because of the
1450s	amount of effort because of what we're
1452s	trying to create is a is a Next
1456s	Generation level of complexity of gaming
1460s	um we feel it's important to provide you
1464s	guys an opportunity to get some insight
1467s	into what we're doing that's unique into
1469s	what we're doing that is going to
1472s	service a long-term mmor RPG at the
1476s	scale that we need Ashes to be and that
1480s	requires some really smart people in the
1482s	room who are creating proprietary
1485s	technology that allows us to achieve
1488s	that in a way that presents players with
1490s	a game at the Quality they expect and so
1493s	I you know I think this is a lot of a
1494s	lot of fun to kind of you know have you
1496s	guys come and and explain this front
1499s	facing it can be a mouthful it can be a
1501s	headful to kind of digest but at the
1504s	same time well yeah I know we're no
1506s	we're near done but at the same time it
1508s	gives you now an opportunity to to
1510s	really see how the sausage is made um
1514s	and I think that's cool I can talk about
1516s	this all
1517s	day people don't talk about this there
1520s	are so many games that are that are
1521s	doing things like this not exactly but
1525s	like you know there are other games that
1526s	have to solve these problems and you
1528s	know other games don't talk about this
1530s	you know this is this is a first
1533s	absolutely
1535s	right yeah so as you see in the video um
1539s	you can see into the other server but
1542s	now you're probably thinking okay what
1544s	if I want to interact with this proxy
1546s	how how do I do that across a server
1549s	boundary well um because the proxy is
1552s	like a puppet controlled by the higher
1555s	authority we basically just now need
1557s	events that we can from the proxy up to
1560s	its Authority and then we tell the
1562s	authority that we want to modify data or
1564s	execute some action on the authority
1566s	server so that it can then replicate
1568s	down to the proxy and then back to you
1571s	um we have what we call a cross server
1574s	event or an inter server event that's
1576s	done through what's normally known as an
1579s	RPC or remote procedure call which
1581s	allows pretty much just for
1583s	this um our goal with entrepren net is
1587s	to make all the routing and actions that
1589s	we do as fast as possible so that we can
1591s	keep the latency and interactions
1593s	between servers a pretty seamless
1595s	experience and you hopefully shouldn't
1597s	even be able to tell the difference
1599s	between like I said a proxy or an
1600s	authoritative actor yeah and I guess to
1604s	add here this is so important because
1606s	you know these these server borders that
1608s	we're talking about they're invisible
1610s	you you you won't know where they are um
1613s	so the I think the big important thing
1617s	here is like we need to have we need
1618s	make it so PVP can happen on these
1620s	borders without people noticing um and
1623s	that's that's that's that's really
1625s	that's the real Crux of it here that's
1626s	why it has to be so
1628s	fast right and as you can see in like
1631s	the video here we have some fighting
1633s	across the border um you can see while
1635s	we're fighting kind of over the boundary
1637s	combat and everything seems pretty
1638s	seamless and smooth as you would expect
1640s	it without any noticeable issues um this
1643s	is kind of showcasing like what we can
1645s	do with just those events and that
1647s	replication we talked about out we can
1649s	have these like indepth battles or
1653s	interactions between the servers with
1655s	hardly any downside yep
1658s	um so but what we're trying to achieve
1661s	here is the seamless experience that you
1664s	hopefully can't tell where that border
1665s	crossing is and and the other important
1668s	thing about this particular you know
1670s	this cross server uh information and in
1673s	the Border itself which I think you guys
1675s	are getting into a little bit later is
1677s	that these borders are not static they
1679s	can change they are Dynamic you're
1680s	you're spoiling it Ste I'm I'm sorry I'm
1684s	sorry I'm so sorry this is my first time
1686s	seeing the
1688s	slide okay go no no go I'm sorry I think
1691s	you were talking about seamless
1692s	transition so we
1694s	can yeah so if you look in or go back
1697s	one more Maggie I'm
1698s	sorry um so if you kind of seen in the
1700s	video there as well um now that we can
1704s	kind of do interactions to the proxies
1705s	across the border you may have noticed
1707s	in the video I did charge myself across
1709s	the border and then I basically attacked
1712s	what we call the proxy of
1714s	Kevin however during that charge we did
1718s	basically a server promotion that took
1719s	place behind the scenes that you
1721s	probably didn't
1723s	notice um so if we go to the next
1726s	slide we can kind of see now we're
1729s	introducing the concept of server
1730s	transfers this is kind of where Intrepid
1733s	net gets very
1736s	complex um this allows us to not
1739s	share and split the statically defined
1741s	server areas but we can also transfer
1743s	actors to new server areas that are
1745s	moving across the world or happen to
1747s	have a new server come up on top of
1750s	it into a different server and then we
1753s	do and then this allows us to do
1755s	basically that Global load balancing
1757s	across the
1758s	room and then so basically when a player
1761s	or an actor is within range of this
1763s	border our system on the back end kind
1766s	of begins a negotiation and the creation
1769s	of a proxy so that it can replicate it
1771s	to the nearby
1773s	server um this doesn't only happen on
1776s	like a one by one scenario that you
1778s	might see here even if it's a 2 by two
1780s	and you're in like the corner of four
1782s	different
1783s	servers your proxy will then get
1785s	replicated be replicated to on three
1787s	different servers from the one Authority
1789s	so that way no matter where you are
1791s	whichever server you're in around that
1792s	area you'll still be able to see this
1794s	actor on a different server
1801s	um if that actor then fully crosses the
1805s	Border going outside of the authority
1807s	region of the server it's currently in
1809s	um this the the two servers then kind of
1812s	start negotiating what we call a
1813s	promotion or a transfer this is when now
1816s	server a is passing over that Authority
1819s	we previously defined to server
1822s	B and then promotions they're like a
1824s	whole different Beast um in order to do
1826s	a lot of like the promotions there's a
1828s	couple things we have to do such as like
1830s	ensuring the first you have to ensure
1831s	the actor is like fully made with all of
1833s	its components and
1835s	children um we have to ensure they're
1837s	all up to date with the entire state
1839s	that was deemed necessary for the new
1840s	copy to have because we don't want to
1842s	lose any data during this entire
1846s	transfer um and Hunter you people hear
1849s	the term like you know children and the
1851s	state and whatnot elaborate a little bit
1853s	about you know what what that means at a
1855s	high level right so
1858s	uh when we have all these entities such
1860s	as like an actor actors can then have on
1863s	them what we call components which is
1866s	basically building blocks that you can
1868s	then add to your actor that kind of make
1870s	up the actor what it does how it's
1872s	supposed to act things like that um so
1875s	for example you could have like an actor
1878s	and then that actor has an interactable
1879s	component so that means you can interact
1881s	with this actor or it could have an
1883s	inventory component so that means okay
1886s	this actor has something of like a bank
1888s	or um it it stores goods and to
1892s	contextualize that let's say you know
1894s	you have your Mount out you're not on
1896s	your Mount right another player could
1898s	walk up to your Mount and they would be
1899s	prompted with an interact that says to
1901s	Mount Your Mount as you're moving across
1903s	the boundary that actor has that
1906s	component that is the interactable
1908s	component that needs to go with it right
1911s	because if you transfer this actor or
1913s	you're proxying it it still you still
1915s	need to be able to do that interaction
1916s	across the boundary so we proxy that
1919s	interaction component and just like we
1921s	can do crossover events on the actor
1923s	itself we can also do them at the
1925s	component level so that way it can
1926s	communicate with its authoritative
1929s	one um because we kind of treat the
1931s	proxies as like a readon concept so the
1934s	proxies are just visually kind of
1936s	representating what the authority
1938s	represents so any data modifications or
1941s	anything that has to happen has to
1942s	happen at the authoritative level so
1944s	that way then the proxy can get updated
1946s	appropriately across any Sur
1948s	um that's kind of how we have like that
1950s	Central truth of the actor is whatever
1952s	server it's Authority on and there's
1954s	only ever one copy of that I just want
1956s	you to know this conversation so far has
1959s	probably induced about 30 to 50 people
1963s	now wanting to take computer science
1965s	degrees that would be awesome yeah no
1968s	it's it's pretty cool stuff
1971s	um it definitely took a lot of time and
1974s	a lot of effort to try to figure out how
1975s	we make these things possible and after
1977s	all this work this is we're excited to
1979s	be able to show it um although watching
1982s	Kevin run back and forth is making me a
1984s	little
1985s	nauseous yeah so in the scenarios 2 this
1989s	also works where you may not be proxying
1991s	to that nearby server because say you
1993s	teleport across the
1994s	map um this kind of means that when
1997s	promoting there it kind of does the same
1999s	exact route except it may just need to
2001s	make your actor instead of making a
2003s	proxy of it so it all falls kind of
2005s	under the same generalized um system
2009s	no um and then when we normally do have
2013s	that proxy actor on the other server
2015s	that means when you're promoting we can
2017s	reuse your existing two actors that are
2019s	from server a and server B your
2020s	Authority and the proxy and we can
2023s	convert that proxy to the authority and
2025s	then demote The Authority one to a proxy
2028s	so this allows us more to have that more
2030s	seamless experience when you're swapping
2032s	servers so that way it's a lot faster
2035s	you don't need to send a lot more data
2037s	to copy over to be created and then
2039s	there's not really any visual hitching
2041s	at all when this happens because we're
2044s	not spawning or destroying a new one on
2045s	the fly during that
2048s	promotion and this helps also with
2050s	client
2051s	performance it it can um not going too
2055s	too into detail but yeah talk actors
2059s	aren't actors really aren't cheap so
2061s	replicating an actor can be expensive
2064s	and every new actor you kind of open and
2066s	close to the client which which is
2068s	basically like a create or a destroy
2071s	that just adds this additional overhead
2073s	that you we didn't want so we can kind
2075s	of negate it by just reusing the
2077s	existing one we already
2079s	have and then doing that swap so that
2082s	way you don't even notice anything
2084s	happened because something Stephen no no
2087s	he got it on the spawn and Destroy nice
2089s	okay yeah yeah like like I said our goal
2092s	is to make it as seamless as possible
2094s	and one of the main pillars when we were
2096s	making this uh ire system
2100s	was we try not doing anything on the
2103s	client if all of our actions are kind of
2105s	done on the server in a fast and
2108s	efficient way the client shouldn't
2109s	notice anything happening at all and
2111s	that's kind of where this comes in
2114s	because we don't want the client to
2115s	notice so every time we're doing
2117s	something we're like okay how can we do
2119s	this without affecting the client's
2120s	experience because we don't want them to
2122s	have a bad time
2124s	yeah okay so um let's talk a little bit
2130s	about how all this stuff is
2133s	working uh because yeah we are
2135s	communicating between servers we're
2137s	sending events between servers we are
2139s	transferring between servers but how is
2143s	that information flowing how do the
2145s	clients end up seeing that stuff and
2149s	typically uh the way this is done is uh
2153s	you create a centralized
2155s	replicator and all the different ser
2158s	servers from all across the realm send
2161s	all the relevant information about all
2163s	the different entities to that
2165s	replicator so let's say one server has
2168s	um a bunny and that bunny moved around
2172s	it sends that information to the
2173s	replicator and now the server has a
2175s	goblin that took damage it will send
2177s	that information to the
2179s	replicator and then all the players will
2182s	connect to that replicator and the
2184s	replicator will pass pass through all
2186s	those different entities
2189s	decide which player should see which
2191s	entities and send that information to
2194s	the specific
2196s	player however this creates a few issues
2200s	and you might actually be able to see it
2201s	from that graph there that we've just
2203s	created a
2205s	bottleneck uh as this singular process
2207s	now may need to keep track of hundreds
2210s	of thousands of
2211s	entities and
2214s	send potentially hundreds of millions of
2217s	updates to all those clients because all
2220s	those bunnies moving and the goblins and
2223s	the players and the trees falling all
2225s	that information has to be passed
2227s	through the replicator and then sent to
2229s	each individual client and I guess
2232s	something to add here too Antoine is
2233s	like this is the that would just be
2235s	moving the problem we're trying to like
2238s	like if we had a single server this this
2240s	would this would be the same problem
2242s	where all all the replication is
2244s	happening on one place and we're trying
2245s	to we're trying to not do that Yes
2248s	actually we tested a solution like this
2251s	uh during alha one and this
2255s	bottleneck it kind of affects how many
2257s	players you can have in the realm as a
2260s	total like you can spawn all the servers
2262s	you want but one at some point you hit a
2264s	limit on the replicator itself and it
2266s	doesn't matter how many more servers you
2268s	spawn you are limited by that cap on the
2271s	replicator uh during alpha 1 the number
2274s	we got was like I think about a th000
2277s	players
2278s	where after hitting that point you can
2281s	no longer really scale anymore and it's
2284s	important to note that because of this
2288s	more I guess for lack of a better term
2291s	archaic and monolithic approach which
2293s	which many of us have likely experienced
2295s	in older games there are certain
2298s	scenarios in those games that are
2300s	multiplayer where you start to see that
2302s	server is having trouble and it's
2304s	because it's hitting these types of
2306s	Choke points
2308s	yes and in fact when this replicator is
2311s	overwhelmed the entire realm will lag it
2313s	doesn't matter if you're alone in one
2315s	server in a remote island you will lag
2318s	because there is a big battle on the
2319s	other side of the realm that's I
2321s	remember in so many so many games where
2323s	I would be out hunting and peaceful
2326s	nobody's nearby and I'm noticing
2328s	everything is starting to churn and I'm
2330s	like what is going on and then I'll look
2332s	into Global chat and I'll see oh [ __ ]
2335s	they're having a huge fight somewhere
2337s	across the realm in a totally different
2339s	part of the world and I am suffering as
2341s	a result of it
2345s	indeed I guess it's a good bridge to our
2348s	to what we're doing yeah it is a pretty
2349s	good Bridge cuz basically we yeah we
2354s	ases of creation is trying to aim for
2356s	player counts that we found out that
2358s	centralized replicator does not support
2361s	so we had to come up with our own
2363s	solution and uh this is from what we
2366s	know not a very
2368s	common solution possibly because it's
2371s	more difficult to pull off uh what we
2375s	basically did is we completely got rid
2377s	of that centralized replicator and
2378s	instead we had the servers themselves
2381s	collaborate with the other servers
2383s	around them and exchange information
2386s	that way that and that makes each
2389s	individual server have all the data they
2392s	need to serve the clients that are
2394s	connected to them uh in our scenario
2398s	over here you can see we have four
2399s	servers and two clients are connected to
2402s	each one of those servers if we had a
2404s	centralized replicator all those four
2407s	servers would be connected to that one
2409s	replicator and then all eight clients
2412s	would be connected to that one
2413s	replicator to so we've actually
2416s	distributed the problem we've eliminated
2419s	the bottleneck and we've also eliminated
2423s	this one point of failure that existed
2426s	beforehand where if that one replicator
2428s	fails the entire realm goes
2431s	down and we can actually yeah have a
2435s	look at the uh next slide where we see
2439s	uh how we are more fault
2443s	tolerant uh than if we have a
2445s	centralized
2446s	replicator uh and we can actually yeah
2450s	you can swap to the next slide there you
2452s	go we have the uh the entire server grid
2455s	over there and if one server goes down
2458s	everyone else can uh keep playing it
2461s	like normal we patch that server up we
2464s	fix the issue we bring it back up and
2468s	everyone else never experienced anything
2469s	the players that were connected to that
2471s	one server got booted and then can
2473s	rejoin just fine uh if we had a
2475s	centralized replicator if that
2477s	replicator went down everyone gets
2479s	booted
2481s	so yeah so that's another uh another
2485s	good uh benefit from our approach the
2488s	the really big gain we achieved with
2490s	this was the elimination of the
2493s	bottleneck uh and it actually lets us
2496s	scale we are
2498s	truly distributed we can just spawn new
2501s	servers to a certain point of course and
2504s	this is this is not and by the way Anton
2506s	like this is not just a strength
2508s	obviously of the of the replication
2510s	model of of the of the server grid
2512s	itself you know being distributed in
2514s	such a good way but when you have to
2516s	bring such a mod lithic server down
2519s	because of that because of let's say
2520s	whatever error you can't determine that
2522s	has a Cascade effect across you know
2525s	everything you're having to to affect
2528s	the database to affect the login servers
2530s	to have the player cues going off again
2533s	the ability for this to truly be
2535s	distributed and for us to be fault
2538s	tolerant is an entirely different player
2540s	experience when it comes to these large
2543s	massive MMO RPG uh type settings yeah I
2546s	mean it's it's very typical you know
2548s	like with you know an MMO like that
2552s	something else is happening in the world
2553s	somewhere someone's hacking someone's
2555s	you know there's a really large maybe
2557s	there's a really large raid or or Siege
2559s	happening and the the server Falls over
2563s	and then everyone gets kicked with our
2565s	model that doesn't happen the the rest
2567s	of the server is fine performance is
2569s	still fine in the rest of the server
2570s	people can keep playing if that server
2572s	goes down we you know we we'll deal with
2574s	it we'll spin it back up and it we'll
2576s	just seamlessly rejoin red once it's
2578s	once we're back in a good
2582s	State and this like eliminating the
2585s	replicator was um definitely not easy uh
2590s	and it actually it caus us quite a few
2593s	problems but let's investigate one of
2595s	those problems and the solutions we came
2598s	up to uh solve it
2602s	so the because we did not want to create
2606s	an
2609s	really large connection network between
2612s	all the different servers we essentially
2614s	connect the servers to the
2617s	nearby servers the neighboring or like
2620s	some that are close by uh and that is
2624s	fine in most situations over here we can
2626s	see there's two players in um two
2629s	neighboring servers those those servers
2631s	communicate so if those two players are
2633s	in a party even though they might be a
2636s	little bit too far away to see each
2637s	other
2638s	they can still see each other on the map
2641s	however in a more complex situation
2644s	where that other player is maybe on the
2647s	other side of their realm now those
2650s	servers do no longer have a connection a
2653s	direct connection to each other so we
2656s	have created a problem how can player
2660s	one see player two when player two is on
2662s	the other side of the world when if we
2664s	had a centralize replicator sure we have
2667s	the data just there we can figure it out
2669s	but we do not uh what do we do about
2672s	that and uh to solve uh that uh issue we
2677s	created a concept called
2681s	microservices microservices
2684s	are Imagine them kind of
2687s	like servers on top of the servers like
2691s	um like some database for example but
2694s	it's active it's not just storage
2697s	they're doing
2698s	work uh we have a node service that is
2701s	responsible for um leveling up and
2704s	keeping track of all the nodes and the
2707s	upgrades that have to be done to to
2709s	nodes we have a population service that
2712s	is doing the spawning across all the
2715s	Realms and we have a Social Service
2718s	which would be for example what
2722s	um what would affect the party example
2725s	that we just uh talked about
2728s	this means that server one and server 8
2731s	on the diagram can connect to that
2733s	social service and exchange the
2735s	locations of those two players that way
2738s	player one and player eight can have
2741s	exactly the information they need to be
2744s	able to see each other on the
2746s	map and we act we yep what was it
2750s	Stephen oh nope that wasn't me oh that
2753s	wasn't you I can't s my bad um so uh
2757s	yeah and this actually gave us some
2759s	extra benefits we were able to make some
2761s	pretty cool tools with it because we
2764s	wanted to be able to see to have like an
2767s	overview of what happens to the entire
2769s	realm we wanted to be able to keep track
2771s	of all the players all the NPCs see all
2774s	the different gathers being spawned see
2777s	all the
2778s	trees and that's kind of hard to do if
2782s	we just connect to a single server and
2784s	then see what's on that one server and
2785s	then jump on another server and see
2787s	what's there so we created a service
2790s	that keeps track of all entities in the
2792s	world and a tool that can allow us to
2796s	view those
2797s	entities and this is a pretty cool tool
2800s	because we can see the players moving
2802s	around we can see them uh participate in
2805s	events we can see the NPCs dying we can
2807s	see them collect gather bles so if
2810s	there's any issue with the spawning on
2812s	GA of Gables will detect it and
2815s	hopefully fix it it's a I think the big
2817s	thing to to say here is like you know we
2820s	can we can see the entire world state
2822s	which we yes generally can't do because
2825s	we're so distributed and everything's so
2827s	server Central but since we have these
2829s	Services we can make a debug service
2831s	like this and we can see the entire
2833s	world yeah watching you yeah I also
2837s	wanted to piggyback off of that Anton of
2839s	your services um the services as well
2842s	are also pretty much distributed like
2844s	going for our model so that if one of
2847s	the systems go down it's not taking down
2849s	the entire all every game system so if
2851s	we just if the chat service goes or the
2853s	Social Service for example goes down you
2855s	may just not be able to chat or form a
2857s	party for a few minutes until it comes
2858s	back up but you should still be able to
2860s	interact with like your guild your node
2862s	your mail Etc and we try making the
2865s	services as well fault tolerant like we
2867s	explained earlier um so that way they
2870s	can come back up go down come back up on
2872s	the Fly get hot fixed and so that we can
2875s	still provide that seamless experience
2877s	while the feature may not be working
2879s	everything else should still be working
2880s	in the meantime right and because
2882s	they're they're componentized in this
2883s	you know microservices approach right
2886s	there is not an interconnectivity where
2888s	you have one failure in a single service
2890s	causes a cascading effect across all the
2893s	services right exactly they're they're
2894s	independent of one another that allows
2896s	for a recovery time a recovery process
2899s	to stand back up while the other
2901s	services continue their operations
2903s	exactly and then these debugging tools
2905s	allow us to see where those problem
2906s	areas are or what what situation or
2909s	actor might be causing a problem right
2911s	exactly we don't have all the tools
2913s	slide for all the tools but we do have
2915s	monitoring tools for all the different
2917s	Services we know when they're lagging we
2919s	know when they're down we know we can
2921s	monitor pretty much everything it's
2922s	pretty cool yes it's very
2925s	cool cool now let's get let's get into
2927s	the really fun stuff now next one is
2929s	this one is the community has actually
2931s	been talking about quite a bit yeah I
2933s	saw a bunch I saw a bunch of videos
2935s	people are people are have been talking
2937s	a about this yeah so go into it Anton
2940s	because this is this is your thing yeah
2942s	oh God I have been I've been looking
2945s	forward to like since I joined the
2947s	company I've been wanting to implement
2949s	this and uh this is um an answer to a
2955s	problem and that problem that he trying
2958s	to answer is sure you have all those
2961s	different servers but usually player
2964s	activity tends to condense to certain
2967s	are areas let's say a large PVP battle
2969s	is happening at some place um and when
2974s	that happens that one server can be
2977s	overwhelmed and you usually wouldn't
2980s	want to just create tiny servers for
2984s	everything because that can be
2985s	incredibly expensive and
2988s	wasteful but having those smaller
2991s	servers would be really useful so what
2994s	if you can do it dynamically what if you
2997s	can detect when a server is overwhelmed
3000s	and then split it this is what dynamic
3003s	grading it is and it can be a little bit
3007s	of a little bit tricky to implement uh
3010s	the difficulty of it is that you just
3013s	now split a server and you want the
3015s	players to not realize they were casting
3018s	abilities they were running around and a
3021s	new server just spawned they were moved
3023s	there and no one realized that is a
3026s	difficulty there I want to add here is
3028s	like these cases in which we're
3029s	splitting a server they're complex
3032s	because if we're splitting a server
3033s	there's a lot going on so it's not like
3036s	we can like there's just a couple people
3037s	we can split them we can do like a a
3039s	quick fade and it will be fine if we're
3042s	splitting the server it's because it's
3044s	it's
3045s	underload and there's there's a lot of
3047s	action going on so we want to keep that
3049s	action going on and not interrupt it
3051s	while we while we balance the load
3052s	across multiple servers and and it's
3054s	important to you know call back again to
3056s	the very beginning of this conversation
3059s	right how do you determine the types of
3061s	architecture necessary you must first
3065s	understand what is the mission statement
3067s	of the game and Ashes is built with a
3070s	lot of open world Dynamic and emergent
3074s	player behaviors that are attempting to
3077s	encourage High player densities at any
3081s	point within the open World locations
3084s	which makes this particular problem a
3087s	requirement when determining
3088s	architecture of how do we address this
3092s	right and I'm not familiar with and you
3094s	know Zach Anton Hunter I'm not sure if
3096s	you guys are but I'm not familiar with
3098s	any other techstack or networking
3100s	approach that has a dynamic griding
3103s	solution in order to better accommodate
3106s	and facilitate that emergent player
3109s	density Behavior yeah I mean we we we
3113s	are not the first people to try this
3114s	this is this is something that a lot of
3116s	people have tried it's it is you know
3119s	the the answer to to scalability if you
3121s	if you can get it right and that is you
3123s	know that is our our mission here is to
3125s	get this right we want to be the people
3128s	that solve this problem and because if
3131s	we solve this the the the the level of
3134s	battles and uh dynamicism I guess we'll
3137s	be able to have in ashes is going to be
3140s	unparalleled to any other game yeah and
3143s	to make something clear we're not
3144s	talking about the far future five years
3147s	from now uh this is current currently in
3150s	development and we are uh hoping at
3154s	least to have it available by Alpha 2 um
3158s	and the server mesing part of it is
3160s	already in game yeah so the dyamic
3164s	aspect of yeah so the static part yeah
3166s	is already is already going it's been
3168s	going for manys now yeah years years now
3171s	um and you know the the all of our Dev
3174s	test servers people are on them every
3175s	day running it it's fine
3177s	uh this is this is like you know if
3180s	where where we're investing our Dev time
3182s	right now is getting this perfect mhm
3185s	get getting this ready for the entrance
3187s	of Alpha 2 Y and hopefully ases of
3191s	creation is going to be uh the first
3193s	game that implements this technology and
3195s	actually deploys it yes that would be
3197s	really fun you got to work quickly guys
3200s	if you want that moving
3204s	moving very cool this is exciting
3210s	it up to uh to Hunter yeah yeah I I
3213s	guess now that we've you know all these
3216s	slides before this we were talking about
3217s	Distributing the problem separating the
3219s	problem um into multiple servers so we
3223s	don't have to deal with like you know
3224s	thousands and thousands of players and
3227s	on on the same server doing
3229s	replication um but you know we still
3231s	have like the core problem here right we
3235s	can only split servers so far we can't
3236s	you know make them like 10 met by 10 met
3238s	we can't do that it's going to be too
3240s	small so we need to we need to still
3243s	support you know a high you get a server
3246s	you get a server exactly exactly so we
3249s	still need to solve the like the core
3251s	problem of how many players can we get
3253s	on a server have them replicate and
3255s	simulate at the same
3258s	time so uh I'll pass it over to 100
3261s	he'll kind of go into like I guess the
3263s	problems that we have when we're when
3265s	we're with with with unal replication
3268s	and how we've gone about speeding them
3270s	up so we can uh support many many many
3274s	uh hundreds of players on a
3276s	server yeah um so kind of piggybacking
3279s	off what Zach said like what do we do
3281s	when a large amount of players are say
3283s	following your favorite streamer into a
3286s	a small area and then like we can't just
3289s	keep dividing our servers into such tiny
3291s	areas and
3293s	that you're needing a single server
3296s	that's still
3297s	of a reasonable size to then be able to
3300s	process the heavier load and replication
3303s	that's happening in that exact area at
3304s	that exact
3305s	moment um so in order to do that we have
3309s	to then figure out how we can increase
3312s	the number of players and entities that
3314s	a single server can support and this
3317s	doing so will also help us increase our
3319s	overall like concurrent user limits per
3321s	realm um because as a rough napkin math
3323s	of course you know you take the players
3325s	per server multiply that by the number
3326s	number of servers and there's your CCU
3328s	right it's that easy yeah that's how it
3332s	obviously um so I'm going to kind of go
3335s	into now uh we're going to touch back up
3338s	on kind of what we explained earlier
3339s	like why is replication
3342s	expensive like what's going on
3345s	um so we kind of wanted to explain that
3350s	like on this slide here for example you
3351s	can see an example of a CPU performance
3355s	profile that we took during a large
3357s	scale bot test um you can see here that
3361s	like the single frame we're staring at
3363s	took 150 milliseconds for example now
3366s	150 milliseconds may not seem like much
3369s	but when you want to for example you're
3370s	talking about servers and tick
3372s	rates um and if you're trying to Target
3375s	say a 30 or 60 frames per second or
3377s	ticks per second server you only get a
3380s	budget of about like 33 milliseconds or
3382s	16 milliseconds per frame in order to
3385s	keep that stable High
3387s	tick time can you talk a little bit
3390s	about what you're referring to when
3391s	you're when you're talking about tick
3392s	rates and
3394s	ticks um
3397s	so as your game server runs
3400s	every it it runs and processes basically
3403s	the entire world at like an interval of
3406s	ticks or frames is what it's called the
3408s	faster you can do all that processing
3410s	the faster your gameplay and experience
3412s	is going to be um if you're targeting
3416s	say roughly 30 to 60 frames per second
3418s	which is a pretty good um targets to
3421s	have for like a game server of like an
3423s	MMO that needs to be very fast paced and
3425s	have like very good performance FPS is
3428s	different than client FPS yes but but
3431s	but they they 30 is not acceptable I
3435s	just want to make they're vaguely
3436s	coupled so you one one of the things you
3439s	would notice like if server performance
3440s	went down is you would notice rubber
3442s	banding like when you see like people
3444s	just jumping around the world or you
3446s	would notice your latency goes up and
3448s	you notice that by like you're you know
3449s	you're trying to you know do PVP and you
3452s	feel you know you you you cast Fireball
3455s	and you feel the latency between the
3457s	time you casted it and when it actually
3460s	executed on the server so the the this
3462s	this this time these like these tick
3464s	times that we're talking about that's
3466s	what it correlates to for the player
3468s	right because normally when you do an
3469s	action the client triggers something it
3471s	tells the server hey do this and the
3473s	server then replicates back to your
3475s	client if these frames are taking a long
3477s	time to process in between that means
3479s	your server just has artificial latency
3481s	and it's taking a lot longer to then
3483s	process that incoming message to then
3485s	return back to you um so like in this
3488s	150 millisecond scenario if our game was
3491s	steadily running every frame at like 150
3493s	milliseconds of time to process that
3496s	would drop our server down to roughly
3498s	like six ticks or six frames per second
3500s	which then you will really heavily
3502s	around that time start feeling like
3503s	latency rubber banding uh Dr packets Etc
3508s	and and then that's not acceptable for
3510s	an action combat PVP MMO like it just
3513s	doesn't work
3515s	right so I played those games it
3519s	definitely doesn't
3520s	work we all have blamed lag before
3524s	right so so so we know that this is
3526s	unacceptable what do we have to do to
3528s	optimize this right so now this kind of
3532s	tells us replication and now it's like
3534s	okay let's dive a bit deeper into why
3536s	this replication is very
3539s	expensive um one of the big excuse
3544s	me one of the big um major factors
3547s	replication can be slow is because first
3550s	when you want to do your replication you
3552s	have to gather and determine basically
3555s	relevancy of every actor in the game to
3558s	not only your connection but every other
3560s	connection that's being processed in
3561s	that
3562s	frame so relevancy is used and needed to
3566s	determine basically how far away an
3568s	actor is from your player and then there
3570s	might be some other like comment some
3572s	other checks that they may need to do
3573s	like is invisible or not but for
3576s	Simplicity sake we're just going to base
3577s	it off a distance for now so then it's
3581s	basically comparing that distance to you
3584s	and saying okay are you eligible for me
3587s	to replicate to you because we obviously
3589s	don't want things being considered to
3591s	replicate to you if it's like across the
3593s	world like you don't care what's
3594s	happening on the far Northern end of the
3596s	the continent while you're all the way
3597s	down on the southern end like that's
3599s	just that's just going to waste a lot of
3600s	processing and
3602s	bandwidth um so the relevancy checks
3607s	like this gets way more expensive the
3609s	more players and more actors you have
3611s	that are replicated because then it's
3613s	for every actor for every player am I
3616s	within range of them yeah and when we're
3618s	talking about an MMO where you know
3620s	you're going to have thousands of
3621s	players on the server thousands of NPCs
3624s	uh tens of thousands of replicated
3626s	actors you know you can't you just can't
3628s	check all you can't do a distance check
3630s	on all of those things in a reasonable
3632s	amount of time right and so this is an
3635s	optimization here for networking
3636s	purposes but it also leads to then
3639s	slowing down your CPU resources because
3642s	again you have to do that like every
3644s	tick or at every whatever interval you
3645s	set to do these
3647s	checks um as we if we looked at the
3650s	previous example for example I believe
3652s	the I don't think it shows in the
3654s	profile but roughly a little bit over
3657s	onethird of that total time and that
3659s	profile was just doing the relevancy
3662s	checks um a good thing about our model
3666s	though because all of our servers are
3667s	distributed they only have to do those
3669s	relevancy checks for just everything
3672s	within that then distributed Zone as of
3675s	if we had that Central replicator it
3677s	would that replicator would then have to
3678s	still process every single player every
3681s	single actor Etc in the world in some
3683s	way shape or form to then determine if
3685s	it should be relevant to you or or not
3687s	so by splitting up and doing the
3690s	distributed server load like we're doing
3692s	this allows us then to kind of reduce
3694s	that problem area but it's still a
3696s	problem yeah it's still really big
3698s	because it's still a really big problem
3700s	now adding more servers to the solution
3701s	that can make it smaller and smaller but
3704s	eventually there's like we want to get
3707s	it as fast as possible so it's like how
3709s	how how do we make this better how do we
3711s	make this faster to be clear that
3713s	profile was using uh inter it had a
3717s	server messing and it still was 40 35%
3722s	uh uh relevancy checks so we need to
3725s	reduce those relevancy checks uh how how
3728s	do we do that how can we tell unreal not
3732s	to do all those unnecessary
3735s	checks well unreal in recent years
3739s	introduced something called the
3741s	replication graph and that is something
3744s	that gives the
3746s	developers
3748s	the ability to write a custom solution
3752s	into how to handle all those uh
3755s	relevance checks and how to determine
3757s	which player should see which actors
3759s	when it comes to
3761s	replication uh and this can actually
3766s	lead to some really great performance
3768s	games we'll look at it in a bit uh but
3771s	let's start and uh to and see how we use
3775s	this replication graph to solve
3777s	specifically this relevancy
3779s	check on the map over there we see uh we
3784s	you still cannot see too far away but
3787s	there are a couple actors that are
3789s	really close to the player and that
3791s	Goblin over there and a loot box and
3795s	there is also um a bear a little bit on
3799s	a medium distance away and a bunny
3801s	really far away now imagine that keep
3804s	stretching for 500 met more and there's
3806s	200 more bunnies and without the
3809s	replication graph we would have to check
3812s	every single one of those bunnies and
3814s	say is this within my distance is this
3816s	within my distance and that's that's a
3819s	problem that's expensive to do so what
3822s	we're going to use the replication graph
3825s	um to do is we are going to create this
3828s	grid where we eliminate all the cells
3832s	that are far further away from us and
3834s	only check the nearby
3837s	cells that allows us to only check these
3839s	three actors that are around us skipping
3842s	potentially hundreds of others actors
3844s	that are further
3845s	away and uh this can lead to a pretty
3849s	big performance Improvement we noticed
3852s	and this is like highly varying
3854s	depending of the scenario of course but
3857s	on an average scenario we noticed about
3859s	a 1.5 times performance Improvement like
3862s	a 35%
3864s	reduction uh in uh our replication
3867s	cost this is this is pretty
3871s	good uh but it's still not fast enough
3874s	and if we analyze the problem a little
3876s	bit we'll see
3878s	that we are we we're talking and
3881s	thinking about like the NPCs and the
3884s	bear and the
3885s	Goblin but we are also comparing our
3888s	distance with all the trees with all the
3890s	bushes that you can pick up with
3892s	potentially some buildings that are
3895s	destructible
3897s	why are we doing that can we do
3900s	something to improve this and the answer
3902s	is yes we can uh what we can do is we
3906s	can monitor and see when these um ores
3912s	or these trees are
3915s	changing and when they're not changing
3918s	we can move them out over our grid so
3921s	we're not even considering them at
3923s	all uh we might have 2,000 trees in the
3927s	area well we're not even going to look
3929s	at them because none of those trees just
3931s	got cut down if someone is cutting them
3933s	down then yes we're going to move them
3936s	temporarily into the special grid and
3938s	then once they're done cutting out and
3940s	everyone sees that they were cut down we
3942s	put them back to the dormy bucket it's
3945s	essentially looking imagine like the
3947s	actor is going to slip for it a little
3949s	bit and this this gives us a really
3954s	really large Improvement we noticed in
3956s	our test about five times Improvement in
3959s	replication times so so
3963s	far uh if we also include the previous
3966s	optimizations we've improved our
3967s	replication performance by seven
3970s	times and yet that's still not fast
3973s	enough it's still not fast enough one of
3976s	mainly because of our worst case
3980s	scenario and that worst case scenario is
3983s	people keep packing in that one small
3985s	area
3986s	there's an event happening there's some
3988s	point they need to capture 200 people
3990s	run in the same area yes she can ignore
3993s	the um relevancy to some Far Away tree
3997s	that's great that's a game that's uh
4000s	more than welcome but if 200 players can
4003s	see each other that means that 200 time
4007s	200 you're sending 40,000 updates per
4010s	tick yeah and I guess one thing to add
4014s	there Anon this is typically when a game
4017s	would introduce a new layer or um a new
4020s	Shard this is like typically that case
4023s	um but as we talked about earlier one of
4025s	our goals is we don't want we don't we
4027s	don't want to do layering we don't want
4028s	to do sharting we want everyone who's in
4030s	the same area to be able to see each
4032s	other right and you could you could then
4035s	use like Dynamic greeting to help in the
4037s	scenario but again we can only make the
4038s	servers so small and dynamic before we
4041s	kind of run out of that room so we still
4043s	have to improve the scenario that's kind
4046s	of happening around you indeed so we
4049s	need to figure out a way to further
4053s	improve this worst case scenario and
4056s	this is where we kind of hit an engine
4060s	limit um and this engine limit is that
4065s	the replication graph of unreal
4069s	is the replication tick in general not
4071s	just the replication graph is single
4073s	threaded and that's because the engine
4077s	in general is single-threaded now what
4080s	does that mean what does single threaded
4081s	mean it means that all the processing
4085s	happens in one core of the
4089s	CPU uh and uh we can move on to the next
4093s	Slide by the way but
4098s	the that means that even though the
4101s	server might have four cores um all the
4106s	other three cores are going to stand
4108s	there and wait for the first thread to
4110s	do all the
4112s	processing uh this is done normally in
4115s	unreal because it's it was not designed
4118s	to be
4120s	multi-threaded uh it was it is a very
4122s	single-threaded engine and replication
4125s	is probably the most single threaded
4128s	system out of all the systems in
4131s	unreal uh so making it multi- threaded
4136s	is incredibly difficult and uh at least
4141s	I am not aware of it being done before I
4144s	tried to do some research around it at
4146s	the time and uh everyone was saying
4149s	basically that it's
4151s	impossible but uh we did it anyway uh we
4155s	persevered through and uh we
4157s	multi-threaded the unreal replication
4160s	Tech and uh well looking at the
4164s	results they did not disappoint uh we
4168s	actually gained and there's a graph
4171s	showing it on the next slide but we
4174s	gained a
4176s	2.5 times Improvement 2.4 times to be
4180s	more exact in a replication text huge
4185s	yeah it was a really really big
4187s	Improvement in uh a replication and it's
4190s	probably necessary to achieve the player
4193s	counts that we're targeting for I don't
4195s	I don't thinking would be possible
4196s	without doing
4198s	this uh and the the really important
4202s	thing to note about this optimization
4204s	and why it's like such a big deal
4206s	compared to like the five times
4209s	Improvement we saw before from dorcy is
4212s	that this
4215s	Improvement helps equally even on the
4218s	worst case
4219s	scenario the other ones in the worst
4221s	case scenario they're helpful but
4224s	they're not that helpful this
4227s	is incredibly important even on the
4229s	worst case
4230s	scenario and if we add it on all the
4234s	other improvements we've done so far
4236s	we've actually achieved a total
4238s	Improvement of total reduction or
4241s	replication take by
4243s	94% so we are only at 6% of the original
4246s	cost for uh replication uh this was
4251s	actually a huge win for uh our team um
4255s	but but um
4257s	it there are still there's still many
4260s	more things that need to be
4262s	done uh as of creation is
4265s	um an incredibly ambitious MMO in terms
4268s	of scale uh and uh we need to do a lot
4275s	of work in order to heit our goals and
4278s	you know this is this is obviously again
4280s	going back to the start of this whole
4282s	conversation what is ashes of creation
4284s	trying to be
4286s	and what does it then require of the
4288s	text act we are using in order to become
4291s	that and when we said in 2017 that we
4295s	are looking to bring massive back to
4297s	massively multiplayer having an engine
4300s	with the graphical Fidelity the front
4302s	end that unreal provides is
4306s	so is so important to the immersive
4311s	nature to the beauty of the world to you
4314s	know the quality of the setting but then
4317s	coupled with a proprietary backend that
4321s	can facilitate the same player
4324s	experience that a massively multiplayer
4327s	World wants to provide that's the dream
4331s	that ises is trying to be
4336s	exactly phenomenal work here guys this
4338s	is this is huge yeah I I'd like to like
4342s	take a moment I guess to shout out the
4344s	team the team has done an incredible job
4346s	you know we
4348s	are we we we are we are pushing MMO Tech
4351s	to where it's never gone uh and we
4353s	continue to we we intend to keep doing
4355s	this moving forward Hunter Alex Kevin
4358s	Andrew Anton Adam ish Doug you know they
4362s	they've all done amazing work uh you
4364s	know to get us to this point where we're
4366s	at today and I'm sure they'll keep doing
4367s	it moving forward um if you're
4371s	interested in working on problems like
4373s	this you know yes send us an application
4375s	you know we're always looking for people
4377s	who want to work on this who have
4379s	experience working on Tech like this
4381s	please send this an application we we're
4384s	we're always interested and we know that
4385s	like this this this showcase this this
4388s	Tech talk this Ted Talk this this GDC
4391s	talk you know what whatever whatever we
4393s	want to apply to this uh to this indepth
4396s	technical discussion is going to appeal
4399s	not just to to people who are aspiring
4402s	obviously uh within this field to learn
4404s	it to do it to you know this is their
4407s	career going forward but also to those
4409s	of you who have been within this field
4412s	work and who yearn to work on the types
4415s	of problems that we're discussing here
4417s	in this showcase um we have a phenomenal
4420s	culture at Intrepid we have a phenomenal
4422s	project in our hands that is you know no
4426s	shortage of problems that need to be
4428s	solved um and one thing that our
4431s	community has done a phenomenal job with
4433s	is is helping to evangelize the project
4436s	and the company and we're here to say
4438s	that we have apps open we have positions
4442s	that are available and if you want to be
4444s	a part of this team um or if you want to
4446s	help others find this team so that they
4449s	can apply please share this link share
4452s	this uh video with others uh go on you
4455s	know social media and let people know
4457s	we're working on some pretty Next
4459s	Generation Tech that is going to change
4461s	the face of massively multiplayer when
4464s	it comes to a genre so you know very
4466s	excited and proud of what your team has
4469s	accomplished you guys I know are working
4471s	every single day and doing a phenomenal
4473s	job um congratulations just very well
4477s	done thank you thank you and it's great
4480s	to experience it in game to and I know
4482s	that you know our PI folks have already
4484s	experienced that in our node Wars test
4486s	as well so maybe they didn't really even
4488s	realize it um yeah they might not have
4491s	noticed um but there were definitely
4493s	server workers working hard and that um
4496s	and we were capturing data so we super
4498s	duper appreciate it and as we head into
4500s	Alpha 2 we're only going to get more
4501s	data and be able to continue to push on
4504s	and as uh the folks here have stated um
4507s	moving towards more of that Dynamic
4509s	State yeah um it's been in progress It's
4513s	going to be in progress still for many
4514s	years as we continually find new ways to
4516s	advance it and make it better um we're
4520s	committed to providing kind of like that
4521s	quality we want for our players and as a
4524s	lot of us have played MMO in the past we
4526s	just like networking was the big stopper
4530s	for a lot of things and we're trying to
4532s	do something different and trying to
4535s	make that a better experience for not
4537s	only our game but many more games to
4538s	come if other companies were able to
4541s	like wanting to adapt to a new tech
4543s	style or try to figure out their their
4545s	networking
4546s	issues yeah and of course all of us here
4548s	are huge MMO fans and I know those of
4551s	you who are watching are as well
4553s	obviously if you're sitting through this
4555s	um but of course the the plan for us is
4558s	to continue to create an amazing game
4560s	and push forward and continue to change
4562s	things even after post launch we'll
4564s	continue to try to drive things forward
4566s	so thank you for joining us and thank
4568s	you to Hunter and Zach and Anton for
4570s	your time and putting together an
4572s	amazing deck a little extra shout out to
4574s	Mike for putting together all the graphs
4576s	gra Mike did such a good
4579s	job for helping capture some of the
4581s	little footage pieces to make sure that
4583s	they looked great we super appreciate it
4585s	any final words step inter wrap yeah
4587s	absolutely just you know out in the
4588s	community let us know you know these are
4590s	always the more risky takes that we
4592s	provide insights too in our monthly
4594s	updates because they are so Tech heavy
4597s	and you know we know that this has an
4600s	appeal to perhaps a smaller audience but
4603s	give us your feedback give us your
4604s	thoughts what did you think of this
4605s	presentation how do you enjoy you know
4608s	learning a little bit more indepth under
4610s	the hood conversation about what it
4613s	takes to make these types of games to
4615s	make you know ashes of creation that
4618s	that dream of all of ours um how does it
4621s	appeal to you what are your questions
4623s	what are your thoughts um leave us
4625s	comments in the YouTube channel in the
4627s	uh in on Twitter and different social
4629s	medias and Discord and on our forums
4631s	obviously the community team continues
4633s	to do a bang-up job of collecting your
4635s	insights and bringing it back to the
4636s	team um but we enjoy you know sharing
4639s	this and and giving giving our perhaps
4642s	um perhaps more sheltered uh de Vel
4645s	opers who are who are in the behind the
4649s	scenes making all the magic and
4650s	everything work an opportunity to come
4652s	out and talk about how they are
4654s	literally the magic that makes the
4656s	wheels turn for this bus um and it it's
4659s	it's a great opportunity I think for
4661s	them to come out see a little daylight
4663s	and let and bask in the sun of what is
4665s	the glory that our community looks to
4666s	them and says thank you for providing an
4669s	opportunity to enjoy a game at this
4670s	scale because it would not be possible
4673s	obviously without your guys's hard work
4675s	um and so with that I think we'll see
4678s	you guys back on stream we'll see you on
4681s	the flip side
4688s	[Music]

Transcript (by Youtube)

Recent Ashes of Creation Posts

🧩 Dungeon Puzzles!

Next Livestream - Friday, January 31, 2025 at 11am Pacific

January 23, 10 AM PT - Realms Are Online for Testing!

Pirate Software Interview with Steven Sharif!