Transcript (by Youtube)

2s [Music]
20s hello everyone Welcome to our glorious
23s server mesh technology showcase segment
27s exciting it is I'm Margaret Cron your
30s director of communications and of course
32s with me as always is Stephen shareif our
35s wonderful creative director I'm excited
38s we have a fun showcase today I mean it's
40s this is something that I don't think a
41s lot of companies go into a lot of
44s projects don't really talk about and and
46s you know it can be a little bit on the
48s techie side but I think this is stuff
50s that a lot of players uh like hearing
52s about and I know we got some we got some
54s exciting people to chat with today what
56s do you think and um of course we tried
59s to make it as visually appealing
60s possible shout out to Mike beio as you
64s you may know him from all of our other
65s Graphics but he's our Graphics designer
67s and he did a wonderful job on these
69s slides so hopefully they help portray
72s what our wonderful engineering team is
75s trying to to uh showcase so we will kind
78s of introduce them we have some new
80s people who you've never met before we'll
83s kick it off first with Zack if you want
85s to talk about who you are what you do at
86s the studio and maybe a little fun fact
90s hi guys uh I'm Zach I'm the technical
92s director here uh at Intrepid Studios I
95s run the networking team and the engine
98s team
100s um I'm generally responsible for
102s performance and making sure that the
103s game scales to the amount of players
106s that we wanted to a hefty and lofty goal
110s many of us players will have thoughts on
113s that and of course we have Hunter with
116s us which some of you if you hang out on
118s our Discord maybe have interacted with
120s before
122s for hi yeah thanks for having me um my
125s name is Hunter I am a game networking
127s engineer at inet studios I've been here
129s for almost about four years now and
133s first initially started off as a
134s gameplay engineer and then I slowly work
136s my way over to networking because that's
138s what I've come to find that I love and I
139s love solving the hard problems and very
142s excited to Showcase to you all what
143s we've been working on the gameplay
145s engineering just wasn't torture enough
147s for Hunter so he wanted to go straight
149s straight the be impossible
152s problems oh my God I like I like solving
154s problems like this hard problems I think
157s fun problem to solve yeah um and then of
160s course last but not least
162s Anton hello I'm Anton uh I'm a senior
165s networking engineer over here at
168s Intrepid I moved straight into this
170s position cuz uh yeah I wanted to suffer
174s immediately I didn't want to start with
177s a game play
180s all right well let's kick it off Zach I
183s think you have a little intro here for
185s us as we move
186s in yeah um so Zach just I'm sorry just
190s to start a lot of people you know
192s sometimes don't know what networking is
196s yeah talk to talk to us about what is
198s networking what is a network layer at at
201s its core networking is just the concept
206s of moving a state from the server to the
209s clients
210s that that at its core that is the
212s problem we are trying to solve we are
213s trying to do that as fast as possible
216s with as many players as possible um
219s since we are making a very very large
221s game uh we have a lot of problems that
223s most games don't usually
227s have yeah and and you know with with
231s regards to like an mmpg versus like a
233s peer-to peer versus like a small
235s dedicated server like our networking
237s needs are intense because the types of
241s numbers we're looking at are extreme
244s yeah we we we every everyone on the
246s networking team here has worked on other
248s multiplayer games before and this is
251s definitely the project where we have to
252s get the most creative with how we are
254s going to solve uh our scaling problems
257s you know like we are we we we are
260s working on an MMO most MMOs don't don't
262s do what we're about to do here they
263s don't talk about how they you know made
266s it so they can support thousands of
267s players per server um so we it's very
271s much we have to you know kind of like
272s figure out what what has been done and
274s and do a lot of um R&D to figure out
276s what what we need to do for our game
279s right and kind of answering that
280s question of what we need to do for our
282s game starts out with well what is our
285s game trying to achieve right yes yeah
287s there are a lot of different types of
288s MMOs some that heavily you know utilize
292s instancing um some that you know maybe
295s don't care about replicating to hundreds
297s of players at a given time um so first
300s it starts with a vision what is ashes of
303s creation and then that kind of leads you
305s down the architectural path of okay what
308s does that mean we have to set up because
310s this isn't out of the box Unreal Engine
312s 5 this is a custom Network layer yeah
315s yeah this is this is this is fully
316s Custom Tailor Made for the needs that
319s ashes has like this is not none of this
323s is is is off the shelf none of this is
324s standard unreal this is all built for
326s the kind of game that we are
328s making right and so how do you go about
331s kind of you know determining that that
333s architecture what does that look like
334s for a game like ashes and and you know
336s we've been working on this for a number
338s of years yeah for years um so I guess I
341s guess the first the first step is we
342s need to figure out the requirements like
345s what do we actually need to like what
349s what are Ash's requirements um you know
352s the the first place we start is you know
355s we're an open world PVP game so we can't
357s really control where players go we can't
359s do match uh typical matchmaking where we
361s just you know spin up more servers SP
363s put them on the new matchmaking servers
366s and you know not have these issues we're
368s an open world game anyone can go
370s anywhere um you know we're a highly PVP
373s game so we need to have low latency
375s people can't just uh we we can't do what
378s other games do where you just like lower
381s your tick rates really low and
383s then um just simulate at a really really
386s low rate since we're you know action PVP
389s we can't do that um one of our I guess
394s would you call this a game design pillar
396s Stephen like this is we don't that we
398s don't want to do layering like that's a
399s that's that's that's a very like
402s absolutely game thing we wanted we
404s wanted so like you can go anywhere in
406s the world stand there and the people who
410s are there are there like
412s yeah and I was just going to say like
414s game laring you know obviously everyone
416s on this call is a gamer who's played
418s MMOs and we played games that that do
420s layering yep you know it's just a very
422s different experience when you can switch
425s channels you
426s know yeah yeah it's like a different
429s world a core like MMO memory for me is
433s like you know old wow you would go to
436s the same place
438s like uh and and there would be the same
440s person there you know dancing on top of
442s a mailbox every single time or dancing
444s on top of a corpse and looking at you
447s with Hungry Eyes they were they were
449s always they were always there and they
450s were always and they were always there
452s and with layering you don't get that uh
454s you don't get that that same level of
456s experience that we that we that we want
457s to give uh to our
459s players um and then I guess our last
462s requirement which is like you know just
463s a typical MMO requirement is that we
465s have you know very very high player
466s count goals um and you know typically uh
471s High player count goals with no layering
473s don't really go together so right that's
476s where it comes into like you know we
477s need to get really creative with how
478s we're going to solve these problems some
480s of us you know Hunter Anton we've played
483s these games that have high player counts
485s but a lot of ways that they get around
489s having higher player counts is by being
491s less server adjudicated less server
493s authoritative and more player which has
496s its own risks and isn't really
498s preserving the Integrity of that player
502s experience indeed like some some games
504s do not um do proper physic simulations
508s on the server side to validate movements
510s in order to achieve that kind of thing
512s which means like cheaters uh can have an
515s easier time right going through walls
517s and stuff like that or they well I guess
521s they do layering or sharding where uh
525s they yes they have real high player
527s counts but technically not really CU
530s they're kind of broken up I can't tell
532s you how many times I've I've jumped into
534s either um you know some session based
537s game or I don't want to call any any of
539s them out out but where there's just one
541s single player that's either hacking or
544s or or doing something and they're just
545s running around and ruining the
547s experience for everyone and it's because
549s the way they've architected in such a
551s way it's more it's easier for those
553s types of experiences to occur always
555s trust the client right
559s yeah you just made everyone sweat Hunter
563s do we want to kind of transition that
565s into talking about the issues that we
567s are trying to address yeah of course
572s yeah so um I guess the issue start with
575s it's like you know we're making MMO and
578s unreal um the amount of entities we want
581s on the server um it's just not really
584s going to work with standard outof the
586s Box unreal replication unreal
588s replication and unreal simulation so
591s there there are two problems here it's
592s like okay even if we could fit all the
594s players we want on a single server um
597s you
598s know we we we still can't replicate them
601s because those are two separate problems
603s we have the simulation and then the
604s replication problem um so since this is
609s is a problem and you know there there is
612s no standard unreal solution for this we
615s we we had to go about you know making
617s our own custom solution to solve these
620s problems um and this is where Intrepid
622s net um I guess was born um and trepid
626s net is a fully distributed networking
628s stack it's tailor made for creation
630s fully built in house um none of this is
633s offthe shelf technology it's all custom
635s built for our game
638s um we you know we'll we'll get into
641s these more in a bit but Intrepid net is
644s made up of quite a few different
646s Technologies um the the one that you
649s know people are most interested in
650s people are always talking about is
652s server meshing we'll talk a lot about
653s server meshing and how we how we
655s implemented it for our game um we have
658s inter server replic
660s which is you know we have we have
663s multiple servers and they're replicating
664s to each other how do we deal with that
666s problem we have our own microservice
669s layer for servicing our
673s um uh communication I guess between our
676s servers um we have uh Dynamic griding
681s which is very exciting and you guys and
683s you guys will like a lot and then we
685s have the replication graph um which is
687s just more um custom replication uh I
692s would almost call them formulas um for
694s our game so we'll be going into all of
697s this it's very this is very detailed
699s very Tech heavy
701s so buckle in yeah we need some like
703s theme music for the Intrepid net logo
705s there too I know they were so excited
708s when Mike made a logo they're like so
710s nice of a logo I'm just thinking of like
713s the Indiana Jones theme song right
716s now right
718s totally the word do we start off with
721s this yeah so I'm going to kick it over
723s to Hunter and he'll kind of like start
724s with the like giving some of the base
727s knowledge for like how unreal works and
729s then he'll start with you know talking
731s about how some of our server technology
733s our server meshing technology
735s works yeah hello um so before we start
738s diving too deep into Intrepid net um
741s there's probably a few terminologies
742s that we wanted to preface with just so
744s that it helps the community and everyone
746s else here follow along better with like
747s the lingo and the terminology that we
749s will
750s using um the first concept that we
752s wanted to touch on is an actor um in
755s Unreal Engine you can think of an actor
757s as like almost any physical
759s representation of an object or an entity
762s that's inside the
763s world as you can see on the photo that
766s an actor can be anything such as an NPC
769s a mount a tint a piece of card cargo um
774s pretty much anything that you can
775s visually see and represent is what we
777s call an actor and these actors they be
779s replicated which we'll get into on the
781s next slide and PC would be a player
784s character for those who don't
789s know personal
792s Caravan so touching on replication here
795s um replication is a term you might often
797s hear in networking I believe Zach
799s touched upon that a little bit briefly
800s in his
801s introduction um replication is the
804s process of synchronizing some sort of
806s State you can think of it like a value
808s or a property that's on an actor on the
810s server and then it's synchronizing that
812s state from the server to all the clients
814s that are relevant to the said actor um
818s this is an important because of a
819s concept called server authoritative
822s networking um the server when you think
825s of like networking in video games the
827s server normally has authority over your
829s game world and the properties they're
831s normally updated and checked against the
832s server which can then be shared and
834s synchronized between your clients which
836s is through the form of replication
840s um this is also part of why you never
843s want to trust the client as we said
845s earlier um because when networking you
847s want when the client does some action to
849s the server for instance you always want
851s to verify that action on the server in
853s some way or shape or that's how you can
855s get hackers or cheaters or something
857s disrupting your game because they're
859s able to just send anything they want to
860s to the server and your server will just
862s trust it without verifying and then
864s that's where you can get the concept of
866s like I said hacking Andor
868s cheating um so as long as the server
870s kind of holds your source of Truth the
871s client sends the data that your server
873s doesn't agree with you can decline the
874s actions and or kick or ban the user
876s depending on how severe the action is
878s they're taking that you're not trying to
883s allow and obviously we've seen you know
886s a lot of games that we've played in
887s where as um as as the latency becomes uh
893s worse for certain players or as the load
896s on the server becomes higher um to
899s process
900s that replication can can show some weird
903s effects in games right wait a minute
906s what's that person doing that doesn't
907s look right yep that's when you would
910s that's typically when you would get like
911s rubber banding like if your serers
913s really slow down because you would only
914s get an update every you know X period of
917s time yeah so as you can kind of see in
919s the the video we showed you can kind of
922s see like player one is going into the
923s water this is from the perspective of
925s the server so as player one enters the
927s server it'll be like okay player one is
929s swimming so now player two and three
931s since they're visually seeing player one
933s swimming they're going to get a state
935s for player one saying they are swimming
938s and then that can kind of perform like
939s the necessary actions of like seeing
941s them play the swim animation or whatnot
943s and and and and like I guess to add on
945s to what Hunter is saying this is the
947s this is pretty much the entire problem
949s like you're doing this all the time for
951s everything in the game right that's
953s replicated all the time every movement
956s every ability every thing you click on
958s interact with
960s T yeah all the Goblins that are running
962s around they're playing animations uh the
965s players moving stuff in their inventory
967s the the trees leaves moving subtly yeah
972s the the trees falling being cut down
974s yeah so not everything needs to be
977s replicated across the server right some
980s things can be client only like the ways
982s the leaves might moves specifically but
984s when the tree gets cut down that needs
986s to get replicated across yeah well and
989s and and you you can do interesting
991s things to try and like reduce this
994s problem like like if we're talking about
995s the wind for example you could just
997s replicate you know the wind direction
999s and speed yes and then the client can
1001s just apply that to all the Tres instead
1003s of you having you know a specific
1005s replicated you know wind and speed for
1008s each tree for example right and that's
1010s part of the Finesse of replication right
1012s trying to identifing the problem exactly
1015s that's right yeah a big part of our job
1017s especially with working with gameplay is
1018s trying to figure out how make that as
1020s efficient and optimized as possible so
1021s that our game and servers can perform to
1023s the expectations we need speaking of
1026s servers the server meshing what
1028s everyone's been excited to talk about
1030s right so now we do replication so now
1033s originally we just talked about how
1035s replicating a one server works so now
1038s the first part of intrepid net is what
1039s we like to call what we basically
1042s normally know as as server
1044s meshing
1046s um in Unreal Engine um your game
1048s normally works under a single process
1050s server scenario that means all your
1052s clients are connected to a singular
1054s server that runs the entire game now for
1056s an MMO RPG like ashes of creation um
1060s this can't work for us like for a few
1061s reasons Zach talked about in the
1063s previous slide there's just with the
1065s mass amount of entities and actors we
1067s have simulating and processing the
1069s replication of all that for everyone
1071s every actor in one server that's pretty
1075s insane um our solution first and
1077s foremost here is basically we had to
1080s modify Unreal Engine so that it is
1082s capable of having many server processes
1086s that are kind of meshed together or
1087s quilted
1088s together that allows us to use all these
1092s servers together to make up your whole
1093s entire game world or your game
1096s room um this allows us to split up that
1099s simulation work between many of the
1100s server processes and each server process
1103s has its own authoritative state of the
1105s world that it kind of owns and controls
1109s um this task it was not a trivial task
1112s and it required us ripping apart deep
1114s layers of unal engine and their
1116s networking stack in order to really
1118s modify it to fit the architecture that
1120s we have really
1122s envisioned um outside of this too
1125s there's like a lot of proprietary
1126s programs as well that we made that kind
1128s of run in the background to help it
1129s orchestrate all this functionality as
1132s well and by using under engines API um
1137s and doing all this we were also able to
1139s add a lot of minimal overhead for our
1141s Engineers When developing for our this
1143s solution and it allows us General
1144s control over like what how things
1147s replicate and where they replicate that
1149s we'll get into in the next few
1152s slides
1154s um so right now you see we have like the
1157s static Dynamic world grid okay that's
1159s great but now you're probably thinking
1161s okay now that we stitch many service
1163s together what is my experience like that
1165s in regards to traversing the world will
1168s I be able to see into the other servers
1170s from mine or will I randomly just start
1171s seeing actors pop in and out of
1173s existence right in front of
1175s me that kind of takes us to this
1177s slide um so our answer to that of course
1180s is we want to make this experience as
1182s seamless as possible for our players if
1184s you cannot tell what's going on behind
1186s the scenes and everything that we've we
1188s are making then that means we did our
1191s job right if you can tell then it
1192s probably means something's wrong and we
1194s need to look at what we're doing here
1196s and how we can improve and optimize it
1197s further
1200s um we've developed what we call here
1203s inter server replication for this case
1205s which means that instead of traditional
1208s replication as you might have as we
1210s explained was like server to clients
1213s this is basically representing between
1215s servers now or replication between
1217s servers and we do this replication to
1221s what we call a proxy
1223s actor um as you can see in the video
1226s here we can see there was like a
1227s transparent White uh wall going across
1230s the screen that represents our in-game
1232s debug tools with our border regions
1234s being
1235s drawn you can see that vexent is on one
1238s server and then Kevin is on another one
1241s as he's on the other side of that border
1243s so what vexent is seeing right now is
1244s actually not Kevin but a proxy of Kevin
1247s is what we' have deemed it is it really
1249s Kevin yeah who knows if only could
1252s actually duplicate Kevin Hunter expand a
1255s little bit on you know a lot of lot of
1258s people who obviously aren't familiar
1260s with game development they might not
1262s understand the the terminology of
1263s debugging tools right so to talk a
1265s little bit about what a debug tool is
1268s right so when making a networking
1270s solution as complicated as this we need
1273s a lot of information about the state of
1275s the world what's happening where things
1276s are going on so we've developed a lot of
1280s like debugging tools that can be used
1282s inside the game and outside of the game
1284s which these tools kind of help us
1286s collect that information see where all
1288s the important things are happening see
1290s where a failure might be going on so in
1293s this particular instance we have a debug
1295s tool that's in game that allows our
1297s developers to kind of draw these server
1299s borders so that you can kind of see
1301s while you're playing like okay there's a
1303s border here um and then if QA is testing
1306s something they can be like okay we
1308s transferred across the server and then
1309s this broke so that means something's
1311s probably wrong with the server transfer
1312s so that can kind of just help you like
1314s narrow down more of the problem space
1316s and give you that necessary information
1318s of like what was going on at the
1320s time um because without that if if we
1323s have it turned off you pretty much
1326s hopefully should not be able to tell the
1327s difference between the proxy we talked
1329s about or the actual
1331s actor um you can kind of think of the
1334s proxies here as a puppet any
1338s actual um action that was done by the
1341s actual actor on its own like Authority
1344s server which is the server that owns it
1347s it's then replicated to this proxy on
1349s your server which then gets replicated
1351s to yourself so that you can kind of see
1354s the representation of Kevin as if he was
1357s on the same server as
1359s you um we do it in a form of like a we
1362s call it lightweight because it's an
1364s up-to-date copy of the original actor
1366s but we can specifically our developers
1368s can specifically control like what does
1370s and doesn't get sent to this actor so
1372s that we're more optimal in like what
1375s we're sending how we're sending it and
1377s when we're sending it
1381s that was a
1382s [Laughter]
1384s mouthful I I think I think the other
1386s thing you know to think about Hunter
1388s Zach Anton is that um not a lot of
1391s players obviously followed development
1394s of the games that they're excited about
1396s right they they wait until you know the
1398s game is ready to play and and marketing
1400s begins and you know they can join in and
1402s and purchase or whatever obviously ashes
1405s has followed a little bit of a different
1406s path where we're giving people a bit of
1408s insight into you know what it actually
1410s takes to make these types of games and
1413s we generally tend not to delve too
1416s deeply into the highly technical
1418s components of our development process
1421s because that's where a little bit of a
1423s cliff starts to form of player interest
1426s even you know viewership interest um
1429s because uh it's nice to have the magic
1432s work in the background and you don't
1434s necessarily care or want to know how or
1436s why it's working just that it feels good
1439s that the you know it's responsive and
1441s that you know it accomplishes from a
1443s gameplay Loop perspective what you're
1445s expecting it to but you know because of
1448s the level of complexity because of the
1450s amount of effort because of what we're
1452s trying to create is a is a Next
1456s Generation level of complexity of gaming
1460s um we feel it's important to provide you
1464s guys an opportunity to get some insight
1467s into what we're doing that's unique into
1469s what we're doing that is going to
1472s service a long-term mmor RPG at the
1476s scale that we need Ashes to be and that
1480s requires some really smart people in the
1482s room who are creating proprietary
1485s technology that allows us to achieve
1488s that in a way that presents players with
1490s a game at the Quality they expect and so
1493s I you know I think this is a lot of a
1494s lot of fun to kind of you know have you
1496s guys come and and explain this front
1499s facing it can be a mouthful it can be a
1501s headful to kind of digest but at the
1504s same time well yeah I know we're no
1506s we're near done but at the same time it
1508s gives you now an opportunity to to
1510s really see how the sausage is made um
1514s and I think that's cool I can talk about
1516s this all
1517s day people don't talk about this there
1520s are so many games that are that are
1521s doing things like this not exactly but
1525s like you know there are other games that
1526s have to solve these problems and you
1528s know other games don't talk about this
1530s you know this is this is a first
1533s absolutely
1535s right yeah so as you see in the video um
1539s you can see into the other server but
1542s now you're probably thinking okay what
1544s if I want to interact with this proxy
1546s how how do I do that across a server
1549s boundary well um because the proxy is
1552s like a puppet controlled by the higher
1555s authority we basically just now need
1557s events that we can from the proxy up to
1560s its Authority and then we tell the
1562s authority that we want to modify data or
1564s execute some action on the authority
1566s server so that it can then replicate
1568s down to the proxy and then back to you
1571s um we have what we call a cross server
1574s event or an inter server event that's
1576s done through what's normally known as an
1579s RPC or remote procedure call which
1581s allows pretty much just for
1583s this um our goal with entrepren net is
1587s to make all the routing and actions that
1589s we do as fast as possible so that we can
1591s keep the latency and interactions
1593s between servers a pretty seamless
1595s experience and you hopefully shouldn't
1597s even be able to tell the difference
1599s between like I said a proxy or an
1600s authoritative actor yeah and I guess to
1604s add here this is so important because
1606s you know these these server borders that
1608s we're talking about they're invisible
1610s you you you won't know where they are um
1613s so the I think the big important thing
1617s here is like we need to have we need
1618s make it so PVP can happen on these
1620s borders without people noticing um and
1623s that's that's that's that's really
1625s that's the real Crux of it here that's
1626s why it has to be so
1628s fast right and as you can see in like
1631s the video here we have some fighting
1633s across the border um you can see while
1635s we're fighting kind of over the boundary
1637s combat and everything seems pretty
1638s seamless and smooth as you would expect
1640s it without any noticeable issues um this
1643s is kind of showcasing like what we can
1645s do with just those events and that
1647s replication we talked about out we can
1649s have these like indepth battles or
1653s interactions between the servers with
1655s hardly any downside yep
1658s um so but what we're trying to achieve
1661s here is the seamless experience that you
1664s hopefully can't tell where that border
1665s crossing is and and the other important
1668s thing about this particular you know
1670s this cross server uh information and in
1673s the Border itself which I think you guys
1675s are getting into a little bit later is
1677s that these borders are not static they
1679s can change they are Dynamic you're
1680s you're spoiling it Ste I'm I'm sorry I'm
1684s sorry I'm so sorry this is my first time
1686s seeing the
1688s slide okay go no no go I'm sorry I think
1691s you were talking about seamless
1692s transition so we
1694s can yeah so if you look in or go back
1697s one more Maggie I'm
1698s sorry um so if you kind of seen in the
1700s video there as well um now that we can
1704s kind of do interactions to the proxies
1705s across the border you may have noticed
1707s in the video I did charge myself across
1709s the border and then I basically attacked
1712s what we call the proxy of
1714s Kevin however during that charge we did
1718s basically a server promotion that took
1719s place behind the scenes that you
1721s probably didn't
1723s notice um so if we go to the next
1726s slide we can kind of see now we're
1729s introducing the concept of server
1730s transfers this is kind of where Intrepid
1733s net gets very
1736s complex um this allows us to not
1739s share and split the statically defined
1741s server areas but we can also transfer
1743s actors to new server areas that are
1745s moving across the world or happen to
1747s have a new server come up on top of
1750s it into a different server and then we
1753s do and then this allows us to do
1755s basically that Global load balancing
1757s across the
1758s room and then so basically when a player
1761s or an actor is within range of this
1763s border our system on the back end kind
1766s of begins a negotiation and the creation
1769s of a proxy so that it can replicate it
1771s to the nearby
1773s server um this doesn't only happen on
1776s like a one by one scenario that you
1778s might see here even if it's a 2 by two
1780s and you're in like the corner of four
1782s different
1783s servers your proxy will then get
1785s replicated be replicated to on three
1787s different servers from the one Authority
1789s so that way no matter where you are
1791s whichever server you're in around that
1792s area you'll still be able to see this
1794s actor on a different server
1801s um if that actor then fully crosses the
1805s Border going outside of the authority
1807s region of the server it's currently in
1809s um this the the two servers then kind of
1812s start negotiating what we call a
1813s promotion or a transfer this is when now
1816s server a is passing over that Authority
1819s we previously defined to server
1822s B and then promotions they're like a
1824s whole different Beast um in order to do
1826s a lot of like the promotions there's a
1828s couple things we have to do such as like
1830s ensuring the first you have to ensure
1831s the actor is like fully made with all of
1833s its components and
1835s children um we have to ensure they're
1837s all up to date with the entire state
1839s that was deemed necessary for the new
1840s copy to have because we don't want to
1842s lose any data during this entire
1846s transfer um and Hunter you people hear
1849s the term like you know children and the
1851s state and whatnot elaborate a little bit
1853s about you know what what that means at a
1855s high level right so
1858s uh when we have all these entities such
1860s as like an actor actors can then have on
1863s them what we call components which is
1866s basically building blocks that you can
1868s then add to your actor that kind of make
1870s up the actor what it does how it's
1872s supposed to act things like that um so
1875s for example you could have like an actor
1878s and then that actor has an interactable
1879s component so that means you can interact
1881s with this actor or it could have an
1883s inventory component so that means okay
1886s this actor has something of like a bank
1888s or um it it stores goods and to
1892s contextualize that let's say you know
1894s you have your Mount out you're not on
1896s your Mount right another player could
1898s walk up to your Mount and they would be
1899s prompted with an interact that says to
1901s Mount Your Mount as you're moving across
1903s the boundary that actor has that
1906s component that is the interactable
1908s component that needs to go with it right
1911s because if you transfer this actor or
1913s you're proxying it it still you still
1915s need to be able to do that interaction
1916s across the boundary so we proxy that
1919s interaction component and just like we
1921s can do crossover events on the actor
1923s itself we can also do them at the
1925s component level so that way it can
1926s communicate with its authoritative
1929s one um because we kind of treat the
1931s proxies as like a readon concept so the
1934s proxies are just visually kind of
1936s representating what the authority
1938s represents so any data modifications or
1941s anything that has to happen has to
1942s happen at the authoritative level so
1944s that way then the proxy can get updated
1946s appropriately across any Sur
1948s um that's kind of how we have like that
1950s Central truth of the actor is whatever
1952s server it's Authority on and there's
1954s only ever one copy of that I just want
1956s you to know this conversation so far has
1959s probably induced about 30 to 50 people
1963s now wanting to take computer science
1965s degrees that would be awesome yeah no
1968s it's it's pretty cool stuff
1971s um it definitely took a lot of time and
1974s a lot of effort to try to figure out how
1975s we make these things possible and after
1977s all this work this is we're excited to
1979s be able to show it um although watching
1982s Kevin run back and forth is making me a
1984s little
1985s nauseous yeah so in the scenarios 2 this
1989s also works where you may not be proxying
1991s to that nearby server because say you
1993s teleport across the
1994s map um this kind of means that when
1997s promoting there it kind of does the same
1999s exact route except it may just need to
2001s make your actor instead of making a
2003s proxy of it so it all falls kind of
2005s under the same generalized um system
2009s no um and then when we normally do have
2013s that proxy actor on the other server
2015s that means when you're promoting we can
2017s reuse your existing two actors that are
2019s from server a and server B your
2020s Authority and the proxy and we can
2023s convert that proxy to the authority and
2025s then demote The Authority one to a proxy
2028s so this allows us more to have that more
2030s seamless experience when you're swapping
2032s servers so that way it's a lot faster
2035s you don't need to send a lot more data
2037s to copy over to be created and then
2039s there's not really any visual hitching
2041s at all when this happens because we're
2044s not spawning or destroying a new one on
2045s the fly during that
2048s promotion and this helps also with
2050s client
2051s performance it it can um not going too
2055s too into detail but yeah talk actors
2059s aren't actors really aren't cheap so
2061s replicating an actor can be expensive
2064s and every new actor you kind of open and
2066s close to the client which which is
2068s basically like a create or a destroy
2071s that just adds this additional overhead
2073s that you we didn't want so we can kind
2075s of negate it by just reusing the
2077s existing one we already
2079s have and then doing that swap so that
2082s way you don't even notice anything
2084s happened because something Stephen no no
2087s he got it on the spawn and Destroy nice
2089s okay yeah yeah like like I said our goal
2092s is to make it as seamless as possible
2094s and one of the main pillars when we were
2096s making this uh ire system
2100s was we try not doing anything on the
2103s client if all of our actions are kind of
2105s done on the server in a fast and
2108s efficient way the client shouldn't
2109s notice anything happening at all and
2111s that's kind of where this comes in
2114s because we don't want the client to
2115s notice so every time we're doing
2117s something we're like okay how can we do
2119s this without affecting the client's
2120s experience because we don't want them to
2122s have a bad time
2124s yeah okay so um let's talk a little bit
2130s about how all this stuff is
2133s working uh because yeah we are
2135s communicating between servers we're
2137s sending events between servers we are
2139s transferring between servers but how is
2143s that information flowing how do the
2145s clients end up seeing that stuff and
2149s typically uh the way this is done is uh
2153s you create a centralized
2155s replicator and all the different ser
2158s servers from all across the realm send
2161s all the relevant information about all
2163s the different entities to that
2165s replicator so let's say one server has
2168s um a bunny and that bunny moved around
2172s it sends that information to the
2173s replicator and now the server has a
2175s goblin that took damage it will send
2177s that information to the
2179s replicator and then all the players will
2182s connect to that replicator and the
2184s replicator will pass pass through all
2186s those different entities
2189s decide which player should see which
2191s entities and send that information to
2194s the specific
2196s player however this creates a few issues
2200s and you might actually be able to see it
2201s from that graph there that we've just
2203s created a
2205s bottleneck uh as this singular process
2207s now may need to keep track of hundreds
2210s of thousands of
2211s entities and
2214s send potentially hundreds of millions of
2217s updates to all those clients because all
2220s those bunnies moving and the goblins and
2223s the players and the trees falling all
2225s that information has to be passed
2227s through the replicator and then sent to
2229s each individual client and I guess
2232s something to add here too Antoine is
2233s like this is the that would just be
2235s moving the problem we're trying to like
2238s like if we had a single server this this
2240s would this would be the same problem
2242s where all all the replication is
2244s happening on one place and we're trying
2245s to we're trying to not do that Yes
2248s actually we tested a solution like this
2251s uh during alha one and this
2255s bottleneck it kind of affects how many
2257s players you can have in the realm as a
2260s total like you can spawn all the servers
2262s you want but one at some point you hit a
2264s limit on the replicator itself and it
2266s doesn't matter how many more servers you
2268s spawn you are limited by that cap on the
2271s replicator uh during alpha 1 the number
2274s we got was like I think about a th000
2277s players
2278s where after hitting that point you can
2281s no longer really scale anymore and it's
2284s important to note that because of this
2288s more I guess for lack of a better term
2291s archaic and monolithic approach which
2293s which many of us have likely experienced
2295s in older games there are certain
2298s scenarios in those games that are
2300s multiplayer where you start to see that
2302s server is having trouble and it's
2304s because it's hitting these types of
2306s Choke points
2308s yes and in fact when this replicator is
2311s overwhelmed the entire realm will lag it
2313s doesn't matter if you're alone in one
2315s server in a remote island you will lag
2318s because there is a big battle on the
2319s other side of the realm that's I
2321s remember in so many so many games where
2323s I would be out hunting and peaceful
2326s nobody's nearby and I'm noticing
2328s everything is starting to churn and I'm
2330s like what is going on and then I'll look
2332s into Global chat and I'll see oh [ __ ]
2335s they're having a huge fight somewhere
2337s across the realm in a totally different
2339s part of the world and I am suffering as
2341s a result of it
2345s indeed I guess it's a good bridge to our
2348s to what we're doing yeah it is a pretty
2349s good Bridge cuz basically we yeah we
2354s ases of creation is trying to aim for
2356s player counts that we found out that
2358s centralized replicator does not support
2361s so we had to come up with our own
2363s solution and uh this is from what we
2366s know not a very
2368s common solution possibly because it's
2371s more difficult to pull off uh what we
2375s basically did is we completely got rid
2377s of that centralized replicator and
2378s instead we had the servers themselves
2381s collaborate with the other servers
2383s around them and exchange information
2386s that way that and that makes each
2389s individual server have all the data they
2392s need to serve the clients that are
2394s connected to them uh in our scenario
2398s over here you can see we have four
2399s servers and two clients are connected to
2402s each one of those servers if we had a
2404s centralized replicator all those four
2407s servers would be connected to that one
2409s replicator and then all eight clients
2412s would be connected to that one
2413s replicator to so we've actually
2416s distributed the problem we've eliminated
2419s the bottleneck and we've also eliminated
2423s this one point of failure that existed
2426s beforehand where if that one replicator
2428s fails the entire realm goes
2431s down and we can actually yeah have a
2435s look at the uh next slide where we see
2439s uh how we are more fault
2443s tolerant uh than if we have a
2445s centralized
2446s replicator uh and we can actually yeah
2450s you can swap to the next slide there you
2452s go we have the uh the entire server grid
2455s over there and if one server goes down
2458s everyone else can uh keep playing it
2461s like normal we patch that server up we
2464s fix the issue we bring it back up and
2468s everyone else never experienced anything
2469s the players that were connected to that
2471s one server got booted and then can
2473s rejoin just fine uh if we had a
2475s centralized replicator if that
2477s replicator went down everyone gets
2479s booted
2481s so yeah so that's another uh another
2485s good uh benefit from our approach the
2488s the really big gain we achieved with
2490s this was the elimination of the
2493s bottleneck uh and it actually lets us
2496s scale we are
2498s truly distributed we can just spawn new
2501s servers to a certain point of course and
2504s this is this is not and by the way Anton
2506s like this is not just a strength
2508s obviously of the of the replication
2510s model of of the of the server grid
2512s itself you know being distributed in
2514s such a good way but when you have to
2516s bring such a mod lithic server down
2519s because of that because of let's say
2520s whatever error you can't determine that
2522s has a Cascade effect across you know
2525s everything you're having to to affect
2528s the database to affect the login servers
2530s to have the player cues going off again
2533s the ability for this to truly be
2535s distributed and for us to be fault
2538s tolerant is an entirely different player
2540s experience when it comes to these large
2543s massive MMO RPG uh type settings yeah I
2546s mean it's it's very typical you know
2548s like with you know an MMO like that
2552s something else is happening in the world
2553s somewhere someone's hacking someone's
2555s you know there's a really large maybe
2557s there's a really large raid or or Siege
2559s happening and the the server Falls over
2563s and then everyone gets kicked with our
2565s model that doesn't happen the the rest
2567s of the server is fine performance is
2569s still fine in the rest of the server
2570s people can keep playing if that server
2572s goes down we you know we we'll deal with
2574s it we'll spin it back up and it we'll
2576s just seamlessly rejoin red once it's
2578s once we're back in a good
2582s State and this like eliminating the
2585s replicator was um definitely not easy uh
2590s and it actually it caus us quite a few
2593s problems but let's investigate one of
2595s those problems and the solutions we came
2598s up to uh solve it
2602s so the because we did not want to create
2606s an
2609s really large connection network between
2612s all the different servers we essentially
2614s connect the servers to the
2617s nearby servers the neighboring or like
2620s some that are close by uh and that is
2624s fine in most situations over here we can
2626s see there's two players in um two
2629s neighboring servers those those servers
2631s communicate so if those two players are
2633s in a party even though they might be a
2636s little bit too far away to see each
2637s other
2638s they can still see each other on the map
2641s however in a more complex situation
2644s where that other player is maybe on the
2647s other side of their realm now those
2650s servers do no longer have a connection a
2653s direct connection to each other so we
2656s have created a problem how can player
2660s one see player two when player two is on
2662s the other side of the world when if we
2664s had a centralize replicator sure we have
2667s the data just there we can figure it out
2669s but we do not uh what do we do about
2672s that and uh to solve uh that uh issue we
2677s created a concept called
2681s microservices microservices
2684s are Imagine them kind of
2687s like servers on top of the servers like
2691s um like some database for example but
2694s it's active it's not just storage
2697s they're doing
2698s work uh we have a node service that is
2701s responsible for um leveling up and
2704s keeping track of all the nodes and the
2707s upgrades that have to be done to to
2709s nodes we have a population service that
2712s is doing the spawning across all the
2715s Realms and we have a Social Service
2718s which would be for example what
2722s um what would affect the party example
2725s that we just uh talked about
2728s this means that server one and server 8
2731s on the diagram can connect to that
2733s social service and exchange the
2735s locations of those two players that way
2738s player one and player eight can have
2741s exactly the information they need to be
2744s able to see each other on the
2746s map and we act we yep what was it
2750s Stephen oh nope that wasn't me oh that
2753s wasn't you I can't s my bad um so uh
2757s yeah and this actually gave us some
2759s extra benefits we were able to make some
2761s pretty cool tools with it because we
2764s wanted to be able to see to have like an
2767s overview of what happens to the entire
2769s realm we wanted to be able to keep track
2771s of all the players all the NPCs see all
2774s the different gathers being spawned see
2777s all the
2778s trees and that's kind of hard to do if
2782s we just connect to a single server and
2784s then see what's on that one server and
2785s then jump on another server and see
2787s what's there so we created a service
2790s that keeps track of all entities in the
2792s world and a tool that can allow us to
2796s view those
2797s entities and this is a pretty cool tool
2800s because we can see the players moving
2802s around we can see them uh participate in
2805s events we can see the NPCs dying we can
2807s see them collect gather bles so if
2810s there's any issue with the spawning on
2812s GA of Gables will detect it and
2815s hopefully fix it it's a I think the big
2817s thing to to say here is like you know we
2820s can we can see the entire world state
2822s which we yes generally can't do because
2825s we're so distributed and everything's so
2827s server Central but since we have these
2829s Services we can make a debug service
2831s like this and we can see the entire
2833s world yeah watching you yeah I also
2837s wanted to piggyback off of that Anton of
2839s your services um the services as well
2842s are also pretty much distributed like
2844s going for our model so that if one of
2847s the systems go down it's not taking down
2849s the entire all every game system so if
2851s we just if the chat service goes or the
2853s Social Service for example goes down you
2855s may just not be able to chat or form a
2857s party for a few minutes until it comes
2858s back up but you should still be able to
2860s interact with like your guild your node
2862s your mail Etc and we try making the
2865s services as well fault tolerant like we
2867s explained earlier um so that way they
2870s can come back up go down come back up on
2872s the Fly get hot fixed and so that we can
2875s still provide that seamless experience
2877s while the feature may not be working
2879s everything else should still be working
2880s in the meantime right and because
2882s they're they're componentized in this
2883s you know microservices approach right
2886s there is not an interconnectivity where
2888s you have one failure in a single service
2890s causes a cascading effect across all the
2893s services right exactly they're they're
2894s independent of one another that allows
2896s for a recovery time a recovery process
2899s to stand back up while the other
2901s services continue their operations
2903s exactly and then these debugging tools
2905s allow us to see where those problem
2906s areas are or what what situation or
2909s actor might be causing a problem right
2911s exactly we don't have all the tools
2913s slide for all the tools but we do have
2915s monitoring tools for all the different
2917s Services we know when they're lagging we
2919s know when they're down we know we can
2921s monitor pretty much everything it's
2922s pretty cool yes it's very
2925s cool cool now let's get let's get into
2927s the really fun stuff now next one is
2929s this one is the community has actually
2931s been talking about quite a bit yeah I
2933s saw a bunch I saw a bunch of videos
2935s people are people are have been talking
2937s a about this yeah so go into it Anton
2940s because this is this is your thing yeah
2942s oh God I have been I've been looking
2945s forward to like since I joined the
2947s company I've been wanting to implement
2949s this and uh this is um an answer to a
2955s problem and that problem that he trying
2958s to answer is sure you have all those
2961s different servers but usually player
2964s activity tends to condense to certain
2967s are areas let's say a large PVP battle
2969s is happening at some place um and when
2974s that happens that one server can be
2977s overwhelmed and you usually wouldn't
2980s want to just create tiny servers for
2984s everything because that can be
2985s incredibly expensive and
2988s wasteful but having those smaller
2991s servers would be really useful so what
2994s if you can do it dynamically what if you
2997s can detect when a server is overwhelmed
3000s and then split it this is what dynamic
3003s grading it is and it can be a little bit
3007s of a little bit tricky to implement uh
3010s the difficulty of it is that you just
3013s now split a server and you want the
3015s players to not realize they were casting
3018s abilities they were running around and a
3021s new server just spawned they were moved
3023s there and no one realized that is a
3026s difficulty there I want to add here is
3028s like these cases in which we're
3029s splitting a server they're complex
3032s because if we're splitting a server
3033s there's a lot going on so it's not like
3036s we can like there's just a couple people
3037s we can split them we can do like a a
3039s quick fade and it will be fine if we're
3042s splitting the server it's because it's
3044s it's
3045s underload and there's there's a lot of
3047s action going on so we want to keep that
3049s action going on and not interrupt it
3051s while we while we balance the load
3052s across multiple servers and and it's
3054s important to you know call back again to
3056s the very beginning of this conversation
3059s right how do you determine the types of
3061s architecture necessary you must first
3065s understand what is the mission statement
3067s of the game and Ashes is built with a
3070s lot of open world Dynamic and emergent
3074s player behaviors that are attempting to
3077s encourage High player densities at any
3081s point within the open World locations
3084s which makes this particular problem a
3087s requirement when determining
3088s architecture of how do we address this
3092s right and I'm not familiar with and you
3094s know Zach Anton Hunter I'm not sure if
3096s you guys are but I'm not familiar with
3098s any other techstack or networking
3100s approach that has a dynamic griding
3103s solution in order to better accommodate
3106s and facilitate that emergent player
3109s density Behavior yeah I mean we we we
3113s are not the first people to try this
3114s this is this is something that a lot of
3116s people have tried it's it is you know
3119s the the answer to to scalability if you
3121s if you can get it right and that is you
3123s know that is our our mission here is to
3125s get this right we want to be the people
3128s that solve this problem and because if
3131s we solve this the the the the level of
3134s battles and uh dynamicism I guess we'll
3137s be able to have in ashes is going to be
3140s unparalleled to any other game yeah and
3143s to make something clear we're not
3144s talking about the far future five years
3147s from now uh this is current currently in
3150s development and we are uh hoping at
3154s least to have it available by Alpha 2 um
3158s and the server mesing part of it is
3160s already in game yeah so the dyamic
3164s aspect of yeah so the static part yeah
3166s is already is already going it's been
3168s going for manys now yeah years years now
3171s um and you know the the all of our Dev
3174s test servers people are on them every
3175s day running it it's fine
3177s uh this is this is like you know if
3180s where where we're investing our Dev time
3182s right now is getting this perfect mhm
3185s get getting this ready for the entrance
3187s of Alpha 2 Y and hopefully ases of
3191s creation is going to be uh the first
3193s game that implements this technology and
3195s actually deploys it yes that would be
3197s really fun you got to work quickly guys
3200s if you want that moving
3204s moving very cool this is exciting
3210s it up to uh to Hunter yeah yeah I I
3213s guess now that we've you know all these
3216s slides before this we were talking about
3217s Distributing the problem separating the
3219s problem um into multiple servers so we
3223s don't have to deal with like you know
3224s thousands and thousands of players and
3227s on on the same server doing
3229s replication um but you know we still
3231s have like the core problem here right we
3235s can only split servers so far we can't
3236s you know make them like 10 met by 10 met
3238s we can't do that it's going to be too
3240s small so we need to we need to still
3243s support you know a high you get a server
3246s you get a server exactly exactly so we
3249s still need to solve the like the core
3251s problem of how many players can we get
3253s on a server have them replicate and
3255s simulate at the same
3258s time so uh I'll pass it over to 100
3261s he'll kind of go into like I guess the
3263s problems that we have when we're when
3265s we're with with with unal replication
3268s and how we've gone about speeding them
3270s up so we can uh support many many many
3274s uh hundreds of players on a
3276s server yeah um so kind of piggybacking
3279s off what Zach said like what do we do
3281s when a large amount of players are say
3283s following your favorite streamer into a
3286s a small area and then like we can't just
3289s keep dividing our servers into such tiny
3291s areas and
3293s that you're needing a single server
3296s that's still
3297s of a reasonable size to then be able to
3300s process the heavier load and replication
3303s that's happening in that exact area at
3304s that exact
3305s moment um so in order to do that we have
3309s to then figure out how we can increase
3312s the number of players and entities that
3314s a single server can support and this
3317s doing so will also help us increase our
3319s overall like concurrent user limits per
3321s realm um because as a rough napkin math
3323s of course you know you take the players
3325s per server multiply that by the number
3326s number of servers and there's your CCU
3328s right it's that easy yeah that's how it
3332s obviously um so I'm going to kind of go
3335s into now uh we're going to touch back up
3338s on kind of what we explained earlier
3339s like why is replication
3342s expensive like what's going on
3345s um so we kind of wanted to explain that
3350s like on this slide here for example you
3351s can see an example of a CPU performance
3355s profile that we took during a large
3357s scale bot test um you can see here that
3361s like the single frame we're staring at
3363s took 150 milliseconds for example now
3366s 150 milliseconds may not seem like much
3369s but when you want to for example you're
3370s talking about servers and tick
3372s rates um and if you're trying to Target
3375s say a 30 or 60 frames per second or
3377s ticks per second server you only get a
3380s budget of about like 33 milliseconds or
3382s 16 milliseconds per frame in order to
3385s keep that stable High
3387s tick time can you talk a little bit
3390s about what you're referring to when
3391s you're when you're talking about tick
3392s rates and
3394s ticks um
3397s so as your game server runs
3400s every it it runs and processes basically
3403s the entire world at like an interval of
3406s ticks or frames is what it's called the
3408s faster you can do all that processing
3410s the faster your gameplay and experience
3412s is going to be um if you're targeting
3416s say roughly 30 to 60 frames per second
3418s which is a pretty good um targets to
3421s have for like a game server of like an
3423s MMO that needs to be very fast paced and
3425s have like very good performance FPS is
3428s different than client FPS yes but but
3431s but they they 30 is not acceptable I
3435s just want to make they're vaguely
3436s coupled so you one one of the things you
3439s would notice like if server performance
3440s went down is you would notice rubber
3442s banding like when you see like people
3444s just jumping around the world or you
3446s would notice your latency goes up and
3448s you notice that by like you're you know
3449s you're trying to you know do PVP and you
3452s feel you know you you you cast Fireball
3455s and you feel the latency between the
3457s time you casted it and when it actually
3460s executed on the server so the the this
3462s this this time these like these tick
3464s times that we're talking about that's
3466s what it correlates to for the player
3468s right because normally when you do an
3469s action the client triggers something it
3471s tells the server hey do this and the
3473s server then replicates back to your
3475s client if these frames are taking a long
3477s time to process in between that means
3479s your server just has artificial latency
3481s and it's taking a lot longer to then
3483s process that incoming message to then
3485s return back to you um so like in this
3488s 150 millisecond scenario if our game was
3491s steadily running every frame at like 150
3493s milliseconds of time to process that
3496s would drop our server down to roughly
3498s like six ticks or six frames per second
3500s which then you will really heavily
3502s around that time start feeling like
3503s latency rubber banding uh Dr packets Etc
3508s and and then that's not acceptable for
3510s an action combat PVP MMO like it just
3513s doesn't work
3515s right so I played those games it
3519s definitely doesn't
3520s work we all have blamed lag before
3524s right so so so we know that this is
3526s unacceptable what do we have to do to
3528s optimize this right so now this kind of
3532s tells us replication and now it's like
3534s okay let's dive a bit deeper into why
3536s this replication is very
3539s expensive um one of the big excuse
3544s me one of the big um major factors
3547s replication can be slow is because first
3550s when you want to do your replication you
3552s have to gather and determine basically
3555s relevancy of every actor in the game to
3558s not only your connection but every other
3560s connection that's being processed in
3561s that
3562s frame so relevancy is used and needed to
3566s determine basically how far away an
3568s actor is from your player and then there
3570s might be some other like comment some
3572s other checks that they may need to do
3573s like is invisible or not but for
3576s Simplicity sake we're just going to base
3577s it off a distance for now so then it's
3581s basically comparing that distance to you
3584s and saying okay are you eligible for me
3587s to replicate to you because we obviously
3589s don't want things being considered to
3591s replicate to you if it's like across the
3593s world like you don't care what's
3594s happening on the far Northern end of the
3596s the continent while you're all the way
3597s down on the southern end like that's
3599s just that's just going to waste a lot of
3600s processing and
3602s bandwidth um so the relevancy checks
3607s like this gets way more expensive the
3609s more players and more actors you have
3611s that are replicated because then it's
3613s for every actor for every player am I
3616s within range of them yeah and when we're
3618s talking about an MMO where you know
3620s you're going to have thousands of
3621s players on the server thousands of NPCs
3624s uh tens of thousands of replicated
3626s actors you know you can't you just can't
3628s check all you can't do a distance check
3630s on all of those things in a reasonable
3632s amount of time right and so this is an
3635s optimization here for networking
3636s purposes but it also leads to then
3639s slowing down your CPU resources because
3642s again you have to do that like every
3644s tick or at every whatever interval you
3645s set to do these
3647s checks um as we if we looked at the
3650s previous example for example I believe
3652s the I don't think it shows in the
3654s profile but roughly a little bit over
3657s onethird of that total time and that
3659s profile was just doing the relevancy
3662s checks um a good thing about our model
3666s though because all of our servers are
3667s distributed they only have to do those
3669s relevancy checks for just everything
3672s within that then distributed Zone as of
3675s if we had that Central replicator it
3677s would that replicator would then have to
3678s still process every single player every
3681s single actor Etc in the world in some
3683s way shape or form to then determine if
3685s it should be relevant to you or or not
3687s so by splitting up and doing the
3690s distributed server load like we're doing
3692s this allows us then to kind of reduce
3694s that problem area but it's still a
3696s problem yeah it's still really big
3698s because it's still a really big problem
3700s now adding more servers to the solution
3701s that can make it smaller and smaller but
3704s eventually there's like we want to get
3707s it as fast as possible so it's like how
3709s how how do we make this better how do we
3711s make this faster to be clear that
3713s profile was using uh inter it had a
3717s server messing and it still was 40 35%
3722s uh uh relevancy checks so we need to
3725s reduce those relevancy checks uh how how
3728s do we do that how can we tell unreal not
3732s to do all those unnecessary
3735s checks well unreal in recent years
3739s introduced something called the
3741s replication graph and that is something
3744s that gives the
3746s developers
3748s the ability to write a custom solution
3752s into how to handle all those uh
3755s relevance checks and how to determine
3757s which player should see which actors
3759s when it comes to
3761s replication uh and this can actually
3766s lead to some really great performance
3768s games we'll look at it in a bit uh but
3771s let's start and uh to and see how we use
3775s this replication graph to solve
3777s specifically this relevancy
3779s check on the map over there we see uh we
3784s you still cannot see too far away but
3787s there are a couple actors that are
3789s really close to the player and that
3791s Goblin over there and a loot box and
3795s there is also um a bear a little bit on
3799s a medium distance away and a bunny
3801s really far away now imagine that keep
3804s stretching for 500 met more and there's
3806s 200 more bunnies and without the
3809s replication graph we would have to check
3812s every single one of those bunnies and
3814s say is this within my distance is this
3816s within my distance and that's that's a
3819s problem that's expensive to do so what
3822s we're going to use the replication graph
3825s um to do is we are going to create this
3828s grid where we eliminate all the cells
3832s that are far further away from us and
3834s only check the nearby
3837s cells that allows us to only check these
3839s three actors that are around us skipping
3842s potentially hundreds of others actors
3844s that are further
3845s away and uh this can lead to a pretty
3849s big performance Improvement we noticed
3852s and this is like highly varying
3854s depending of the scenario of course but
3857s on an average scenario we noticed about
3859s a 1.5 times performance Improvement like
3862s a 35%
3864s reduction uh in uh our replication
3867s cost this is this is pretty
3871s good uh but it's still not fast enough
3874s and if we analyze the problem a little
3876s bit we'll see
3878s that we are we we're talking and
3881s thinking about like the NPCs and the
3884s bear and the
3885s Goblin but we are also comparing our
3888s distance with all the trees with all the
3890s bushes that you can pick up with
3892s potentially some buildings that are
3895s destructible
3897s why are we doing that can we do
3900s something to improve this and the answer
3902s is yes we can uh what we can do is we
3906s can monitor and see when these um ores
3912s or these trees are
3915s changing and when they're not changing
3918s we can move them out over our grid so
3921s we're not even considering them at
3923s all uh we might have 2,000 trees in the
3927s area well we're not even going to look
3929s at them because none of those trees just
3931s got cut down if someone is cutting them
3933s down then yes we're going to move them
3936s temporarily into the special grid and
3938s then once they're done cutting out and
3940s everyone sees that they were cut down we
3942s put them back to the dormy bucket it's
3945s essentially looking imagine like the
3947s actor is going to slip for it a little
3949s bit and this this gives us a really
3954s really large Improvement we noticed in
3956s our test about five times Improvement in
3959s replication times so so
3963s far uh if we also include the previous
3966s optimizations we've improved our
3967s replication performance by seven
3970s times and yet that's still not fast
3973s enough it's still not fast enough one of
3976s mainly because of our worst case
3980s scenario and that worst case scenario is
3983s people keep packing in that one small
3985s area
3986s there's an event happening there's some
3988s point they need to capture 200 people
3990s run in the same area yes she can ignore
3993s the um relevancy to some Far Away tree
3997s that's great that's a game that's uh
4000s more than welcome but if 200 players can
4003s see each other that means that 200 time
4007s 200 you're sending 40,000 updates per
4010s tick yeah and I guess one thing to add
4014s there Anon this is typically when a game
4017s would introduce a new layer or um a new
4020s Shard this is like typically that case
4023s um but as we talked about earlier one of
4025s our goals is we don't want we don't we
4027s don't want to do layering we don't want
4028s to do sharting we want everyone who's in
4030s the same area to be able to see each
4032s other right and you could you could then
4035s use like Dynamic greeting to help in the
4037s scenario but again we can only make the
4038s servers so small and dynamic before we
4041s kind of run out of that room so we still
4043s have to improve the scenario that's kind
4046s of happening around you indeed so we
4049s need to figure out a way to further
4053s improve this worst case scenario and
4056s this is where we kind of hit an engine
4060s limit um and this engine limit is that
4065s the replication graph of unreal
4069s is the replication tick in general not
4071s just the replication graph is single
4073s threaded and that's because the engine
4077s in general is single-threaded now what
4080s does that mean what does single threaded
4081s mean it means that all the processing
4085s happens in one core of the
4089s CPU uh and uh we can move on to the next
4093s Slide by the way but
4098s the that means that even though the
4101s server might have four cores um all the
4106s other three cores are going to stand
4108s there and wait for the first thread to
4110s do all the
4112s processing uh this is done normally in
4115s unreal because it's it was not designed
4118s to be
4120s multi-threaded uh it was it is a very
4122s single-threaded engine and replication
4125s is probably the most single threaded
4128s system out of all the systems in
4131s unreal uh so making it multi- threaded
4136s is incredibly difficult and uh at least
4141s I am not aware of it being done before I
4144s tried to do some research around it at
4146s the time and uh everyone was saying
4149s basically that it's
4151s impossible but uh we did it anyway uh we
4155s persevered through and uh we
4157s multi-threaded the unreal replication
4160s Tech and uh well looking at the
4164s results they did not disappoint uh we
4168s actually gained and there's a graph
4171s showing it on the next slide but we
4174s gained a
4176s 2.5 times Improvement 2.4 times to be
4180s more exact in a replication text huge
4185s yeah it was a really really big
4187s Improvement in uh a replication and it's
4190s probably necessary to achieve the player
4193s counts that we're targeting for I don't
4195s I don't thinking would be possible
4196s without doing
4198s this uh and the the really important
4202s thing to note about this optimization
4204s and why it's like such a big deal
4206s compared to like the five times
4209s Improvement we saw before from dorcy is
4212s that this
4215s Improvement helps equally even on the
4218s worst case
4219s scenario the other ones in the worst
4221s case scenario they're helpful but
4224s they're not that helpful this
4227s is incredibly important even on the
4229s worst case
4230s scenario and if we add it on all the
4234s other improvements we've done so far
4236s we've actually achieved a total
4238s Improvement of total reduction or
4241s replication take by
4243s 94% so we are only at 6% of the original
4246s cost for uh replication uh this was
4251s actually a huge win for uh our team um
4255s but but um
4257s it there are still there's still many
4260s more things that need to be
4262s done uh as of creation is
4265s um an incredibly ambitious MMO in terms
4268s of scale uh and uh we need to do a lot
4275s of work in order to heit our goals and
4278s you know this is this is obviously again
4280s going back to the start of this whole
4282s conversation what is ashes of creation
4284s trying to be
4286s and what does it then require of the
4288s text act we are using in order to become
4291s that and when we said in 2017 that we
4295s are looking to bring massive back to
4297s massively multiplayer having an engine
4300s with the graphical Fidelity the front
4302s end that unreal provides is
4306s so is so important to the immersive
4311s nature to the beauty of the world to you
4314s know the quality of the setting but then
4317s coupled with a proprietary backend that
4321s can facilitate the same player
4324s experience that a massively multiplayer
4327s World wants to provide that's the dream
4331s that ises is trying to be
4336s exactly phenomenal work here guys this
4338s is this is huge yeah I I'd like to like
4342s take a moment I guess to shout out the
4344s team the team has done an incredible job
4346s you know we
4348s are we we we are we are pushing MMO Tech
4351s to where it's never gone uh and we
4353s continue to we we intend to keep doing
4355s this moving forward Hunter Alex Kevin
4358s Andrew Anton Adam ish Doug you know they
4362s they've all done amazing work uh you
4364s know to get us to this point where we're
4366s at today and I'm sure they'll keep doing
4367s it moving forward um if you're
4371s interested in working on problems like
4373s this you know yes send us an application
4375s you know we're always looking for people
4377s who want to work on this who have
4379s experience working on Tech like this
4381s please send this an application we we're
4384s we're always interested and we know that
4385s like this this this showcase this this
4388s Tech talk this Ted Talk this this GDC
4391s talk you know what whatever whatever we
4393s want to apply to this uh to this indepth
4396s technical discussion is going to appeal
4399s not just to to people who are aspiring
4402s obviously uh within this field to learn
4404s it to do it to you know this is their
4407s career going forward but also to those
4409s of you who have been within this field
4412s work and who yearn to work on the types
4415s of problems that we're discussing here
4417s in this showcase um we have a phenomenal
4420s culture at Intrepid we have a phenomenal
4422s project in our hands that is you know no
4426s shortage of problems that need to be
4428s solved um and one thing that our
4431s community has done a phenomenal job with
4433s is is helping to evangelize the project
4436s and the company and we're here to say
4438s that we have apps open we have positions
4442s that are available and if you want to be
4444s a part of this team um or if you want to
4446s help others find this team so that they
4449s can apply please share this link share
4452s this uh video with others uh go on you
4455s know social media and let people know
4457s we're working on some pretty Next
4459s Generation Tech that is going to change
4461s the face of massively multiplayer when
4464s it comes to a genre so you know very
4466s excited and proud of what your team has
4469s accomplished you guys I know are working
4471s every single day and doing a phenomenal
4473s job um congratulations just very well
4477s done thank you thank you and it's great
4480s to experience it in game to and I know
4482s that you know our PI folks have already
4484s experienced that in our node Wars test
4486s as well so maybe they didn't really even
4488s realize it um yeah they might not have
4491s noticed um but there were definitely
4493s server workers working hard and that um
4496s and we were capturing data so we super
4498s duper appreciate it and as we head into
4500s Alpha 2 we're only going to get more
4501s data and be able to continue to push on
4504s and as uh the folks here have stated um
4507s moving towards more of that Dynamic
4509s State yeah um it's been in progress It's
4513s going to be in progress still for many
4514s years as we continually find new ways to
4516s advance it and make it better um we're
4520s committed to providing kind of like that
4521s quality we want for our players and as a
4524s lot of us have played MMO in the past we
4526s just like networking was the big stopper
4530s for a lot of things and we're trying to
4532s do something different and trying to
4535s make that a better experience for not
4537s only our game but many more games to
4538s come if other companies were able to
4541s like wanting to adapt to a new tech
4543s style or try to figure out their their
4545s networking
4546s issues yeah and of course all of us here
4548s are huge MMO fans and I know those of
4551s you who are watching are as well
4553s obviously if you're sitting through this
4555s um but of course the the plan for us is
4558s to continue to create an amazing game
4560s and push forward and continue to change
4562s things even after post launch we'll
4564s continue to try to drive things forward
4566s so thank you for joining us and thank
4568s you to Hunter and Zach and Anton for
4570s your time and putting together an
4572s amazing deck a little extra shout out to
4574s Mike for putting together all the graphs
4576s gra Mike did such a good
4579s job for helping capture some of the
4581s little footage pieces to make sure that
4583s they looked great we super appreciate it
4585s any final words step inter wrap yeah
4587s absolutely just you know out in the
4588s community let us know you know these are
4590s always the more risky takes that we
4592s provide insights too in our monthly
4594s updates because they are so Tech heavy
4597s and you know we know that this has an
4600s appeal to perhaps a smaller audience but
4603s give us your feedback give us your
4604s thoughts what did you think of this
4605s presentation how do you enjoy you know
4608s learning a little bit more indepth under
4610s the hood conversation about what it
4613s takes to make these types of games to
4615s make you know ashes of creation that
4618s that dream of all of ours um how does it
4621s appeal to you what are your questions
4623s what are your thoughts um leave us
4625s comments in the YouTube channel in the
4627s uh in on Twitter and different social
4629s medias and Discord and on our forums
4631s obviously the community team continues
4633s to do a bang-up job of collecting your
4635s insights and bringing it back to the
4636s team um but we enjoy you know sharing
4639s this and and giving giving our perhaps
4642s um perhaps more sheltered uh de Vel
4645s opers who are who are in the behind the
4649s scenes making all the magic and
4650s everything work an opportunity to come
4652s out and talk about how they are
4654s literally the magic that makes the
4656s wheels turn for this bus um and it it's
4659s it's a great opportunity I think for
4661s them to come out see a little daylight
4663s and let and bask in the sun of what is
4665s the glory that our community looks to
4666s them and says thank you for providing an
4669s opportunity to enjoy a game at this
4670s scale because it would not be possible
4673s obviously without your guys's hard work
4675s um and so with that I think we'll see
4678s you guys back on stream we'll see you on
4681s the flip side
4688s [Music]