5 months ago - Play New World - Direct link

Transcript (by Youtube)


0s like the uh the skeleton archers when
2s they like Duck and you always like shoot
3s right over them and stuff well yeah and
5s it it totally plays into our combat
7s right like we have head shots uh they do
9s bonus damage there's certain abilities
10s where if you shoot in the legs you get a
12s slow like so we take advantage of all
14s this sort of uh cool stuff yeah we could
16s totally optimize by just becoming tap
18s Target right
19s no let's go to the next
23s [Applause]
26s question welcome to forge an ET turna
29s where we talk about all things new world
31s today we have a very special episode we
33s are joined by four senior Engineers on
36s the team this is the brains of the
38s operation this is what makes stuff
39s actually happen uh so uh we're going to
43s talk a little bit about what sort of
44s technical problems we've seen and like
46s what we're doing to work on them and a
48s little bit on the future of our Tech uh
50s but before I do that let me just quickly
51s introduce all the Geniuses here so we
53s have uh Scott geyser uh we have Julian
57s we have Kevin and Ed uh are one
60s wonderful senior
61s Engineers uh so let's start off with a
64s question something I see a lot like hey
66s I play a lot of opr or ecrs and you know
70s my my performance isn't really good I'm
72s I'm feeling laggy I get stutters uh
74s let's first talk about like what is that
76s actually from a technical perspective
78s like what are people seeing what are the
79s causes of that type of thing uh yeah so
83s um there's a there's quite a few things
85s that can happen when you're experiencing
88s um latency or or you know sometimes you
91s might see rubber banding um but the
95s basics of it is we send inputs up to the
97s server and the server needs to verify
100s those inputs uh with physics and then
102s come back down to your
104s machine and when you start to experience
107s that um that step of the pipeline on the
111s server um could take longer and so
115s typically like if you have a debug
117s console open you can see your rtt and
120s and once that rtt starts extending beond
123s 200 milliseconds um you may start
126s experiencing issues like that and that
129s can start to happen um specifically in
132s combat because um we're spawning a ton
135s of projectiles for one um typically in
137s combat of around 100 people uh we see
141s around 6,000 projectiles spells being
144s cast per minute and that can be an
146s expensive operation for the server so um
150s some of that time that was originally
152s dedicated to kind of making sure your
154s player updates is going to start to
155s distribute um amongst those
158s spells um another thing that can happen
160s is um we have sort of like um whenever
164s you're in like a close density with a
166s lot of people um we're trying to send
170s data uh to the um players that you see
173s and they're trying to send uh data to
175s you so you can see their positional
176s updates and a bunch of other things like
179s animation and stuff like that
181s and that problem um becomes basically
184s quadratic um once uh um once the density
189s of the number of players starts to
190s become too much
193s and so that's like another place where
195s the CPU on the server which is you know
197s sort of trying to update your inputs can
200s go and when those two things start to
203s happen um we start to see um basically
206s your rtt go up because we don't have
208s enough um CPU to handle the
211s density and so like the the creation of
214s all those new entities like every
215s projectile and stuff is like an
217s expensive operation itself right so the
219s fact that projectiles are shortlived and
220s created and then destroyed and then a
222s new one's created like that's expensive
224s and then like you were talking about
225s it's not just the players but also those
227s projectiles and every other Ai and opr
229s wherever you happen to be maybe it's an
231s influence tower all those things are
232s also updating down to every single
234s player yeah uh in in that area too and
237s wouldn't it be fair to say like I think
239s we do a lot more in terms of combat and
242s gameplay on the server than a lot of
243s games right so that stresses the systems
245s a little bit yeah I mean like every
247s basically Arrow every bullet now is an
250s actual um physical thing in the world
252s that actually moves through the space
254s and as it's moving through the space
256s it's trying to figure out what it's
258s colliding with and um you know we don't
261s use raycast um for like uh hit scan
263s anymore um so there's just a lot of um
267s basically things moving through space in
269s our game um
271s and yeah it can be computationally
273s expensive the character is also
274s represented by sorry the character is
277s also represented by physics uh bounds
279s all over their body uh so that we can
281s distinguish a head shot or an arm shot
283s or a leg shot and you can also duck or
286s go prone under these things so as you're
288s moving through the world it's moving
290s these representation with you um and
292s that ends up being more expensive than
294s than a usual MMO how they handle it like
297s the uh the skeleton archers when they
298s like Duck and you always like shoot
299s right over
301s well yeah and it it totally plays into
302s our combat right like we have head shots
304s they do bonus damage there's certain
306s abilities where if you shoot in the legs
307s you get a slow like so we take advantage
309s of all this sort of uh cool stuff yeah
312s we could totally optimize by just
313s becoming tap Target right no no let's go
316s to the next question what is scale
319s testing scale testing yeah what is it so
321s we try to we try to emulate what's going
323s to happen in the game using butts so
325s basically AIS uh and they do very simple
329s task uh that we Define that we write uh
332s for example in combat uh the most common
334s one is like having them fight and run
337s around and see uh you know look at some
340s dashboards that we create as well uh and
343s see you know uh the metrics see they
344s align with what we expect um the
347s beginning of that process is like we
350s always write write like technical
352s documents and we explain how the feature
354s is going to work how much it's going to
355s cost the various different costs we have
357s and then we verify using the scale test
360s um running scale test and so we use bus
362s for that uh different like different
364s ways all the features are kind of
366s different so require different uh
368s testing um but in the end that's that's
371s what we're doing we're trying to emulate
372s what's going to happen in the game um as
374s best as we can because you know it's an
376s open world game so everything is kind of
379s happening sometimes differently than we
380s expect so we we we do our best in that
384s that are you yeah I think like the hard
387s part in creating those scale tests is
390s making it um behave in a way that's like
393s what players are doing like players are
396s always surprising us with what they're
398s doing and like we'll think we've got
401s like a good scale test and then we'll
402s see in live like performance starting to
405s to drop and we go in and we see what
407s they're doing and it's just like we
409s didn't think they were going to do that
410s and they did it and so now we have to re
412s you know write our scale test to kind of
415s help mimic that worst case yeah a good
418s example is like which one it's hard for
420s me to just evaluate is like World buses
423s cuz you don't know how many players are
424s going to show up so you try to aim for
426s like a certain number of
427s players but that could go over that
430s number very easily so you always tend to
433s aim for the worst case scenario but even
435s that is kind of hard to gauge right can
437s be different than what you're going to
438s get in the game in the end so it's very
440s hard and um so like when we're talking
443s about like that the targeted number of
444s players for like a particular activity
446s right um we're we we that's part partly
450s the difficulty of the the world boss and
452s all that that's sort of like the floor
454s of how many players we expect like are
455s necessary to to complete that type of
457s content but then like in the open world
459s the ceiling is basically you know until
462s the server explodes right so we have to
465s have mitigations for that so that you
467s know other people playing in a
469s completely different part of the world
470s but like sort of on that same Hub like
473s don't have a bad experience because
475s people are like crammed together
476s shoulder-to-shoulder fighting a world
477s boss like in a different place and so
480s you know for the influence races for
482s example like that feature had to take
485s some of the tech we used for the uh our
487s Fort capture uh areas because we want
490s PVP especially to be responsive um and
494s so that sort of has a lower kind of
497s ceiling of how many players we want to
499s be in that area that we feel like still
501s gives us like a good experience even
503s even like at maximum capacity um and
507s then for PVP also because we're you know
510s a faction vaction vaction game um you
513s know it wouldn't be fair for one faction
514s just you know take up all the capacity
516s so for towers and and Forts and stuff
519s you know that feature broke it down even
520s further by having limits for each
523s faction and how many of those flag
524s people can be in there um but in other
528s just open parts of the world we have
530s higher higher capacity and different
532s mechanisms of either you know if
534s something gets really dense maybe we'll
535s teleport some of the people away um or
538s you'll see like a physical barrier you
539s just won't be able to get into an area
541s like a long time ago when we had Wars
543s that weren't instanced right you'd be
545s running towards or like kind of like
546s near the fort and you just run into that
548s wall um and you wouldn't know what's
550s going on oh it must be a war going on
552s over there um but now like you know we
554s take that out of the open world and and
555s so that gives that area back to the
558s players yeah one of the things that we
560s did improve for the expansion is um
563s influence races um like Scott was saying
566s we we did cap that at around I think 100
568s 150 um but other areas in the game we
571s were able to increase the number of
573s players we could get into areas uh the
575s density of basically the number of
578s players um and so what that means is
581s like you know you you hopefully won't be
583s seeing that that kind of wall anymore um
586s hopefully we did a good enough job where
587s that doesn't happen anymore
590s but cool well you've talked about a
592s couple things we've done already we've
593s instanced War which has sort of helped
595s uh we've done some things to increase
596s density what else have we done already
598s like sort of in the last year what else
600s is is on top of mine to continue to
602s improve in
605s performance well so one of the most
607s expensive parts of of the game is the
609s player simulation uh because the player
612s has so many things that they can do so
614s many abilities they can cast spells or
617s shoot weapons or interact with different
619s things in the environment like Gathering
620s things so um in order to help improve on
625s that we're we're working on lots of
626s optimizations on on the player and
629s trying to improve its memory footprint
630s as well as its uh the amount of time it
633s takes for it to to do all these
634s abilities uh and so this is a
636s significant amount of work that we've
637s been working that we've been doing this
639s this past year and uh it's it's getting
642s close to being done we're we're working
644s on bugs with it now and uh it's it's
647s making a a it's showing up some pretty
649s good
650s results yeah and now that we're at the
652s point sort of that like we feel we're
655s we've got you know we were in that phase
656s where we're cramming features in and
657s trying to get up to you know parody and
659s or you know with other games in the the
661s industry and things like that and now I
663s feel like we're we're taking a step back
665s and trying to really optimize things and
667s and get things to run smoother and
668s quicker um and more performant just to
671s both to increase the experience for
673s players but also it gives us some wiggle
675s room then to add more stuff you know in
677s the future yeah I mean I think that's
679s important for people to realize right
680s like an engineer can only do X things at
683s once right and so we've got to make
684s decisions between should we make a
686s feature should we fix bugs should we
688s optimize uh and those are all decisions
690s we've had to make and I think like you
691s said a lot of the focus has been on
693s features and sort of getting up to a
695s point where we're happy and now we have
696s more time to start addressing bugs and
701s optimizations uh anything else on
703s performance you guys want to talk about
705s anything I
707s missed maybe we talk about the world
709s itself yeah okay I'm not I'm not a
712s specialist
713s though but I know I specialist on
715s everything I know we're trying to uh
717s title World specialist
720s I'm doomed here um we we also look at uh
724s uh the cost of rendering the world
727s overall uh and we know sometimes we in
730s some areas uh we can be over budget so
733s we look at reducing you know like
735s geometry and reducing drold and stuff
737s like that especially when on top of that
739s you spawn a world bus that's going to
742s bring uh the bus itself all the AIS plus
745s a bunch of players plus all sometimes it
748s spawns also like itions or you know like
750s props or stuff like that so you add more
752s geometry in there and so if you were
754s already over budget you're even more
756s over Bridget so the goal is TR to figure
758s out how we can make all the slices uh
761s cheaper um sometimes just as simple as
765s removing stuff you know uh to just
767s reduce overall the complexity of the
769s scene that you have to render um and
771s that's an ongoing
773s uh uh you know initiative that's going
776s to I think it's going to take time to
778s just go past the point where we are on
781s budget everywhere even if you spawn
783s something uh sometimes also like events
785s like winter events we we spawn entire
787s Village uh so it's a lot you know um or
791s The Summer Event is the same uh so yeah
793s that's that's also one initiative that
795s we're working on takes a lot of
797s time and as we mentioned before that
799s spawning stuff is expensive because we
800s have to replicate all the the
802s information for it down to the client
804s and then reproduce it as well as we can
807s um and and that's an area that we we a
809s lot on for the player as well to improve
811s the player spawning um which will also
813s affect AI as well um since AI does
816s sometimes there's delays on on how
818s quickly they
820s spawn yeah I was just going to say
823s that um you
826s know what Julian was saying um like a
829s lot of the time it's trying to figure
830s out where in the world um our
833s performance is is going bad and we've
835s done a lot lately to um include a lot
838s more metrics
839s um into our um basically Telemetry
843s reporting um so now we can see um like
846s areas in the game that are over budget
848s with um draw calls or materials and on
852s top of that like we're doing a lot more
854s on the server as well where we can see
856s um where things are becoming less
858s performant and we can really drill into
860s areas and and see why and what in our
863s pipeline is sort of taking the most time
865s and I think you know finding the issues
869s is is sort of like half the battle and
871s then like Distributing it out to people
874s to fix is the other half and um I think
877s we've made a lot of progress in the last
878s year of like creating better systems to
881s find the
883s issues awesome uh one other thing I
886s wanted to touch on so we talked a little
887s bit about performance another technical
889s thing people sometimes complain about
891s are bugs I think you know for instance
893s with the season 3 release we had a
895s couple issues with the season pass and
897s Quest what if anything are we doing to
899s sort of get better at sort of that and
901s how do how does things like that
903s happen I guess I can take the quest not
905s the quest
907s actually uh so for the season pass we
909s had we what happened not going too much
912s into details but a we did like some
915s refactoring cut refactoring and um that
919s that actually broke the way we were
921s saving all the data um uh and so at some
925s point we had issues where our world
927s would not save progression anymore
929s um and it was actually to a point where
932s we had to recover from there so first
933s you have to fix the bug you have to find
935s what what the bug is took us some time
938s then you have to fix it when you have
939s the fix out you can go from into like
941s fixing mode into recovery mode which
943s like how do we get give players back
946s what they lost in this case like SE
947s season
948s XP um so at that point you really like
951s on Telemetry to kind of figure out
953s like how much season they last each of
956s the player right so we can give it back
958s in some case we also have players who
959s bought levels uh they paid for it didn't
961s get them on time or didn't get them at
963s all um so we always work on like trying
966s to extract all the data from our tretry
969s so we can um run what we call make Goods
972s where we give back to players what they
973s lost so it could be like as I said like
976s what you what you paid for or it could
978s be sayon XP so you you get back your XP
981s uh and you're back into where you should
982s be uh but also we had issues where
984s people could not claim anymore um so we
987s had to fix those issues as as well uh
989s and all of these are like kind of new
991s issues that we've never experienced so
993s it's kind of brand new situation for us
995s we've never been put into so we figure
997s out as we go right so and as fast as we
999s can because we we know players are very
1001s unhappy about
1002s it go for it Scott you you you had to
1005s fix like your share your good number of
1007s issues so go for it um yeah and so that
1009s sort of like speaks to especially within
1011s MMO how interconnected all these
1013s different systems are like a bug in the
1015s persistent system right and then also
1018s issues in interacting with like our
1020s entitlement service and all those things
1022s kind of combine only when there's like
1024s high load or you know like tons of
1027s players and it it kind of shows how
1030s tough it can be sometimes to recreate
1032s those exact situations before it goes
1034s out to uh real players um and uh on
1039s another note like what you were talking
1041s about with the make goods and all that
1042s that's something that we've been
1042s investing in you know since launch and
1044s slowly increasing our capacity to do
1046s that and do it more quickly and do it
1048s more accurately I remember at launch
1050s right we're like okay something's
1051s happening but we're not really sure how
1053s or why we just know that this thing is
1055s happening and we didn't have like ways
1057s to query our Telemetry and and see like
1060s okay EXA like this is what's happening
1061s and then this is the order of events and
1063s this is how we reproduce the issue or
1064s these are the players that are you know
1067s that were affected by this loss or are
1069s performing this exploit or something
1070s like that um and so you know like we're
1073s we just have a much better sense now of
1076s exactly what's going on in the game
1077s across all of our worlds
1080s and what explain what a red button is
1082s that's another new tool we've Have and
1084s Have unfortunately had to use every now
1085s and then the the red button is the word
1087s we use for uh like something's wrong
1090s turn it off shut it down um and yeah per
1094s feature and so you know we learned from
1096s our lessons like from launch and and
1098s moving forward uh and so we've as a
1101s standard practice you know we're much
1102s more granular in what we can shut off
1104s like before it was like you know shut
1106s the whole economy down right and now we
1108s can say okay shut off player-to-player
1110s trades or shut off you know certain
1112s items or certain abilities or you know
1114s we have we have a lot more control there
1116s so that if something happens hopefully
1117s we're not affecting everyone's player
1119s appearence yeah actually on this issue
1121s like the season pass we turned off
1123s buying the premium and buying levels
1125s because people would you know buy them
1127s and not receive them so uh when we
1129s figure out an issue like that we tend to
1131s you know uh turn it off as fast as we
1133s can so we can actually assist the the
1135s problem and not making a bigger problem
1138s right so it's always about mitigations
1140s and going back and figure out what
1141s happened and going from there to fixing
1143s the issue and then once the issue is
1145s fixed like okay give back to players
1147s what they lost and at that point we're
1149s done one more question I had is
1151s sometimes when I'm like in the middle of
1152s PVP I'll start an attack or an ability
1154s and like it'll either go backwards or my
1157s position will jump a little bit what
1158s what's going on there from a technical
1160s standpoint uh so it'll depend on what
1162s you see so if if you see the character
1165s their animation reset then you might
1166s have experienced a roll back there um if
1169s it's just their position moving then
1170s it's probably just the server correcting
1172s your position um because we have to
1174s balance uh how much we do on the client
1177s with how much we U validate on the
1180s server um and so in order to be to to
1182s have Fair combat for all the players um
1185s the server really needs to check on if
1187s you if you can attack or if you actually
1189s hit someone or if someone hit you while
1191s you were attacking um and then it it
1193s also takes time for the server to tell
1195s you these things um for you to send up
1197s your input to the server and then the
1199s server to correct it and send it back
1200s down to you uh so sometimes if there's a
1203s bit of a delay and the server is really
1204s busy there's lots of combat going on uh
1207s sometimes that uh that correction can be
1209s uh delayed a little bit so uh the longer
1211s it's delayed the more movement you'll
1213s see being corrected um the more
1216s animation the difference in your
1217s animation there might be um and so like
1220s one of the worst situations is if you're
1222s you're swinging to attack someone else
1225s um but then something kind of out off
1227s screen hit you um but that happened on
1229s the server before you actually attacked
1231s and so you'll see yourself go from the
1233s middle of an attack back to a
1235s reaction um and so it can be pretty
1237s jarring uh and we're there's actually
1240s part of the player optimizations we're
1242s working on are hopefully going to smooth
1243s that a little bit um but that uh that's
1246s kind of the like the real world physics
1248s we can't we can't uh there's nothing to
1250s get around the time it takes for
1252s communications to go to the server and
1254s back so it's just something that happens
1256s in massively multiplayer games
1259s and is that so when people say a desync
1262s that means that their animation changes
1264s generally then yeah so it's the server
1266s did something different um from what
1268s your local client is doing because your
1270s local client it doesn't have all the
1271s information from the other what the
1273s other players are doing um and uh so it
1276s might it might take a little bit of time
1277s for your your client to react so the
1279s server might just come down and say no
1281s you were you're supposed to be in this
1283s animation and you're supposed to be at
1284s this position and so then the client
1286s will move you there and and roll your
1288s back roll your animation back to that
1291s state awesome hey well thanks to
1294s everyone for joining us today I I love
1297s diving into this stuff I don't really
1298s understand all it so I trust these
1302s guys uh and uh I do have a community
1305s question for the day we'd love to hear
1307s sort of what are the technical issues
1309s you're experiencing the game let us know
1311s uh put it in the comments and and these
1313s guys will fix it as soon as they can SC
1316s fix
1316s it and uh as always thanks for joining
1319s us please like And subscribe and see you
1321s in a
1330s [Music]
1331s [Applause]
1336s [Music]
1339s turnament