12 months
ago -
Play New World
-
Direct link
Transcript (by Youtube)
0s | like the uh the skeleton archers when |
---|---|
2s | they like Duck and you always like shoot |
3s | right over them and stuff well yeah and |
5s | it it totally plays into our combat |
7s | right like we have head shots uh they do |
9s | bonus damage there's certain abilities |
10s | where if you shoot in the legs you get a |
12s | slow like so we take advantage of all |
14s | this sort of uh cool stuff yeah we could |
16s | totally optimize by just becoming tap |
18s | Target right |
19s | no let's go to the next |
23s | [Applause] |
26s | question welcome to forge an ET turna |
29s | where we talk about all things new world |
31s | today we have a very special episode we |
33s | are joined by four senior Engineers on |
36s | the team this is the brains of the |
38s | operation this is what makes stuff |
39s | actually happen uh so uh we're going to |
43s | talk a little bit about what sort of |
44s | technical problems we've seen and like |
46s | what we're doing to work on them and a |
48s | little bit on the future of our Tech uh |
50s | but before I do that let me just quickly |
51s | introduce all the Geniuses here so we |
53s | have uh Scott geyser uh we have Julian |
57s | we have Kevin and Ed uh are one |
60s | wonderful senior |
61s | Engineers uh so let's start off with a |
64s | question something I see a lot like hey |
66s | I play a lot of opr or ecrs and you know |
70s | my my performance isn't really good I'm |
72s | I'm feeling laggy I get stutters uh |
74s | let's first talk about like what is that |
76s | actually from a technical perspective |
78s | like what are people seeing what are the |
79s | causes of that type of thing uh yeah so |
83s | um there's a there's quite a few things |
85s | that can happen when you're experiencing |
88s | um latency or or you know sometimes you |
91s | might see rubber banding um but the |
95s | basics of it is we send inputs up to the |
97s | server and the server needs to verify |
100s | those inputs uh with physics and then |
102s | come back down to your |
104s | machine and when you start to experience |
107s | that um that step of the pipeline on the |
111s | server um could take longer and so |
115s | typically like if you have a debug |
117s | console open you can see your rtt and |
120s | and once that rtt starts extending beond |
123s | 200 milliseconds um you may start |
126s | experiencing issues like that and that |
129s | can start to happen um specifically in |
132s | combat because um we're spawning a ton |
135s | of projectiles for one um typically in |
137s | combat of around 100 people uh we see |
141s | around 6,000 projectiles spells being |
144s | cast per minute and that can be an |
146s | expensive operation for the server so um |
150s | some of that time that was originally |
152s | dedicated to kind of making sure your |
154s | player updates is going to start to |
155s | distribute um amongst those |
158s | spells um another thing that can happen |
160s | is um we have sort of like um whenever |
164s | you're in like a close density with a |
166s | lot of people um we're trying to send |
170s | data uh to the um players that you see |
173s | and they're trying to send uh data to |
175s | you so you can see their positional |
176s | updates and a bunch of other things like |
179s | animation and stuff like that |
181s | and that problem um becomes basically |
184s | quadratic um once uh um once the density |
189s | of the number of players starts to |
190s | become too much |
193s | and so that's like another place where |
195s | the CPU on the server which is you know |
197s | sort of trying to update your inputs can |
200s | go and when those two things start to |
203s | happen um we start to see um basically |
206s | your rtt go up because we don't have |
208s | enough um CPU to handle the |
211s | density and so like the the creation of |
214s | all those new entities like every |
215s | projectile and stuff is like an |
217s | expensive operation itself right so the |
219s | fact that projectiles are shortlived and |
220s | created and then destroyed and then a |
222s | new one's created like that's expensive |
224s | and then like you were talking about |
225s | it's not just the players but also those |
227s | projectiles and every other Ai and opr |
229s | wherever you happen to be maybe it's an |
231s | influence tower all those things are |
232s | also updating down to every single |
234s | player yeah uh in in that area too and |
237s | wouldn't it be fair to say like I think |
239s | we do a lot more in terms of combat and |
242s | gameplay on the server than a lot of |
243s | games right so that stresses the systems |
245s | a little bit yeah I mean like every |
247s | basically Arrow every bullet now is an |
250s | actual um physical thing in the world |
252s | that actually moves through the space |
254s | and as it's moving through the space |
256s | it's trying to figure out what it's |
258s | colliding with and um you know we don't |
261s | use raycast um for like uh hit scan |
263s | anymore um so there's just a lot of um |
267s | basically things moving through space in |
269s | our game um |
271s | and yeah it can be computationally |
273s | expensive the character is also |
274s | represented by sorry the character is |
277s | also represented by physics uh bounds |
279s | all over their body uh so that we can |
281s | distinguish a head shot or an arm shot |
283s | or a leg shot and you can also duck or |
286s | go prone under these things so as you're |
288s | moving through the world it's moving |
290s | these representation with you um and |
292s | that ends up being more expensive than |
294s | than a usual MMO how they handle it like |
297s | the uh the skeleton archers when they |
298s | like Duck and you always like shoot |
299s | right over |
301s | well yeah and it it totally plays into |
302s | our combat right like we have head shots |
304s | they do bonus damage there's certain |
306s | abilities where if you shoot in the legs |
307s | you get a slow like so we take advantage |
309s | of all this sort of uh cool stuff yeah |
312s | we could totally optimize by just |
313s | becoming tap Target right no no let's go |
316s | to the next question what is scale |
319s | testing scale testing yeah what is it so |
321s | we try to we try to emulate what's going |
323s | to happen in the game using butts so |
325s | basically AIS uh and they do very simple |
329s | task uh that we Define that we write uh |
332s | for example in combat uh the most common |
334s | one is like having them fight and run |
337s | around and see uh you know look at some |
340s | dashboards that we create as well uh and |
343s | see you know uh the metrics see they |
344s | align with what we expect um the |
347s | beginning of that process is like we |
350s | always write write like technical |
352s | documents and we explain how the feature |
354s | is going to work how much it's going to |
355s | cost the various different costs we have |
357s | and then we verify using the scale test |
360s | um running scale test and so we use bus |
362s | for that uh different like different |
364s | ways all the features are kind of |
366s | different so require different uh |
368s | testing um but in the end that's that's |
371s | what we're doing we're trying to emulate |
372s | what's going to happen in the game um as |
374s | best as we can because you know it's an |
376s | open world game so everything is kind of |
379s | happening sometimes differently than we |
380s | expect so we we we do our best in that |
384s | that are you yeah I think like the hard |
387s | part in creating those scale tests is |
390s | making it um behave in a way that's like |
393s | what players are doing like players are |
396s | always surprising us with what they're |
398s | doing and like we'll think we've got |
401s | like a good scale test and then we'll |
402s | see in live like performance starting to |
405s | to drop and we go in and we see what |
407s | they're doing and it's just like we |
409s | didn't think they were going to do that |
410s | and they did it and so now we have to re |
412s | you know write our scale test to kind of |
415s | help mimic that worst case yeah a good |
418s | example is like which one it's hard for |
420s | me to just evaluate is like World buses |
423s | cuz you don't know how many players are |
424s | going to show up so you try to aim for |
426s | like a certain number of |
427s | players but that could go over that |
430s | number very easily so you always tend to |
433s | aim for the worst case scenario but even |
435s | that is kind of hard to gauge right can |
437s | be different than what you're going to |
438s | get in the game in the end so it's very |
440s | hard and um so like when we're talking |
443s | about like that the targeted number of |
444s | players for like a particular activity |
446s | right um we're we we that's part partly |
450s | the difficulty of the the world boss and |
452s | all that that's sort of like the floor |
454s | of how many players we expect like are |
455s | necessary to to complete that type of |
457s | content but then like in the open world |
459s | the ceiling is basically you know until |
462s | the server explodes right so we have to |
465s | have mitigations for that so that you |
467s | know other people playing in a |
469s | completely different part of the world |
470s | but like sort of on that same Hub like |
473s | don't have a bad experience because |
475s | people are like crammed together |
476s | shoulder-to-shoulder fighting a world |
477s | boss like in a different place and so |
480s | you know for the influence races for |
482s | example like that feature had to take |
485s | some of the tech we used for the uh our |
487s | Fort capture uh areas because we want |
490s | PVP especially to be responsive um and |
494s | so that sort of has a lower kind of |
497s | ceiling of how many players we want to |
499s | be in that area that we feel like still |
501s | gives us like a good experience even |
503s | even like at maximum capacity um and |
507s | then for PVP also because we're you know |
510s | a faction vaction vaction game um you |
513s | know it wouldn't be fair for one faction |
514s | just you know take up all the capacity |
516s | so for towers and and Forts and stuff |
519s | you know that feature broke it down even |
520s | further by having limits for each |
523s | faction and how many of those flag |
524s | people can be in there um but in other |
528s | just open parts of the world we have |
530s | higher higher capacity and different |
532s | mechanisms of either you know if |
534s | something gets really dense maybe we'll |
535s | teleport some of the people away um or |
538s | you'll see like a physical barrier you |
539s | just won't be able to get into an area |
541s | like a long time ago when we had Wars |
543s | that weren't instanced right you'd be |
545s | running towards or like kind of like |
546s | near the fort and you just run into that |
548s | wall um and you wouldn't know what's |
550s | going on oh it must be a war going on |
552s | over there um but now like you know we |
554s | take that out of the open world and and |
555s | so that gives that area back to the |
558s | players yeah one of the things that we |
560s | did improve for the expansion is um |
563s | influence races um like Scott was saying |
566s | we we did cap that at around I think 100 |
568s | 150 um but other areas in the game we |
571s | were able to increase the number of |
573s | players we could get into areas uh the |
575s | density of basically the number of |
578s | players um and so what that means is |
581s | like you know you you hopefully won't be |
583s | seeing that that kind of wall anymore um |
586s | hopefully we did a good enough job where |
587s | that doesn't happen anymore |
590s | but cool well you've talked about a |
592s | couple things we've done already we've |
593s | instanced War which has sort of helped |
595s | uh we've done some things to increase |
596s | density what else have we done already |
598s | like sort of in the last year what else |
600s | is is on top of mine to continue to |
602s | improve in |
605s | performance well so one of the most |
607s | expensive parts of of the game is the |
609s | player simulation uh because the player |
612s | has so many things that they can do so |
614s | many abilities they can cast spells or |
617s | shoot weapons or interact with different |
619s | things in the environment like Gathering |
620s | things so um in order to help improve on |
625s | that we're we're working on lots of |
626s | optimizations on on the player and |
629s | trying to improve its memory footprint |
630s | as well as its uh the amount of time it |
633s | takes for it to to do all these |
634s | abilities uh and so this is a |
636s | significant amount of work that we've |
637s | been working that we've been doing this |
639s | this past year and uh it's it's getting |
642s | close to being done we're we're working |
644s | on bugs with it now and uh it's it's |
647s | making a a it's showing up some pretty |
649s | good |
650s | results yeah and now that we're at the |
652s | point sort of that like we feel we're |
655s | we've got you know we were in that phase |
656s | where we're cramming features in and |
657s | trying to get up to you know parody and |
659s | or you know with other games in the the |
661s | industry and things like that and now I |
663s | feel like we're we're taking a step back |
665s | and trying to really optimize things and |
667s | and get things to run smoother and |
668s | quicker um and more performant just to |
671s | both to increase the experience for |
673s | players but also it gives us some wiggle |
675s | room then to add more stuff you know in |
677s | the future yeah I mean I think that's |
679s | important for people to realize right |
680s | like an engineer can only do X things at |
683s | once right and so we've got to make |
684s | decisions between should we make a |
686s | feature should we fix bugs should we |
688s | optimize uh and those are all decisions |
690s | we've had to make and I think like you |
691s | said a lot of the focus has been on |
693s | features and sort of getting up to a |
695s | point where we're happy and now we have |
696s | more time to start addressing bugs and |
701s | optimizations uh anything else on |
703s | performance you guys want to talk about |
705s | anything I |
707s | missed maybe we talk about the world |
709s | itself yeah okay I'm not I'm not a |
712s | specialist |
713s | though but I know I specialist on |
715s | everything I know we're trying to uh |
717s | title World specialist |
720s | I'm doomed here um we we also look at uh |
724s | uh the cost of rendering the world |
727s | overall uh and we know sometimes we in |
730s | some areas uh we can be over budget so |
733s | we look at reducing you know like |
735s | geometry and reducing drold and stuff |
737s | like that especially when on top of that |
739s | you spawn a world bus that's going to |
742s | bring uh the bus itself all the AIS plus |
745s | a bunch of players plus all sometimes it |
748s | spawns also like itions or you know like |
750s | props or stuff like that so you add more |
752s | geometry in there and so if you were |
754s | already over budget you're even more |
756s | over Bridget so the goal is TR to figure |
758s | out how we can make all the slices uh |
761s | cheaper um sometimes just as simple as |
765s | removing stuff you know uh to just |
767s | reduce overall the complexity of the |
769s | scene that you have to render um and |
771s | that's an ongoing |
773s | uh uh you know initiative that's going |
776s | to I think it's going to take time to |
778s | just go past the point where we are on |
781s | budget everywhere even if you spawn |
783s | something uh sometimes also like events |
785s | like winter events we we spawn entire |
787s | Village uh so it's a lot you know um or |
791s | The Summer Event is the same uh so yeah |
793s | that's that's also one initiative that |
795s | we're working on takes a lot of |
797s | time and as we mentioned before that |
799s | spawning stuff is expensive because we |
800s | have to replicate all the the |
802s | information for it down to the client |
804s | and then reproduce it as well as we can |
807s | um and and that's an area that we we a |
809s | lot on for the player as well to improve |
811s | the player spawning um which will also |
813s | affect AI as well um since AI does |
816s | sometimes there's delays on on how |
818s | quickly they |
820s | spawn yeah I was just going to say |
823s | that um you |
826s | know what Julian was saying um like a |
829s | lot of the time it's trying to figure |
830s | out where in the world um our |
833s | performance is is going bad and we've |
835s | done a lot lately to um include a lot |
838s | more metrics |
839s | um into our um basically Telemetry |
843s | reporting um so now we can see um like |
846s | areas in the game that are over budget |
848s | with um draw calls or materials and on |
852s | top of that like we're doing a lot more |
854s | on the server as well where we can see |
856s | um where things are becoming less |
858s | performant and we can really drill into |
860s | areas and and see why and what in our |
863s | pipeline is sort of taking the most time |
865s | and I think you know finding the issues |
869s | is is sort of like half the battle and |
871s | then like Distributing it out to people |
874s | to fix is the other half and um I think |
877s | we've made a lot of progress in the last |
878s | year of like creating better systems to |
881s | find the |
883s | issues awesome uh one other thing I |
886s | wanted to touch on so we talked a little |
887s | bit about performance another technical |
889s | thing people sometimes complain about |
891s | are bugs I think you know for instance |
893s | with the season 3 release we had a |
895s | couple issues with the season pass and |
897s | Quest what if anything are we doing to |
899s | sort of get better at sort of that and |
901s | how do how does things like that |
903s | happen I guess I can take the quest not |
905s | the quest |
907s | actually uh so for the season pass we |
909s | had we what happened not going too much |
912s | into details but a we did like some |
915s | refactoring cut refactoring and um that |
919s | that actually broke the way we were |
921s | saving all the data um uh and so at some |
925s | point we had issues where our world |
927s | would not save progression anymore |
929s | um and it was actually to a point where |
932s | we had to recover from there so first |
933s | you have to fix the bug you have to find |
935s | what what the bug is took us some time |
938s | then you have to fix it when you have |
939s | the fix out you can go from into like |
941s | fixing mode into recovery mode which |
943s | like how do we get give players back |
946s | what they lost in this case like SE |
947s | season |
948s | XP um so at that point you really like |
951s | on Telemetry to kind of figure out |
953s | like how much season they last each of |
956s | the player right so we can give it back |
958s | in some case we also have players who |
959s | bought levels uh they paid for it didn't |
961s | get them on time or didn't get them at |
963s | all um so we always work on like trying |
966s | to extract all the data from our tretry |
969s | so we can um run what we call make Goods |
972s | where we give back to players what they |
973s | lost so it could be like as I said like |
976s | what you what you paid for or it could |
978s | be sayon XP so you you get back your XP |
981s | uh and you're back into where you should |
982s | be uh but also we had issues where |
984s | people could not claim anymore um so we |
987s | had to fix those issues as as well uh |
989s | and all of these are like kind of new |
991s | issues that we've never experienced so |
993s | it's kind of brand new situation for us |
995s | we've never been put into so we figure |
997s | out as we go right so and as fast as we |
999s | can because we we know players are very |
1001s | unhappy about |
1002s | it go for it Scott you you you had to |
1005s | fix like your share your good number of |
1007s | issues so go for it um yeah and so that |
1009s | sort of like speaks to especially within |
1011s | MMO how interconnected all these |
1013s | different systems are like a bug in the |
1015s | persistent system right and then also |
1018s | issues in interacting with like our |
1020s | entitlement service and all those things |
1022s | kind of combine only when there's like |
1024s | high load or you know like tons of |
1027s | players and it it kind of shows how |
1030s | tough it can be sometimes to recreate |
1032s | those exact situations before it goes |
1034s | out to uh real players um and uh on |
1039s | another note like what you were talking |
1041s | about with the make goods and all that |
1042s | that's something that we've been |
1042s | investing in you know since launch and |
1044s | slowly increasing our capacity to do |
1046s | that and do it more quickly and do it |
1048s | more accurately I remember at launch |
1050s | right we're like okay something's |
1051s | happening but we're not really sure how |
1053s | or why we just know that this thing is |
1055s | happening and we didn't have like ways |
1057s | to query our Telemetry and and see like |
1060s | okay EXA like this is what's happening |
1061s | and then this is the order of events and |
1063s | this is how we reproduce the issue or |
1064s | these are the players that are you know |
1067s | that were affected by this loss or are |
1069s | performing this exploit or something |
1070s | like that um and so you know like we're |
1073s | we just have a much better sense now of |
1076s | exactly what's going on in the game |
1077s | across all of our worlds |
1080s | and what explain what a red button is |
1082s | that's another new tool we've Have and |
1084s | Have unfortunately had to use every now |
1085s | and then the the red button is the word |
1087s | we use for uh like something's wrong |
1090s | turn it off shut it down um and yeah per |
1094s | feature and so you know we learned from |
1096s | our lessons like from launch and and |
1098s | moving forward uh and so we've as a |
1101s | standard practice you know we're much |
1102s | more granular in what we can shut off |
1104s | like before it was like you know shut |
1106s | the whole economy down right and now we |
1108s | can say okay shut off player-to-player |
1110s | trades or shut off you know certain |
1112s | items or certain abilities or you know |
1114s | we have we have a lot more control there |
1116s | so that if something happens hopefully |
1117s | we're not affecting everyone's player |
1119s | appearence yeah actually on this issue |
1121s | like the season pass we turned off |
1123s | buying the premium and buying levels |
1125s | because people would you know buy them |
1127s | and not receive them so uh when we |
1129s | figure out an issue like that we tend to |
1131s | you know uh turn it off as fast as we |
1133s | can so we can actually assist the the |
1135s | problem and not making a bigger problem |
1138s | right so it's always about mitigations |
1140s | and going back and figure out what |
1141s | happened and going from there to fixing |
1143s | the issue and then once the issue is |
1145s | fixed like okay give back to players |
1147s | what they lost and at that point we're |
1149s | done one more question I had is |
1151s | sometimes when I'm like in the middle of |
1152s | PVP I'll start an attack or an ability |
1154s | and like it'll either go backwards or my |
1157s | position will jump a little bit what |
1158s | what's going on there from a technical |
1160s | standpoint uh so it'll depend on what |
1162s | you see so if if you see the character |
1165s | their animation reset then you might |
1166s | have experienced a roll back there um if |
1169s | it's just their position moving then |
1170s | it's probably just the server correcting |
1172s | your position um because we have to |
1174s | balance uh how much we do on the client |
1177s | with how much we U validate on the |
1180s | server um and so in order to be to to |
1182s | have Fair combat for all the players um |
1185s | the server really needs to check on if |
1187s | you if you can attack or if you actually |
1189s | hit someone or if someone hit you while |
1191s | you were attacking um and then it it |
1193s | also takes time for the server to tell |
1195s | you these things um for you to send up |
1197s | your input to the server and then the |
1199s | server to correct it and send it back |
1200s | down to you uh so sometimes if there's a |
1203s | bit of a delay and the server is really |
1204s | busy there's lots of combat going on uh |
1207s | sometimes that uh that correction can be |
1209s | uh delayed a little bit so uh the longer |
1211s | it's delayed the more movement you'll |
1213s | see being corrected um the more |
1216s | animation the difference in your |
1217s | animation there might be um and so like |
1220s | one of the worst situations is if you're |
1222s | you're swinging to attack someone else |
1225s | um but then something kind of out off |
1227s | screen hit you um but that happened on |
1229s | the server before you actually attacked |
1231s | and so you'll see yourself go from the |
1233s | middle of an attack back to a |
1235s | reaction um and so it can be pretty |
1237s | jarring uh and we're there's actually |
1240s | part of the player optimizations we're |
1242s | working on are hopefully going to smooth |
1243s | that a little bit um but that uh that's |
1246s | kind of the like the real world physics |
1248s | we can't we can't uh there's nothing to |
1250s | get around the time it takes for |
1252s | communications to go to the server and |
1254s | back so it's just something that happens |
1256s | in massively multiplayer games |
1259s | and is that so when people say a desync |
1262s | that means that their animation changes |
1264s | generally then yeah so it's the server |
1266s | did something different um from what |
1268s | your local client is doing because your |
1270s | local client it doesn't have all the |
1271s | information from the other what the |
1273s | other players are doing um and uh so it |
1276s | might it might take a little bit of time |
1277s | for your your client to react so the |
1279s | server might just come down and say no |
1281s | you were you're supposed to be in this |
1283s | animation and you're supposed to be at |
1284s | this position and so then the client |
1286s | will move you there and and roll your |
1288s | back roll your animation back to that |
1291s | state awesome hey well thanks to |
1294s | everyone for joining us today I I love |
1297s | diving into this stuff I don't really |
1298s | understand all it so I trust these |
1302s | guys uh and uh I do have a community |
1305s | question for the day we'd love to hear |
1307s | sort of what are the technical issues |
1309s | you're experiencing the game let us know |
1311s | uh put it in the comments and and these |
1313s | guys will fix it as soon as they can SC |
1316s | fix |
1316s | it and uh as always thanks for joining |
1319s | us please like And subscribe and see you |
1321s | in a |
1330s | [Music] |
1331s | [Applause] |
1336s | [Music] |
1339s | turnament |