over 3 years ago - /u/rkrigney - Direct link

UPDATE: The AMA has officially ended, so we're locking additional comments. If you have additional questions, please reach out to us on Twitter (mine's here, and Samy's is here) and of course we'll be watching the conversation in other threads over the next few days to see what other issues we should address.

Samy and I both really enjoyed ourselves, and y'all have been great. Thank you!

__

Hi Legends.

Earlier today we published (as long promised) a developer deep-dive into "what makes Apex online tick." It's a post that acknowledges common issues players have when playing Apex online, explains how we're working to improve things, and even addresses spicy topics like tick rate.

If you haven't already, you can read the blog here: https://www.ea.com/games/apex-legends/news/servers-netcode-developer-deep-dive

Now I've got the blog's author, Samy (aka @ricklesaucer here to answer followup questions from players. We're hoping we can keep today's thread focused on questions about servers, netcode, tickrate, etc, since that's the stuff Samy is best equipped to address. We know you probably have a ton of questions that aren't necessarily about netcode or servers, so we'll do another big seasonal AMA shortly after the Legacy update launches with the broader Apex team to address those sorts of things.

We'll be actively responding to questions in this thread from 3:30pm–4:30pm PT, after that we'll probably sign off for the day, but we'll keep an eye out for lingering questions that get highly upvoted and find a way to address those when we can.

External link →
over 3 years ago - /u/rkrigney - Direct link

Originally posted by casev7

The most common issue I see are no-fill of squad members. Is there any chance of having the game bail back to matchmaking if we end up without a full team?

Not servers/netcode related, but: The team's got some fixes for no-fill in the Legacy update—should help a lot. Unsure about that specific suggestion but will bump it with the team this week.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by TheHeuman

In the article you mention that getting to the root of packet loss/big variations in ping, is difficult and requires users to run "network traces". Is this something that is available for all players or just the pros and content creators?

If its available to everyone, how would one go about reporting that they have a netcode issue and possibly have respawn track down the cause of the problem?

It is for everybody. We have automated the packet loss collection from the server. We will be able to react faster on the data center side. The issue remains between you and your ISP. We cannot do much on that front sadly.

over 3 years ago - /u/rkrigney - Direct link

Originally posted by [deleted]

[deleted]

Yeah, there's a number of issues at stake here. DDOS attacks are a thing, but so are non-distributed denial-of-service attacks (which I guess would just be called "DOS?"). We've been working on closing a number of loopholes that allow people to do this, but closing loopholes isn't the only thing we're doing.

Beyond that, some of the things we're talking about are:

1) Improving tools for identifying those launching the attacks (several promising approaches here)

2) Working with some experienced partners in the space to develop more holistic solutions

3) Investigating more serious consequences beyond bans. We think legal action may be justifiable in some cases.

This stuff may take some time but it's definitely high priority for us.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by Schadenfreude11

Arena matches will probably be better connection-wise just by virtue of having fewer players. It's a lot easier to match 6 players of similar skill with a good connection to a common server than to do the same for 60.

This is completely correct! It's not that we'd never consider it, but we think there are higher-priority changes we could do that would help both BR players and Arenas players. I'm very excited to see people's feedback on the new mode!

over 3 years ago - /u/rkrigney - Direct link

Originally posted by XKM5543

Really appreciate the insight behind the curtain on some of these issues. I think that this level of transparency and communication does wonders to keep the community from feeling like they’re yelling into a void when facing frustrations.

I know it’s mentioned as a future topic in the post, but the current DDOS epidemic (especially on console) deserves the same level of attention, without waiting ages for a response. This has been going on for several seasons now and reached a peak in s8. On console, the game is borderline unplayable above diamond queues due to 50% of lobbies getting DDOS’d. Banning these individuals doesn’t seem like enough, this is not a problem most of the player-base sees impacting similar games, and some insight on why this is a unique problem with Apex would be appreciated, as I’m sure the team is (hopefully) working to resolve this behind the scenes instead of focusing on delayed bans.

Addressed this in detail above.

Banning these individuals doesn’t seem like enough...

Totally agree with you.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by TheMilkMan0907

Since Apex legends has just reached 100 million players. Where are the servers for Africa?? We've been playing the game with 190ms.

To open a new datacenter we need to be sure that the number of player will be healthy enough to support matchmaking. Sadly for the moment it is not the case. For example, our lowest population datacenter is in Australia. South Africa account barely for 1/3rd of that.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by Jestersage

All I can say is.... thanks. But I still have my comments:

Regarding ticks: would you mind explaining why we cannot use Battlefield series (which is also published by EA) or even other games that have high number of players and claim to have high tickrate servers? Is it due to the lower TTK means they don't have to process as many information overall?

Regarding the walls-hide situation: Wouldn't it be okay to just go with the flow and award Low-Ping Players? I mean it was the way since Quakeworld (yes, I am old), where having better IRL equipment will give you advantages. (And in fact you acknowledge quite a bit of games did that)

Addressing the first part of your question: At the end of the day, our job is to fulfill the designer's vision on how the game feel given a set of constraints. Apex supports 60 players dropping at the same spot while guaranteeing bandwidth / latency. You can see that a lot if you watch our competitive scene for example. A lot of teams play the placement and get to the final ring. Battlefield works a little bit differently. Without getting into their netcode (not my game!), each game makes different choices and accepts different trade-offs.

over 3 years ago - /u/rkrigney - Direct link

Originally posted by Jestersage

All I can say is.... thanks. But I still have my comments:

Regarding ticks: would you mind explaining why we cannot use Battlefield series (which is also published by EA) or even other games that have high number of players and claim to have high tickrate servers? Is it due to the lower TTK means they don't have to process as many information overall?

Regarding the walls-hide situation: Wouldn't it be okay to just go with the flow and award Low-Ping Players? I mean it was the way since Quakeworld (yes, I am old), where having better IRL equipment will give you advantages. (And in fact you acknowledge quite a bit of games did that)

Samy got the first part of your question so I'll take the second:

Wouldn't it be okay to just go with the flow and award Low-Ping Players? I mean it was the way since Quakeworld (yes, I am old), where having better IRL equipment will give you advantages. (And in fact you acknowledge quite a bit of games did that)

In many ways, you definitely have advantages in Apex if you have better hardware, or faster, more stable internet, or even if you just happen to live near a data center. We don't think our netcode should try to erase those advantages, but we believe we shouldn't exacerbate them or make them more punishing than they already are. People who live in rural areas—or just who happen to not be as close to a server—shouldn't be forced to deal with netcode that always punishes them.

The blog talks about situations where our netcode favors low-ping players as well, so it's important to point out that we actually aren't always favoring or making things tip in the favor of high-ping players by any means.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by -BINK2014-

Probably a stupid question, but have you playtested the mode personally?

Curious to hear your impressions on it compared to BR.

I'm excited as hell for it ever since the leaks. 😁

Worked a lot on the mod. It is very hard to give you an objective opinion of your own baby (of course I love it). I hope you will enjoy it too :)

over 3 years ago - /u/rkrigney - Direct link

Originally posted by KrimSoN1648

Yes. Higher tickrate=more demanding on bandwidth. FPS further increases this figure.

Correct.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by NeonAnderson

As part of the blog post you mention about balancing the experience for both the "low" and the "high" ping person. But one situation I am wondering about if Respawn will do anything about are people with ping so bad that all other players in the match literally see them teleport and cannot hit them as a result as they are teleporting from location to location across your screen. Is Respawn looking into this and going to resolve this issue?

Player teleporting have definitely bad latency but also certainly packet loss. When you have dramatic packet loss, it will mechanically also raise your latency over a threshold that the game cannot compensate. While I do not believe it can be exploited to the high latency player advantage, I agree that the game feel / look broken. We will look into it, maybe our current solution is too defensive.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by piercecraft

With SIDs, does that also mean it's possible to transfer a game instance to a different server on the same rack? For example, if a lobby is slow-mo and its detected, could there be a system put in place to transfer the game to server 3 if server 7 is lagging? Are there any roadblocks that would prevent that, or can the lobby just be paused for the time being while the game instance loads on a different server and redirecting player traffic to that server? Could be a really cool way to prevent slow-mo lobbies from happening entirely because they're extremely frustrating to get in ranked and just lose 60RP for something that wasn't your fault. Really happy about all the interesting new tech going into helping the server side!

It is hard to move around game servers because they consume a lot of memory. Usually, slow-mo is due to server not running up to spec. We hope, even if some suffer a loss because of it, that after a game those servers will not be re-assigned to new players.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by eagles310

Is the Battle(non)sensevideo still relevant to this day https://www.youtube.com/watch?v=9PfFPW9a90w

I think most points raised in the video are explained in the blog post:

  • user command send rate
  • lag compensation
  • server tick rate

Some high numbers are also explained simply by the fact that we use an authoritative-server model. For example for damage, we wait for the server to return the exact number to give you an accurate representation of your opponent health. This makes tracking our server response speed based on the numbers you see a little misleading (which seems to be how Battle(non)sense is doing it.

The downside of our approach is that you have to wait for the server response after shooting someone to see the numbers, which inflates the time to display it on screen. But the upside is that the numbers are accurate, and you can be certain that your opponent lost life, and you can base your strategy on that information.

You can see the hit registration issues this way too, and at least it is factual data. Different game have different constraints. Our Time To Kill being pretty high, having accurate information on your opponent seems for important than instant feedback (that you still get through FX).

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by -BINK2014-

Off-topic:

  • What's a Lead Engineer's Mains in Apex?

Lead means I have a team. Everything I am speaking about is the result of a collaboration between great individuals!

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by Lex_Loewenherz

I had 1 game with around 30 package loss last week, every enemy I had did not even shot me, they were standing still for 1 second every 5 seconds and my hits to them would count as normal hits. So even if the enemy was standing still (which was clearly because of my connection not because of the movement of him) my dmg was being registered. I can only imagine that I teleported and lasered like crazy for the others...

It felt like a big advantage but it only occurred once and could very well be an on-time thing, but I wanted to tell you about it.

Thank you Lex for the feedback. My gut feeling would be that everyone was experiencing massive packet loss due to an infrastructure problem. Do not feel too bad about it.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by Mist_n_Vapor

I think they were asking about which legends you main lol

haha sorry, I mainly play Banga / Wattson.

over 3 years ago - /u/ricklesauceur - Direct link

Originally posted by jeesh101

With the introduction of streaming services like Stadia, xCloud etc, can you see this having an effect on how online-centric games like Apex work, and are designed, in the future?

Would imagine it opens the door for games to be designed with a vastly bigger scope in mind, be it more players or more advanced mechanics, as it effectively removes the client from the equation which formed a lot of discussion in today's blog post. Keen to hear your thoughts on that from a tech standpoint.

Second, not a question, but thank you for all the hard work you and the rest of the team put into the game, and for the blog post today. I love the game, and I can really tell how passionate you all are about it which is great to see. Think it's really cool you're willing to talk at length about the behind-the-scenes systems and the decision making around them. I bet there were some awkward conversations in the office about posting it!

I think the key point in your question is "are designed". I believe that if publishers want to make those experience compelling, a game needs to be designed from the ground up to account for the platform constraints. I am particularly thinking about input delay. For example, aim assist is the consequence of consoles offering a gamepad as interface with the player. You need to run the same train of thought when you design your player / game interactions.

Thank you for the nice words! We are happy to share.