CCP_Explorer

This appears to be isolated to Jita local. Your corp channels are only affected if you were in Jita when you logged in – a relog outside Jita will fix the corp channels. We are investigating what is happening in Jita, please enjoy the peace and quiet while it lasts.

Fixed, sorry about this.

When you are gathering information for the tickets to Customer Support, please do me a favour and try a couple of things:

In case of disconnects or lag:

• Reboot your internet connection – reboot your computer, your wi-fi, your router, your fiber box. What exactly you need to reboot depends on your setup; in my case the router and the origin wi-fi is the same unit but I have a wi-fi range extender in addition.
• Update your computer – in particular make sure that your antivirus and firewall software is at the latest version and with latest definitions.

In case of lag, but not disconnects:

• Turn off in-game audio – we are investigating one audio defect since 13 Oct in particular that may contribute to a perception of lag. It is being worked on with high priority.

When we are still investigating, I would not want to state that. Right now we are gathering reports from players, looking at metrics, speaking with our network providers, and looking at possible defects that might contribute to a perception of lag.

There could be many things, and one of them would be general network issues in between a player and the nearest Cloudflare location. Those types of issues can be very hard to pinpoint. In addition it could be the players antivirus or firewall software, their wi-fi or router. This is why we kindly ask for support tickets to Customer Support so they can go through debugging of often seen local issues and if needed get the necessary information for the network admins.

The issue was and remains unknown, some disturbance on the internet. Mostly affected European players and cleared up after 17 o’clock (yesterday):

image1463×553 86.7 KB

There was a very specific unknown problem that corrected itself at around 17 o’clock yesterday. If the issue did not happen before then and not after then, then it fixed. We don’t know what that was, something network related somewhere on the internet.

If it happened before then and is still happening now, but is only happening to a very small portion of players, since metrics do not indicate any large-scale wide-spread issues, then there is some probability that this is a network issue that would be located near each player, somewhere on the path from their houses house to the nearest Cloudflare colocation (from where it is routed to TQ). We have seen this before where suddenly the path from certain ISPs to Cloudflare goes bonkers.

Tickets to Customer Support, please, if this is still ongoing and started happening before yesterday.

Please submit a support ticket to Customer Support and make sure to state your RL location, what ISP your are connecting through, what kind of network (wired vs wireless, ADSL vs fiber).

Please send a traceroute from your local computer to tranquility.servers.eveonline.com and a SpeedTest to a nearby location with the ticket.

Customer Support may be able to spot a pattern and figure out what ISP might having problems.

Please submit a support ticket to Customer Support and make sure to state your RL location, what ISP your are connecting through, what kind of network (wired vs wireless, ADSL vs fiber). Please send a traceroute from your local computer to tranquility.servers.eveonline.com and a SpeedTest to a nearby location with the ticket. Customer Support may be able to spot a pattern and figure out what ISP might having problems.

This was fixed on Tuesday 20 October along with various other related improvements: https://www.eveonline.com/article/qi4t9e/patch-notes-for-version-18-10#2020-10-20.1

Tickets to Customer Support, please. We will need your RL location, which I don’t expect you to post here (in fact I don’t want you to post that information here), and we need to be able to collect and correlate the tickets we get.

Again, tickets to Customer Support, please.

Which entails that there is some probability that the issue is network related since TQ has better hardware than SiSi, they are running approx. the same code, but they are located in different parts of the world. Since you are affected and not all players then the network issue would be located near you, somewhere on the path from your house to the nearest Cloudflare colocation (from where it is routed to TQ). We have seen this before where suddenly the path from certain ISPs to Cloudflare goes bonkers.

Please submit a support ticket to Customer Support and make sure to state your RL location, what ISP your...

It’s really tricky to pin-point this if there is not a general problem affecting all players. In the last 30 days there haven’t been much change in latency to the nearest Cloudflare POP

image1851×581 105 KB

(a...

This change has no effect on the Location Node in the EVE Cluster that is hosting the solarsystem in which the fleet fight is happening.

To explain in a bit more detail*: Thousands of EVE Clients connect to the EVE Proxy nodes and their calls get routed to the EVE Server. But the EVE Server is 280 nodes, each with their specific purpose. One such purpose is being a Location Node, which is a node hosting a solarsystem. Another such purpose is being a Cluster Singleton Node, which is a node hosting global / location-agnostic singleton services.

Fleet fights happen in solarsystems and are handled by a specific Location Node. In the case of large fleet fights then...

We had two separate problems this morning; both were network related: An internal one affected the DB machine for the Login Server (timeframe 9:12 to 11:49 UTC) and the other was the general internet outage at Level 3/ CenturyLink that affected Cloudflare: https://www.theverge.com/2020/8/30/21407429/cloudflare-down-websites-hulu-feedly-discord

Outbound from the EVE Launcher and Client then we are connecting to ports 443 on the Login Server, 26000 on the Game Server, and 5222 on the Chat Server. All TCP, we don’t use UDP.

The issue started at 21:10 according to player graphs and started to recover at 21:35.

image916×462 19 KB

Cloudflare had significant issues this evening, see e.g. https://techcrunch.com/2020/07/17/cloudflare-dns-goes-down-taking-a-large-piece-of-the-internet-with-it/ and ...

Which is pretty much what we and Cloudflare, our network partner, could determine. We looked at this incident this morning and Cloudflare was able to determine there were issues reaching our origin for a brief period, but the tunnels between EVE’s datacenter and Cloudflare had no errors and there were no errors on any of our IP transit partners. I want to assure you that we did most certainly not ignore this, but we were not able to establish any cause.

I’m very glad to hear that. To update you all on the current status then this is my latest reply to Cloudflare (and copied to WOW! Internet):

To update you on the latest status: A few players came back to us saying that starting from Wednesday the connection got much better. Then starting from Friday there are no disconnects for those players who replied to us.

We here at CCP made no changes on those days.

Can you tell us if Cloudflare made any changes on either of those days (20 and 22 May) or if Cloudflare and WOW! jointly made some changes, such as peering in ORD/Chicago?