CCP_Explorer

And now it’s back to not being broken.

Looking at metrics in the past day then this is probably back to being broken.

The issue is clearly solvable, however, as the temporary reprieve demonstrated where some capacity on the congested links was freed up. It’s just that Deutsche Telekom and Cloudflare need to work together, e.g. on any of the solutions we here at CCP have proposed.

We continue to discuss this matter with the parties involved.

It would be super-interesting to see

• The traceroute to TQ: tracert tranquility.servers.eveonline.com in a CMD window
• The Cloudflare colo info: https://eveonline.com/cdn-cgi/trace in a browser

to check if it has changed from when the issues were happening.

It finally looks like this has been resolved about 3 days ago:

image1191×478 68.5 KB

We are still waiting for details from Deutsche Telekom on what they changed.

Longer-term view splicing together multiple graphs:

...

Tranquility is fronted by Cloudflare’s DDoS protection and TCP proxy and so always has the same address across world, tranquility.servers.eveonline.com // 172.65.201.188, but you are routed to the nearest Cloudflare colo.

On the one hand, please see my post above ...

Hello all; posting a status update even if not much has changed. We haven’t forgotten about this issue, we are still working on this matter, but since we don’t have contractual leverage then we are trying to influence the parties involved to improve the internet and this is moving slowly.

The issue has been identified to lie between Deutsche Telekom and Cloudflare involving a carrier called Lumen (previously known as Level3 and CenturyLink) and a saturated carrier link.

CCP has a contractual arrangement with Cloudflare for DDoS Protection and various other services and then with various carriers (but not Lumen) for network traffic between CCP and Cloudflare. CCP doesn’t have contracts with Deutsche Telekom or Lumen and not with Cloudflare as pertaining to external network traffic and routing.

Deutsche Telekom doesn’t peer directly with Cloudflare but we have been encouraging that such direct peering be established since we have seen that resolve issues such as t...

Deutsche Telekom is now asking for more details; they need the exact timestamps of disconnect events to match up against their logs:

When you have a disconnect, please send us a support ticket with the following information:

• Confirmation that your ISP is Deutsche Telekom
• Your geographical location
• Your IP address
• The timestamp of the disconnect event (date and time as exact as you possibly can; in UTC timezone, EVE Time)
• The traceroute to TQ: tracert tranquility.servers.eveonline.com in a CMD window
• The Cloudflare colo info: https://eveonline.com/cdn-cgi/trace in a browser

If you are up for it, please look at ...

Our internal network partner, Advania, now has senior people working on this with our external network partner, Cloudflare, and DTAG. We are hoping to be able to provide Cloudflare and DTAG with more data to help them narrow down the issue.

EDIT #1: DTAG in their latest reply indicated they would escalate this issue.

EDIT #2: We are not excluding the network path between Cloudflare and CCP, but if there were issues there then we would expect to see more ISPs being affected; so for the moment that is on the sidelines...

The graph is still not good:

image1378×538 92 KB

We are in contact with Deutsche Telekom about this issue. Based on the symptoms observed and previous incidents like this, then this will get fixed; but it’s impossible to tell what the timeline could be.

Thanks for all the details; happy to hear that Proton VPN is working as a work-around for now. Connecting via Netherlands (most likely Amsterdam) is a good choice anyway since it’s on the route to London, where TQ is, and it is a big network hub. What the VPN does in this case is just to reroute your network traffic away from the problematic area, which appears to be somewhere in München.

Yes, running the batch file commands continuously in the background and then on disconnect try and see if the traceroute changed at the same time (scroll through the traceroutes and see if the last few ones are the same or not).

Very interesting, but this network route instability will ...

Can you do us a favour, please, and update that ticket with a traceroute with VPN on and with VPN off?

Can you do us a favour, please, and update that ticket with a traceroute with VPN on and with VPN off?

MUC is Münich and FRA is Frankfurt. Interesting to hear that the disconnects might be linked to a DTAG network route to Münich. Have you been able to get a traceroute while it is connected to MUC?

Tranquility and Singularity are not in the same country, not on the same network, not with the same hosting or network providers, and are protected by different DDoS protections.

For a good...

I’m glad to hear that many of your have been able to resolve or mitigate the issue using a VPN; that builds a case that the underlying root cause is somewhere on the regular network path from your home network to Cloudflare.

A web page is not a real-time system that keeps on ticking. It is state-less and will just stay there until you come back. If you disconnect from EVE then the simulation carries on and we must re-connect you with your session. But the simulation won’t be in the same state as when you left. In fact, as soon as we detect that something is amiss, then you are automatically emergency-warped away. Your web page doesn’t...

What we are initially looking for is basic debug information so that Deutsche Telekom and Cloudflare can investigate this issue:

• Your geographical location, your ISP details, and other relevant information about where and how you connect to the internet. If there are any patterns to the disconnects, please let us know as well.
• The network path from your computer to Tranquility: Run tracert tranquility.servers.eveonline.com from a Command-Line Window, save the output in a text file and send us.
• Your network details and how you connect to Cloudflare: In https://web.ccpgamescdn.com/aws/community/eve_network_diagnostics_v6.zip then you will find a file called eve-network-diagnostics v6.ps1, which is a Windows PowerShell Sc...

We are moving all discussion on disconnect issues affecting customers of Deutsche Telekom AG to this new thread: