[2022] 20220819 - Verbindungsprobleme (Deutschland) / Connection Issues (Germany)

Our internal network partner, Advania, now has senior people working on this with our external network partner, Cloudflare, and DTAG. We are hoping to be able to provide Cloudflare and DTAG with more data to help them narrow down the issue.

EDIT #1: DTAG in their latest reply indicated they would escalate this issue.

EDIT #2: We are not excluding the network path between Cloudflare and CCP, but if there were issues there then we would expect to see more ISPs being affected; so for the moment that is on the sidelines. Current we are focusing on DTAG’s network and the immediate upstream network (Level3) to Cloudflare’s colos in Germany.

EDIT #3: At the end of today then this issue was escalated to DTAG’s network peering team.

Deutsche Telekom is now asking for more details; they need the exact timestamps of disconnect events to match up against their logs:

When you have a disconnect, please send us a support ticket with the following information:

• Confirmation that your ISP is Deutsche Telekom
• Your geographical location
• Your IP address
• The timestamp of the disconnect event (date and time as exact as you possibly can; in UTC timezone, EVE Time)
• The traceroute to TQ: tracert tranquility.servers.eveonline.com in a CMD window
• The Cloudflare colo info: https://eveonline.com/cdn-cgi/trace in a browser

If you are up for it, please look at this post about running a traceroute continuously, but then please don’t send us the entire output but rather at most three runs (before, during, and after disconnect).

Hello all; posting a status update even if not much has changed. We haven’t forgotten about this issue, we are still working on this matter, but since we don’t have contractual leverage then we are trying to influence the parties involved to improve the internet and this is moving slowly.

The issue has been identified to lie between Deutsche Telekom and Cloudflare involving a carrier called Lumen (previously known as Level3 and CenturyLink) and a saturated carrier link.

CCP has a contractual arrangement with Cloudflare for DDoS Protection and various other services and then with various carriers (but not Lumen) for network traffic between CCP and Cloudflare. CCP doesn’t have contracts with Deutsche Telekom or Lumen and not with Cloudflare as pertaining to external network traffic and routing.

Deutsche Telekom doesn’t peer directly with Cloudflare but we have been encouraging that such direct peering be established since we have seen that resolve issues such as this one and in general this improves the internet – there are many internet exchanges around the world and e.g. CCP connects to LINX in London in addition to the carriers we have. Unexpectedly then Deutsche Telekom expects to be paid for this direct peering even if the peering would be reciprocal and benefit both customers of Deutsche Telekom and Cloudflare equally.

On the one hand, please see my post above 20220819 - Verbindungsprobleme (Deutschland) / Connection Issues (Germany) - #58 by CCP_Explorer.

On the other hand, we can only eliminate Level3 on the network path between us and Cloudflare and we have already done so a while back to resolve connection issues for EVE players in South America and Mexico. We did so by making Level3 a “least preferred” carrier in our carrier and BGP setup resulting in our network equipment never routing across Level3. Instead we use GTT, TATA, Zayo, Telia and LINX.

But we don’t control Deutsche Telekom’s network equipment or carrier setup.

Tranquility is fronted by Cloudflare’s DDoS protection and TCP proxy and so always has the same address across world, tranquility.servers.eveonline.com // 172.65.201.188, but you are routed to the nearest Cloudflare colo.

It finally looks like this has been resolved about 3 days ago:

image1191×478 68.5 KB

We are still waiting for details from Deutsche Telekom on what they changed.

Longer-term view splicing together multiple graphs:

image3063×540 271 KB

It would be super-interesting to see

• The traceroute to TQ: tracert tranquility.servers.eveonline.com in a CMD window
• The Cloudflare colo info: https://eveonline.com/cdn-cgi/trace in a browser

to check if it has changed from when the issues were happening.

Looking at metrics in the past day then this is probably back to being broken.

The issue is clearly solvable, however, as the temporary reprieve demonstrated where some capacity on the congested links was freed up. It’s just that Deutsche Telekom and Cloudflare need to work together, e.g. on any of the solutions we here at CCP have proposed.

We continue to discuss this matter with the parties involved.

And now it’s back to not being broken.

That would be most welcome news if Deutsche Telekom and Cloudflare resolved this issue; a direct reciprocal peering is what we at CCP first and foremost suggested. This here still looks like it’s via Level3/Lumen but hopefully a more stable and less congested path.

i can confirm that, hard connection problems today, connection losses, and times with heavy inputlag …

I can’t stress enough that CCP didn’t do anything to fix the issues, except to complain repeatedly (and bitterly) to Deutsche Telekom for weeks, both directly and through our internal and external network partners and our carriers. Eventually Deutsche Telekom fixed the issue. If the issue is back then Deutsche Telekom has to fix it again. Please contact them, refer them to the previous incident, and ask that they look into the matter.