CCP_Explorer

Hello all Virgin Media UK EVE players!

Fantastic engagement in this thread here as we at CCP continue to feed information to Virgin Media UK and Cloudflare on this issue and great team work!!! Special thanks to @Jander_Massek for catching the route change and to @Harvy_Powney, @Cade102 and everyone else for spotting the regularity and sending us details.

We have a favour to ask you. Ther...

I would love to get the full data; if you don’t want to post that here, then please file a support ticket and name drop me and GM Mechanic.

Thanks; the network people are now chewing on this.

BTW, 162.158.32.0/22 is AS13335, which is Cloudflare. So that last route is VM => CF => Telia => CF.

Oh, fantastic! We need that trace route.

#confirmed:

image1134×537 53.8 KB

I’ve been looking at the past few days; and it almost looks like there is some hourly processing happening at 10 to 5 minutes to the hour and approx. at 11:55, 15:55, 19:50, and 23:55 large disconnect happen every day and smaller ones in between.

Last two days with the largest spike (last night) highlighted:

image1579×634 82.1 KB

...

Thanks for telling us here that VPN resolves the issue; it’s important for us to hear this so that we can let Virgin Media UK (and Cloudflare know). Please send us a support ticket on your issue as we may ask people to do some experiments for us but most importantly contact Virgin Media and raise the issue with them.

Yes (mostly) and no. The routing even if inefficient doesn’t necessarily have to be the issue. But as many players in this thread have reported, using a VPN from the UK to the UK is bypassing the fault. so it could be something with the route or on the route.

Yep, VPN-ing, even from the UK to the UK is bypassing the fault. We don’t know at this time if the fault is a routing issue, some filtering issue, BGP flapping, a bad port on a router somewhere, a device in VM’s network that runs out of memory and restarts, …

We are monitoring that different issue, which is mostly manifesting in Chile but also a few EVE players in Mexico have reported. In those cases the connection appears to be outright blocked, whereas as the issue in this thread is that Virgin Media UK EVE players are able to connect and play but are then synchronously disconnected. Please send us a ticket about your issue. We have a code change in the pipes to get better logging of what is happen...

Hey @Induce_Deadline: Can you do me a favour? Create a support ticket, reference the ongoing VM issues and include a trace route so that we have a BT comparison? Please name drop me and GM M...

Virgin Media appears to be routing the traffic on their internal backbone (or that of Liberty Global, who owns VM) to Amsterdam to or through an ISP in the Netherlands called VodafoneZiggo (also owned by Liberty Global). From there the traffic is handed off to Telia in Amsterdam who in turn sends it to Cloudflare in Amsterdam since Telia peers directly with them there. So Telia and Cloudflare are doing the correct thing given where the traffic is handed to them. But Telia and Cloudflare peer directly ...

End of day update:

We have been engaged with Cloudflare today looking at this issue, replying to Cloudflare’s requests for information. We have declared this a “P1 - Urgent” case, the highest priority level we have used since May 2020.

We have also been sending information to Virgin Media but we did not receive any replies today.

Not really:

image1143×531 69 KB

Not according to our metrics or technical support. There was some noise in the metrics for German players overnight on the 23rd and again this morning (as in between 1 and 5 o’clock UTC) but fairly clean apart from that and nothing like the regular spikes at Virgin Media.

Exactly this; we see this clearly in metrics that the disconnects are synchronous and in support tickets that only Virgin Media users are affected. I’m hoping that timing of these disconnect spikes will lead Virgin Media to find the issue.

We opened a ticket with Cloudflare today to get more eyes on this issue. We haven’t had an update from Virgin Media / Liberty Global since this morning, when they told us they couldn’t see any issues. I’m trusting I should interpret that, that they couldn’t yet see anything wrong and would continue to investigate.

If they replace your hardware and everything starts working again, I would love to hear about it.

Yes, we are also working on issues in America, mostly Chile and Mexico. The symptoms there are very different since affected players there are not able to connect at all.

We have been able to reproduce this from one VPN point in Mexico. Based on the analysis so far of that failure then the response from TQ during the initial handshake is all garbled and so we suspect some sort of bad filtering.