EVE Online Dev Tracker

Oh, fantastic! We need that trace route.

#confirmed:

image1134×537 53.8 KB

I’ve been looking at the past few days; and it almost looks like there is some hourly processing happening at 10 to 5 minutes to the hour and approx. at 11:55, 15:55, 19:50, and 23:55 large disconnect happen every day and smaller ones in between.

Last two days with the largest spike (last night) highlighted:

image1579×634 82.1 KB

...

Thanks for telling us here that VPN resolves the issue; it’s important for us to hear this so that we can let Virgin Media UK (and Cloudflare know). Please send us a support ticket on your issue as we may ask people to do some experiments for us but most importantly contact Virgin Media and raise the issue with them.

Yes (mostly) and no. The routing even if inefficient doesn’t necessarily have to be the issue. But as many players in this thread have reported, using a VPN from the UK to the UK is bypassing the fault. so it could be something with the route or on the route.

Yep, VPN-ing, even from the UK to the UK is bypassing the fault. We don’t know at this time if the fault is a routing issue, some filtering issue, BGP flapping, a bad port on a router somewhere, a device in VM’s network that runs out of memory and restarts, …

We are monitoring that different issue, which is mostly manifesting in Chile but also a few EVE players in Mexico have reported. In those cases the connection appears to be outright blocked, whereas as the issue in this thread is that Virgin Media UK EVE players are able to connect and play but are then synchronously disconnected. Please send us a ticket about your issue. We have a code change in the pipes to get better logging of what is happen...

Hey @Induce_Deadline: Can you do me a favour? Create a support ticket, reference the ongoing VM issues and include a trace route so that we have a BT comparison? Please name drop me and GM M...

Virgin Media appears to be routing the traffic on their internal backbone (or that of Liberty Global, who owns VM) to Amsterdam to or through an ISP in the Netherlands called VodafoneZiggo (also owned by Liberty Global). From there the traffic is handed off to Telia in Amsterdam who in turn sends it to Cloudflare in Amsterdam since Telia peers directly with them there. So Telia and Cloudflare are doing the correct thing given where the traffic is handed to them. But Telia and Cloudflare peer directly ...

End of day update:

We have been engaged with Cloudflare today looking at this issue, replying to Cloudflare’s requests for information. We have declared this a “P1 - Urgent” case, the highest priority level we have used since May 2020.

We have also been sending information to Virgin Media but we did not receive any replies today.

Removed some offtopic posts

Not really:

image1143×531 69 KB

Not according to our metrics or technical support. There was some noise in the metrics for German players overnight on the 23rd and again this morning (as in between 1 and 5 o’clock UTC) but fairly clean apart from that and nothing like the regular spikes at Virgin Media.

Exactly this; we see this clearly in metrics that the disconnects are synchronous and in support tickets that only Virgin Media users are affected. I’m hoping that timing of these disconnect spikes will lead Virgin Media to find the issue.

We opened a ticket with Cloudflare today to get more eyes on this issue. We haven’t had an update from Virgin Media / Liberty Global since this morning, when they told us they couldn’t see any issues. I’m trusting I should interpret that, that they couldn’t yet see anything wrong and would continue to investigate.

If they replace your hardware and everything starts working again, I would love to hear about it.

Yes, we are also working on issues in America, mostly Chile and Mexico. The symptoms there are very different since affected players there are not able to connect at all.

We have been able to reproduce this from one VPN point in Mexico. Based on the analysis so far of that failure then the response from TQ during the initial handshake is all garbled and so we suspect some sort of bad filtering.

Just a quick confirmation that we’ve had in the past issues with older Virgin Media hubs. Those issues should be known to Virgin Media and be resolved by now; and also would not explain these synchronous disconnects we see from the UK. Also a quick confirmation that customers of other ISPs in the UK are not complaining to us, only Virgin Media UK customers.

Hey Capsuleers – we have an update with the DBS situation and appreciate your patience as we went through our investigation.

Many players noticed drastic changes to the Bounty Risk Modifiers in their solar systems beginning on February 16th. We immediately investigated this issue and concluded that, while the external system doing the Bounty Risk Modifier calculations was still receiving the correct data, it was not properly relaying that information back to Tranquility.

We are acutely aware of how the Bounty Risk Modifier in the solar system dictates player decisions when it comes to choosing a suitable location, and the impact that erroneous information has. Due to this outage, we will be restoring the Bounty Risk Modifier for every solar system. We will be looking at the Bounty Risk Modifier from the date of the outage and the date of the restoration for each solar system, and restore the BRM to the higher of those two values.

The restoration will not be do...