over 2 years ago - CCP_Explorer - Direct link

Please note that the traceroute we are looking for is to TQ; on Windows this can be done from a command line like this:

tracert tranquility.servers.eveonline.com

We need this information to try and figure out what Cloudflare datacenter the connections are going through.

over 2 years ago - CCP_Explorer - Direct link

A number of players have reported that using VPN resolves the connection issues. This is indicative of (changed) QoS settings at your ISP that negatively impact the direct connectivity to Tranquility or BGP routing issues or cross-connect/peering issues, since a VPN connection will send the network traffic through a different route or to a different Cloudflare datacenter.

If you have successfully resolved the connection issues using VPN, then we would want a traceroute both with and without the VPN active, to be able to see the different network paths.

over 2 years ago - CCP_Explorer - Direct link

A common factor in all problematic traceroutes we have seen so far is that these are the last hops:

I can’t confirm it, but that last few hops look like they are in Amsterdam; which is very strange since Cloudflare has colos in Edinburgh, London, and Manchester in the UK and the most “sensible” colo to route UK network traffic for TQ to would be the London one.

over 2 years ago - CCP_Explorer - Direct link

Also sent to Virgin Media?

over 2 years ago - CCP_Explorer - Direct link

I think they are just showing that the IPs belong to Telia, which is a Swedish company. I think the last hops are in Amsterdam based on what other Geo IP services are telling me and also on the ‘adm’ in the name. My traceroute to that last hop before TQ goes through London

and then to those ‘adm’ entries. But here’s where network admins at Virgin Media would know better what is happening.

over 2 years ago - CCP_Explorer - Direct link

This is not the first time something like this has happened. Explained in this thread here Cannot stay connected for more than a few minutes D/C - #109 by CCP_Explorer from May 2020. That thread is mostly about WOW! Internet in the USA but also about Virgin Media in the UK.

over 2 years ago - CCP_Explorer - Direct link

Based on the data we have received so far then only Virgin Media would be able to resolve this (potentially in collaboration with Telia and Cloudflare).

over 2 years ago - CCP_Explorer - Direct link

A graph showing how this is only affecting players in the UK and based on reports only Virgin Media customers. We here at CCP can help Virgin Media resolve this issue but we need your help, EVE players, to bring this to their attention.

image1591×632 146 KB

This started at around 22 o’clock UTC on 23 February.

Longer-term view (only showing the UK):

image1127×525 68.6 KB

over 2 years ago - Suitonia - Direct link

Also experincing the same issues as everyone else with virginmedia, same problematic route.

  6    27 ms    31 ms    30 ms  m674-mp2.cvx1-b.lis.dial.ntli.net [62.254.42.162]
  7    24 ms    22 ms    24 ms  213.46.182.162
  8    32 ms    22 ms    25 ms  adm-bb3-link.ip.twelve99.net [62.115.136.194]
  9    21 ms    27 ms    29 ms  adm-b10-link.ip.twelve99.net [62.115.120.227]
 10     *        *       26 ms  cloudflare-ic342028-adm-b10.ip.twelve99-cust.net [62.115.172.231]
 11    63 ms    36 ms    36 ms  172.65.201.188
over 2 years ago - Suitonia - Direct link

Yeah I used a VPN with a UK server and that completely solved the disconnects for me.

over 2 years ago - Suitonia - Direct link

I was using NordVPN, I imagine most VPNs should work fine though, maybe you can use one with a free trial.

over 2 years ago - CCP_Explorer - Direct link

That indeed shows that your local home network is not the issue and also that from Cloudflare* to Tranquility then there are no issues either. So the issue is somewhere in Virgin Media’s network, in between your home and Cloudflare. If you are in the UK and VPN-ing to another place in the UK, you are simply redirecting the traffic slightly to bypass whatever fault or issue there is. I’m very confident that senior network administrators at Virgin Media will be able to resolve this, the issue at hand is simply getting their attention.

*Tranquility is fronted by Cloudflare; your connection to TQ should normally go directly to the nearest Cloudflare colo (there are 3 of them in the UK, 250 worldwide) and is then carried on Cloudflare’s backbone to the datacenter in London where TQ is.

over 2 years ago - CCP_Explorer - Direct link

I don’t want to use the word “blame”, but Virgin Media’s senior network admins are the prime position to diagnose and fix this issue, which looks like some sort of a peering issue of some kind with Cloudflare.

over 2 years ago - CCP_Explorer - Direct link

Virgin Media UK has evidently raised an internal ticket F009669229 on this issue. If you reported this issue to Virgin Media UK, please update your tickets with a reference to that ticket. We are still trying to figure out how to get our private contact details into that ticket for direct collaboration on resolving this issue.

EDIT: And now our details are in that ticket; now we wait, standing by to provide information or direct help to Virgin Media to resolve this issue.

over 2 years ago - CCP_Explorer - Direct link

That’s both very interesting and weird. You might want to mention this in your ticket with Virgin Media.

over 2 years ago - CCP_Explorer - Direct link

Please submit a ticket to Virgin Media, referencing their ticket F009669229 on this issue.

over 2 years ago - CCP_Explorer - Direct link

This morning there were UK-based disconnect events at 09:05, 10:10, 12:45 and then a minor one at 13:30.

over 2 years ago - CCP_Explorer - Direct link

I trust that Virgin Media’s logs show more than our logs, and that they find out what happened and fix the issue quickly.

over 2 years ago - CCP_Explorer - Direct link

I’m hoping that they get back to all of you on your tickets that you sent them.

over 2 years ago - CCP_Explorer - Direct link

This thread is specifically for ongoing disconnect issues in the UK (that started at 22 o’clock on 23 February). If you are playing from another country or the issue you have is that you are unable to connect at all to begin with, please send a technical support ticket to CCP’s Customer Support.

over 2 years ago - CCP_Explorer - Direct link

If you want, please start a new thread and I can post details on the difference in what we are seeing in metrics and logs.

over 2 years ago - CCP_Explorer - Direct link

Affected Virgin Media UK EVE players: Has your route to TQ changed? Is it no longer being bounced to Amsterdam but rather being handed off in Edinburgh, London, or Manchester?

image1589×633 58.9 KB

over 2 years ago - CCP_Explorer - Direct link

Yep, must saw that 19:50 disconnect spike in metrics.

over 2 years ago - CCP_Explorer - Direct link

Yep, three spikes in a row. Or one spike with three “sub-spikes”.

over 2 years ago - CCP_Explorer - Direct link

Indeed, this issue resolves around BGP routing and peering/cross-connects and how network traffic that should be all-UK ends up in Amsterdam. This is way above what regular ISP tech support can handle. When you contact Virgin Media, please reference their internal F009669229 ticket.

over 2 years ago - CCP_Explorer - Direct link

No; the patch on the 23rd was almost exclusively content; this weekend’s event and localization fixes.

over 2 years ago - CCP_Explorer - Direct link

Indeed, the cause for this is what Virgin Media UK’s network administrators need to find. This is not happening to other players.

over 2 years ago - CCP_Explorer - Direct link

Not all hops respond all the time; it’s very difficult to actually catch the issues happening in the trace route results.

over 2 years ago - CCP_Explorer - Direct link

Your connection should directly to Cloudflare’s colo in Manchester (and from there on Cloudflare’s backbone to TQ in London); not to Amsterdam as this trace route shows. This is at least one of the things Virgin Media needs to fix.

over 2 years ago - CCP_Explorer - Direct link

Currently Virgin Media UK is not responding on the email thread we have with them (or had with them…) so I don’t have any updates to share with you.

Please file tickets with Virgin Media UK referencing their internal fault ticket F009669229 (or update your existing tickets) to let them know that you are impacted.

I’ve been reading the tickets you have sent us here at CCP and it’s same pattern in them all: Virgin Media UK is sending your TQ network traffic to Amsterdam – and CCP has no control over that. I even saw one ticket where an EVE player said they lived “within 30 miles of your TQ data center” and yet the trace route was very clearly sending the traffic to Amsterdam.

We advertise through DNS+BGP that network traffic should be sent via Cloudflare, which has 3 locations in the UK: Edinburgh, Manchester, London. ISPs should either have direct peering with Cloudflare or hand over the traffic as quickly as possible, somewhere in the UK in this case.

While it is very curious to send all-UK network traffic to Amsterdam then it is not necessarily the root cause of the issue. But to even begin trying to diagnose a cryptic issue such as this one then one should remove all noise and in this case the simplest first step is to hand the traffic over to Cloudflare in the UK.

over 2 years ago - CCP_Explorer - Direct link

That 213.46.182.162 address still puzzles us here at CCP. It’s in Dutch range for Liberty Global (Virgin Media UK’s owner) and VodafoneZiggo (also owned by Liberty Global) as a part of ASN 6830. It looks like Virgin Media UK is sending the traffic on their own backbone to Amsterdam and handing the traffic off there to Telia (twelve99.net) that then peers with Cloudflare. That’s how the traffic ends up in Cloudflare’s Amsterdam PoP (which then has to send the traffic back to the UK, where TQ is located).

over 2 years ago - CCP_Explorer - Direct link

EVE Online is very frugal when it comes to bandwidth requirements but can be sensitive to packet loss, latency spikes, over-capacity spikes, and BGP flapping. It needs a stable TCP/IP connection, which is why we have always invested in good networking equipment and partner with companies such as Cloudflare.

EDIT: Virgin Media UK did provide such a stable TCP/IP connection until 22 o’clock on 23 February

image image1587×545 84.5 KB

but then something happened.

over 2 years ago - CCP_Explorer - Direct link

Per my reply above:

While it is very curious to send all-UK network traffic to Amsterdam then it is not necessarily the root cause of the issue. But to even begin trying to diagnose a cryptic issue such as this one then one should remove all noise and in this case the simplest first step is to hand the traffic over to Cloudflare in the UK.

over 2 years ago - CCP_Explorer - Direct link

Indeed; I find it a bit curious that Virgin Media doesn’t peer directly with Cloudflare in the UK (as I know other UK-based ISPs do) but there could be reasons. What I do find curious is to bounce the traffic to Amsterdam and hand it there over to Telia, instead of handing it over to Telia in the UK (as Telia peers with Cloudflare both in the UK and in the Netherlands).

over 2 years ago - CCP_Explorer - Direct link

Just a quick confirmation that we’ve had in the past issues with older Virgin Media hubs. Those issues should be known to Virgin Media and be resolved by now; and also would not explain these synchronous disconnects we see from the UK. Also a quick confirmation that customers of other ISPs in the UK are not complaining to us, only Virgin Media UK customers.

over 2 years ago - CCP_Explorer - Direct link

Yes, we are also working on issues in America, mostly Chile and Mexico. The symptoms there are very different since affected players there are not able to connect at all.

We have been able to reproduce this from one VPN point in Mexico. Based on the analysis so far of that failure then the response from TQ during the initial handshake is all garbled and so we suspect some sort of bad filtering.

over 2 years ago - CCP_Explorer - Direct link

If they replace your hardware and everything starts working again, I would love to hear about it.

over 2 years ago - CCP_Explorer - Direct link

We opened a ticket with Cloudflare today to get more eyes on this issue. We haven’t had an update from Virgin Media / Liberty Global since this morning, when they told us they couldn’t see any issues. I’m trusting I should interpret that, that they couldn’t yet see anything wrong and would continue to investigate.

over 2 years ago - CCP_Explorer - Direct link

Exactly this; we see this clearly in metrics that the disconnects are synchronous and in support tickets that only Virgin Media users are affected. I’m hoping that timing of these disconnect spikes will lead Virgin Media to find the issue.

over 2 years ago - CCP_Explorer - Direct link

Not according to our metrics or technical support. There was some noise in the metrics for German players overnight on the 23rd and again this morning (as in between 1 and 5 o’clock UTC) but fairly clean apart from that and nothing like the regular spikes at Virgin Media.

over 2 years ago - CCP_Explorer - Direct link

Not really:

image1143×531 69 KB

over 2 years ago - CCP_Explorer - Direct link

End of day update:

We have been engaged with Cloudflare today looking at this issue, replying to Cloudflare’s requests for information. We have declared this a “P1 - Urgent” case, the highest priority level we have used since May 2020.

We have also been sending information to Virgin Media but we did not receive any replies today.

over 2 years ago - CCP_Explorer - Direct link

Virgin Media appears to be routing the traffic on their internal backbone (or that of Liberty Global, who owns VM) to Amsterdam to or through an ISP in the Netherlands called VodafoneZiggo (also owned by Liberty Global). From there the traffic is handed off to Telia in Amsterdam who in turn sends it to Cloudflare in Amsterdam since Telia peers directly with them there. So Telia and Cloudflare are doing the correct thing given where the traffic is handed to them. But Telia and Cloudflare peer directly in many other places, including in London and if the traffic was handed to Telia there it would enter Cloudflare’s PoP in London almost next to TQ. We haven’t found if and where Virgin Media or Liberty Global peer directly with Cloudflare; it would appear there would always be a hop through Telia. We have asked VM multiple times why they are sending the traffic to Amsterdam and if they could route within the UK (if only as a test to see if the issue goes away). There have been no replies to questions or suggestions. That routing may not be the root cause, but we have wanted to test to see what would happen.

over 2 years ago - CCP_Explorer - Direct link

Hey @Induce_Deadline: Can you do me a favour? Create a support ticket, reference the ongoing VM issues and include a trace route so that we have a BT comparison? Please name drop me and GM Mechanic in the ticket.

over 2 years ago - CCP_Explorer - Direct link

We are monitoring that different issue, which is mostly manifesting in Chile but also a few EVE players in Mexico have reported. In those cases the connection appears to be outright blocked, whereas as the issue in this thread is that Virgin Media UK EVE players are able to connect and play but are then synchronously disconnected. Please send us a ticket about your issue. We have a code change in the pipes to get better logging of what is happening. In the meantime: If you are able, can you install a VPN (e.g. get a free trial of some VPN to test) and try VPN-ing to some location on the US east coast or to Ireland, the UK, or Netherlands, and let us know (in the support ticket that you will send us) if that changes anything.

over 2 years ago - CCP_Explorer - Direct link

Yep, VPN-ing, even from the UK to the UK is bypassing the fault. We don’t know at this time if the fault is a routing issue, some filtering issue, BGP flapping, a bad port on a router somewhere, a device in VM’s network that runs out of memory and restarts, …

over 2 years ago - CCP_Explorer - Direct link

Yes (mostly) and no. The routing even if inefficient doesn’t necessarily have to be the issue. But as many players in this thread have reported, using a VPN from the UK to the UK is bypassing the fault. so it could be something with the route or on the route.

over 2 years ago - CCP_Explorer - Direct link

Thanks for telling us here that VPN resolves the issue; it’s important for us to hear this so that we can let Virgin Media UK (and Cloudflare know). Please send us a support ticket on your issue as we may ask people to do some experiments for us but most importantly contact Virgin Media and raise the issue with them.

over 2 years ago - CCP_Explorer - Direct link

Last two days with the largest spike (last night) highlighted:

image1579×634 82.1 KB

If there is a pattern then the next disconnect should be at around 15:50.

over 2 years ago - CCP_Explorer - Direct link

I’ve been looking at the past few days; and it almost looks like there is some hourly processing happening at 10 to 5 minutes to the hour and approx. at 11:55, 15:55, 19:50, and 23:55 large disconnect happen every day and smaller ones in between.

over 2 years ago - CCP_Explorer - Direct link
over 2 years ago - CCP_Explorer - Direct link

Oh, fantastic! We need that trace route.

over 2 years ago - CCP_Explorer - Direct link

Thanks; the network people are now chewing on this.

BTW, 162.158.32.0/22 is AS13335, which is Cloudflare. So that last route is VM => CF => Telia => CF.

over 2 years ago - CCP_Explorer - Direct link

I would love to get the full data; if you don’t want to post that here, then please file a support ticket and name drop me and GM Mechanic.

over 2 years ago - CCP_Explorer - Direct link

Hello all Virgin Media UK EVE players!

Fantastic engagement in this thread here as we at CCP continue to feed information to Virgin Media UK and Cloudflare on this issue and great team work!!! Special thanks to @Jander_Massek for catching the route change and to @Harvy_Powney, @Cade102 and everyone else for spotting the regularity and sending us details.

We have a favour to ask you. There appear to be large disconnect spikes every 4 hours, most notably at 11:55, 15:55, 19:55, 23:55 (between midnight and noon then UK players are more offline so it’s difficult to tell) but possibly there it looks like there are also minor hourly disconnects (but sometimes none). This looks a bit like some regular processing that is happening in the approx. 10 minutes before the hour, most notable every 4 hours.

We would want to get more trace routes when the disconnects happens, so if you could run them at 11:55, 15:55, 19:55, 23:55 and if you disconnect to see if results are different than your regular non-disconnection routes. If they are different, please file a support ticket (name dropping me and GM Mechanic) with the full details.

Also, if you are tech-savy and know your way around Wireshark or similar tools, if you run a network capture during a disconnect, that would also be valuable information to us.

Finally, it would be interesting to hear if you run both a TQ client and a SiSi client at the same time, if the SiSi client stays connected but the TQ client disconnects.

over 2 years ago - CCP_Explorer - Direct link

We are amazed that you caught it; Cloudflare had been asking for this and we had been telling them this would be next to impossible.

BTW, we have already sent this to Cloudflare and Virgin Media. VM has already replied and escalated the case further.

over 2 years ago - CCP_Explorer - Direct link

Indeed, we think the trace route is “corrupt” in the sense that you started it when the new route was in effect and as the trace route was running then new old route came back into effect; so the result is a mix of the two. We are still hoping it provides insight to Cloudflare and Virgin Media, along with the discovery of the regularity, but more trace routes would help us and them piece together what that new route is.

over 2 years ago - CCP_Explorer - Direct link

Our test server network is in a different geographical location to the production network and is not behind Cloudflare’s systems. If you trace the route, then you will get very different paths.

% nslookup tranquility.servers.eveonline.com
Name: d638e439e07f413c97200d057e5ebd05.pacloudflare.com
Address: 172.65.201.188
Aliases: tranquility.servers.eveonline.com
tranquility.servers.eveonline.com.cdn.cloudflare.net

% nslookup singularity.servers.eveonline.com
Name: singularity.servers.eveonline.com
Address: 87.237.38.2

(And this IP address above for TQ is not its “correct” destination IP address; only Cloudflare can access that one.)

over 2 years ago - CCP_Explorer - Direct link

We will know when either they tell us or the disconnects stop happening… There was a disconnect spike at 19:00 this time and nothing at 19:55. But then again, there were minor spikes hourly even there were larger spikes every 4 hours.

over 2 years ago - CCP_Explorer - Direct link

Is it a count-down?!? :grimacing:

over 2 years ago - CCP_Explorer - Direct link

Forwarded to Cloudflare; they were quick to point out they only got half of the data in our first reply. :smiley:

over 2 years ago - CCP_Explorer - Direct link

There is a grain of truth here on why it may feel like this matters. What matters is if the client is active or not, and it always active in space (receiving physics simulation updates) while it may be less active in station and therefore a brief issue like this may pass it by.

The scenario we think we are looking (this is a conjecture/hypothesis; but not a theory yet, for that we are waiting on more data from players and replies from Virgin Media and Cloudflare) are regular but very brief BGP flaps / routing changes. Your EVE Client initially established a TCP connection to Cloudflare’s Amsterdam PoP (which is incorrect, but that’s how Virgin Media routes their traffic :man_shrugging:) and the firewalls and proxies there will have the state of your TCP connection.

That state is not synched in realtime to other routes or colos. During the BGP flap / routing change then your client suddenly sends data or even just an ACK packet to another Cloudflare colo (possibly the London colo) or through a different path to the Amsterdam colo, a route on which there are firewalls that have no information on the state of your connection. Your EVE Client just gets an RST packet from that device because they have no idea who you are and the EVE Client disconnects in response to the RST packet. Or the packet ends up in another colo and the proxy has no record of you in their proxy connection map, and sends an RST packet back.

Meanwhile the Cloudflare equipment in Amsterdam just sees your connection go silent. At some point it times out or when you log back in again on a new TCP connection then Tranquility usurps your previous connection to reconnect you to the game.

If the client is not active and the flap is brief, then through many of these flaps, you may feel like the connection is better while in station. But if you were very active in the station, you would get disconnected.

over 2 years ago - CCP_Explorer - Direct link

Yeah, this something hourly. I think Virgin Media is possibly fetching BGP routes from the internet every hour and applying them to some equipment. But then cost-saving changes are applied on top (that routing to Amsterdam). But for a brief moment before the cost-saving routes are applied then the traffic goes a different route and the equipment on that route sends back RST packets.

over 2 years ago - CCP_Explorer - Direct link

I would prefer to get the full details in support tickets to us, please, and then post the ticket number here.

over 2 years ago - CCP_Explorer - Direct link

Last 16 hours; happens shortly before hour:

image1578×536 52.3 KB

over 2 years ago - Suitonia - Direct link

Didn’t get DC’d between 23:50 to 00:20, was running traceroute the entire time and this was the route.

 5     *        *        *     Request timed out.
  6    31 ms    60 ms    50 ms  m674-mp2.cvx1-b.lis.dial.ntli.net [62.254.42.162]
  7    22 ms    33 ms    22 ms  213.46.182.162
  8    24 ms    21 ms    37 ms  adm-bb3-link.ip.twelve99.net [62.115.136.194]
  9    27 ms    38 ms    27 ms  adm-b10-link.ip.twelve99.net [62.115.120.227]
 10    29 ms    22 ms    23 ms  cloudflare-ic342028-adm-b10.ip.twelve99-cust.net [62.115.172.231]
 11    27 ms    32 ms    32 ms  172.65.201.188

Trace complete.
over 2 years ago - CCP_Explorer - Direct link

For now many Virgin Media customer have used VPN (from the UK to the UK) to bypass the fault. Long-term then either Virgin Media finds that fault and fixes it (I have a relatively good confidence in that they do) or you should consider looking for another ISP that offers a high-quality, stable internet that is suitable for long-lived TCP connections. Honestly then Virgin Media did appear to offer such services before 22 o’clock on 23 February.

I don’t know if you can get compensated by Virgin Media for having to use VPN to bypass the fault.

We can’t fix internet network routing with a patch, and the server is fine.

We don’t control the internet network routing, in particular we do not control how Virgin Media routes their network. CCP with its external-network partner Cloudflare advertise through DNS and BGP where Tranquility is and where to route network traffic to. But ISPs control the routing within their own networks, what carriers they use, and where they peer with other networks. In particular, they control their egress points to other networks and don’t have to follow the shortest path. The best path from an ISP’s perspective is a combination of hops and cost. But BGP flaps and frequent routing changes are the bane of stable long-lived TCP connections, such as the one the EVE Client has to the EVE Server.

over 2 years ago - CCP_Explorer - Direct link

I’ve forwarded this.

over 2 years ago - CCP_Explorer - Direct link

Also forwarded.

over 2 years ago - CCP_Explorer - Direct link

Because the address is global and is routed to the nearest Cloudflare PoP. If you are in New Zealand then you get routed to Auckland, if you are in Scotland then you get routed to Edinburgh – and then Cloudflare transfer the data on their backbone to the GRE tunnels CCP has with them. See https://www.cloudflare.com/network/. My route to Tranquility, when not logged onto CCP’s work-at-home VPN, looks like this:

3 VFIS-GR-1.c.is [193.4.254.160]
4 fo-0-1-1-0-CCR01-Sidumuli.c.is [217.151.183.98]
5 rix-tg-gw.cloudflare.com [195.130.211.50]
6 172.65.201.188

where c.is is the backbone name for Vodafone Iceland and rix-tg-gw is at the Reykjavik Internet Exchange (RIX) where Vodafone Iceland and Cloudflare peer directly.

over 2 years ago - CCP_Explorer - Direct link

Thanks! Love it. :heart:

over 2 years ago - CCP_Explorer - Direct link

BTW, Cloudflare was having Cloudflare Status - Network Performance Issues in Amsterdam, Netherlands region this morning. They should have been resolved at 10:31 UTC. Details in the link. (Normally this should not be relevant to UK players, but this entire thread is about strange routing.)

over 2 years ago - CCP_Explorer - Direct link

Thanks! Anything that doesn’t look like the regular route, would be helpful to know about.

over 2 years ago - CCP_Explorer - Direct link

Excellent, we have seen this hop in previous disconnect trace routes but it is not present in the normal ones.

over 2 years ago - CCP_Explorer - Direct link

@Alaska, can you perhaps send us a support ticket with the full traceroute from hop 1? Please post the ticket number here.

over 2 years ago - CCP_Explorer - Direct link

Many thanks; forwarded.

over 2 years ago - CCP_Explorer - Direct link

Actually, @Alaska, can you also post in the ticket the regular trace route, please; for comparison?

over 2 years ago - CCP_Explorer - Direct link

Right, now I’m interested in hearing if you are all getting new routes? Virgin Media just told us “Issue looks to be due to flapping BGP between Cloudfare and Virgin Media due to prefix limit breach.”

You should now be getting routes that go directly to the nearest Cloudflare PoP, of which there are 3 in the UK: Edinburgh, Manchester, London – so depending on if you are in the north, south or center of the island then you should get the different PoPs and your route should stay that way during your EVE play session.

EDIT: If Virgin Media is only peering with Cloudflare in one of those three locations then you will all be routed through that location. It looks like Virgin Media is peering with Cloudflare in Manchester.

over 2 years ago - CCP_Explorer - Direct link

Using one of the trace routes from above for some quick analysis and details to you, patient EVE players:

3 12 ms 11 ms 10 ms bagu-core-2a-ae70-650.network.virginmedia.net [213.106.214.185]
4 * * * Request timed out.
5 12 ms 16 ms 13 ms tcma-ic-2-ae9-0.network.virginmedia.net [62.253.174.178]
6 22 ms 42 ms 37 ms 162.158.32.254
7 16 ms 16 ms 18 ms 162.158.32.11
8 18 ms 18 ms 14 ms 172.65.201.188

You will note that there is no more Telia in between Virgin Media and Cloudflare, no more Liberty Global, and no more Amsterdam.

Rather in hops 5 and 6 above the route goes from 62.253.174.178 @ AS 5089 (Virgin Media) to 162.158.32.254 @ AS 13335 (CloudFlare) in a direct peering; and that direct peering appears to happen in Manchester.

From what we can tell from this all (and from the email from Virgin Media we got earlier) then the direct peering between Virgin Media and Cloudflare was down because of a prefix limit. It was automatically brought back up regularly to see if the prefix limit had become OK, which it wasn’t and it was taken back down again. That limit has been increased and the connection is under monitoring.

I will be monitoring metrics on my side, will hopefully be able to post a final update tomorrow, fingers crossed.

I have many favours to ask of you:

  • I need you to tell us if your play session is now more stable.
  • Can you find other Cloudflare clients that you are using (Discord is one, what else do you know of that you use) and check the routes for them; they should have changed.
  • Can someone update Eve Online Connection Issues (Fault Ticket = F0096... - Virgin Media Community - 4951509 and reference my last two posts here (this one and the “Right, now I’m interested in hearing if you are all getting new routes?” above).
over 2 years ago - CCP_Explorer - Direct link

Hmmm… but your trace route to TQ has changed to the direct peering? I have to admit, I’m slightly concerned.

over 2 years ago - CCP_Explorer - Direct link

Last 10 days; now we wait to see if this graph stabilizes:

image1584×540 110 KB

over 2 years ago - CCP_Explorer - Direct link

I don’t have the details to be able to answer this question – only Virgin Media would be able to tell what the prefix limit was before and after on their equipment, what alerts there are on the limit being breached and their direct peering with Cloudflare therefore being down, how that limit compares to the overall size of the internet’s BGP table and the number of their direct network peers, etc.

over 2 years ago - CCP_Explorer - Direct link

@Lacrimosa: Referring you to my two posts right above, please see

  • Right, now I’m interested in hearing if you are all getting new routes?
  • Using one of the trace routes from above for some quick analysis

then please note that Virgin Media UK literally fixed the issue themselves by literally fixing

“Issue looks to be due to flapping BGP between Cloudfare and Virgin Media due to prefix limit breach.”

on literally their own equipment, thereby literally fixing/re-establishing their own direct peering with Cloudflare. None of this was or is under CCP’s control or under the control of our internal network partner Advania or our external network partner Cloudflare or any of our network carriers or direct network peers.

over 2 years ago - CCP_Explorer - Direct link

This is looking good since Virgin Media UK’s fix this afternoon:

image1143×535 60.2 KB

over 2 years ago - CCP_Explorer - Direct link

I’m all ears (not literally this time) what that other alternative should have been.

The internet isn’t a single unified entity; it’s a network of networks glued together with BGP and we here at CCP only control our own network.

over 2 years ago - CCP_Explorer - Direct link

OK, I remain a bit puzzled here at Virgin Media UK’s decisions, but if it works and is stable, then I’m not going to complain… Not that I have much say in the matter, it’s their network and ultimately I can only advise, recommend, and provide information to VM and VM’s customers need to complain.

But to explain why this is weird:

EVE’s destination is 172.65.201.188, which is a part of the 172.65.128.0/17 CIDR range on AS 13335 (which is Cloudflare, Inc.). Discord’s destination is 162.159.137.232, which is a part of the 162.159.128.0/17 CIDR range, which is also on AS 13335.

Given that Virgin Media UK has direct peering with Cloudflare, then it’s weird to route some of the ranges on AS 13335 one way and other ranges another way. But perhaps there is something in the BGP routing here that I am missing.

over 2 years ago - CCP_Explorer - Direct link

Could be.

We explicitly suggested to VM on Monday evening that they “try a different egress point to reach 172.65.192.0/20”.

Prior, early in the morning on that same day, we had suggested they could “perhaps change that for 172.65.201.188 (or for Cloudflare in general, AS13335) and route directly within the UK to Cloudflare? Perhaps even initially only as a test to see if a new route makes a difference.”

BTW, the fault was that VM’s systems were trying to establish this direct peering (that they ended up with) but their systems were failing because of prefix limits being breached.

over 2 years ago - CCP_Explorer - Direct link

You are in Mexico or Chile? If, please file a support ticket.

over 2 years ago - CCP_Explorer - Direct link

UK status since 1 Mar:

image1140×541 68.4 KB

I’ll update here and on Twitter on a few days with longer-term data, but this case is closed.

over 2 years ago - CCP_Explorer - Direct link

This thread is only about the disconnect issues that were affecting Virgin Media UK customers. For your issue, which is of a different nature and with a different ISP, please create a technical support request.

over 2 years ago - CCP_Explorer - Direct link

@Adelrad, you are not experiencing disconnect issues but rather that you can’t connect at all? And in the Client logs there is an “Invalid answer from server GetServerStatus” error?

over 2 years ago - CCP_Explorer - Direct link

After Virgin Media made a fix in the early afternoon on 3 Mar (between 13:30 - 13:45 UTC) then all was good over the weekend until last night when there were small disconnect spikes at 01:01 and 03:25. Since then all been quiet.

I replied to the email thread we had with Virgin Media and Liberty Global to let them know of this latest development.

But: If this continues then you, as customers of Virgin Media, need to open cases with them. Then please reference your previous ticket(s), all the details here in this forum thread, as well as the details in https://community.virginmedia.com/t5/Gaming-Support/Eve-Online-Connection-Issues-Fault-Ticket-F009669229/m-p/4951509.

over 2 years ago - CCP_Explorer - Direct link

Even after the fixes before the weekend then the graph was a bit noisy. After the disconnects this morning it is actually a bit more quiet, but then again this is Monday morning and so less people from the UK are online.

image1142×544 65.6 KB

Virgin Media has responded to the email I sent this morning confirming that they received it.

over 2 years ago - CCP_Explorer - Direct link

Status in the last 6 days, closing this incident:

image1140×535 60.3 KB

over 2 years ago - CCP_Explorer - Direct link

Another view on the overall incident:

image1199×392 48 KB

over 2 years ago - CCP_Explorer - Direct link

Please send a technical support ticket to CCP’s Customer Support. Please include the details asked for in the first post: 20220225 - Connection Issues (UK).