[2021] No Downtime - Again!

The second ‘no downtime’ experiment is upon us and there will be no downtime on Thursday, September 9 this week. You can read the development blog from @CCP_Explorer to get some additional insights

We got a pretty cool video in the works and there will be a live stream on CCP TV on Thursday as well. See you there!

BRB changing dev name.

The regular asteroid belts do not get replenished.

Sometimes the changes are internal. For example, in the last 10 days or so, the majority of the updates were build system or technology framework fixes.

Neither; but more the former. It wasn’t targeted optimization but rather code and hardware changes that then resulted in less memory pressure.

This is not related to the #nodowntime experiment. The login server (SSO) was updated at 10:30 on Monday morning.

The issue isn’t that we’re lacking a time sync source, a regular NTP will do just fine. It was getting old code to do new things, such as listening to that NTP source and correcting itself regularly while always moving time forward.

That should not be legal!

That is possible, but more complicated than simply reducing the number of downtimes (and eventually getting rid of them).

Not by any meaningful amount. We were at 4 minutes today and we can realistically only hope to shave off seconds here and there. Any effort towards significantly reducing that would be better invested in reducing the number of downtimes.

No; Windows locks many of the files that need to be updated while the process is running.

No.

There are longer-term ideas on how to replace the asteroid belts. Currently they are actual real items in the inventory system that are replenished during downtime (in a DB job while the inventory system is offline) and need to be replaced with sites that spawn based on a distribution and regularly replenished.

Currently they are actual real items in the inventory system that are replenished during downtime in a DB job while the inventory system is offline. Since the cluster doesn’t go offline then the inventory system is still up-and-running and then the DB jobs must not run.

Yes, there was downtime today, Wednesday, and will be downtime again on Friday.

Since you ask; it has happened, on test servers, that devs have put suns into their cargo holds. It was fun.

It has also happened, on TQ, through misbehaving code that has now been fixed, that the Universe Root was put into a structure that was in a ‘Inventory Graveyard’ (inventory items that are deleted are first moved from the main inventory table into graveyards and then later archived). It was not fun, it caused endless loops in the inventory system under certain conditions since the root item now had a parent.

There should be. If there are any daily things that you do or claim today, and then you can’t tomorrow; then please create a bug report and ping me here again with the EBR number.

For the regular asteroid belts: Confirmed.

That system needs to be reworked to not depend on a DB job during downtime, before we make fewer downtimes the norm.

We’re still on day #1, the main part of the experiment starts after 11 o’clock EVE Time tomorrow.

To engage players to help us test; see e.g. https://twitter.com/KhapriceHeska/status/1435572675171192839

What Geo said – but to find out how all the different systems in EVE respond on this scale, then we are testing what happens.

If there are any daily things that you do or claim today, and then you can’t tomorrow; then please create a bug report and ping me here again with the EBR number.

There has been a fair amount of comments & questions in this thread on why we are looking at downtime; what’s the problem with a few minutes each day.

The first problem is that it interrupts primary play sessions for players in Asia and Oceania. Downtime at 11 o’clock EVE Time doesn’t affect Europe that much and doesn’t really affect the Americas at all. I covered this in detail in my 2019 dev blog on No Downtime Vol. 1 and the various options if downtime was not at 11 o’clock.

The second problem is that this is a larger impact than a few minutes. Shortly before downtime players start looking at the clock, trying to figure out if they have time before downtime to complete more of what they are doing. For example, can they run one more mission? We see this clearly on the online graphs that the population starts to dip around 10 o’clock and then it takes a while for players to start logging in again. It should look like the red line:

image1057×297 16.3 KB

At the very extreme it can be argued that some impact of this daily reboot lasts from 09:40 until 13:20, i.e., from the pre-downtime peak until the population reaches the levels it would have been at if the play session had not been interrupted, or 3 hours and 40 minutes. Complicating this simplistic analysis of course are that players in Asia and Oceania would normally be starting to log off anyway in this period and 11 o’clock EVE Time is noon/lunch time on mainland Europe and people would log out to grab something to eat. We only have one datapoint on the uninterrupted play session, from Vol. 1 of this experiment in 2019, and this Vol. 2 experiment now will tell us more.

There are no changes to code or content today. We are setting up tests, disabling things that would run at tomorrow’s downtime (if there was one), and informing players and getting them engaged. The primary part of this experiment starts at 11 o’clock tomorrow.

EDIT: I checked with devs. The Caldari Union Day event ended today and there are no other time-limited events currently running. The daily login campaign is active.

I’m not sure…

Before #nodowntime becomes are regular pattern, these sort of issues would need to be resolved.

Excellent!

Expected, the regular asteroid belts were not replenished.

No; they will be replenished in tomorrow’s downtime.

Discourse is being weird - I’m trying to fix that feedback thread.

Currently the regular asteroid belts are actual real items in the inventory system that are replenished during downtime in a DB job while the inventory system is offline. Since the cluster didn’t go offline then the inventory system was still up-and-running and then the DB jobs could not run.

Yep, a known issue with the regular asteroid belts.

They are not actual inventory items, but they are “Dungeon Distributions” as far as I know and this particular feature may be something that only runs on server startup. Please send us a bug report and reply back with the EBR number.

I don’t think any events or sprees were active yesterday… (and are therefore not active today and won’t be active tomorrow) but I’m asking around.

I am chiming in on that real quick. A scheduling error caused the lack of daily challenges and the skilling spree yesterday and today on our end, not the No Downtime test itself. If not for that, these would have been active as usual.

However, we currently need DT to activate them, so they will return tomorrow after 11:00 UTC. Good learning and something to investigate for the future

I’ve forwarded the feedback thread issue to the relevant people; there is something weird happening with the settings on that thread.

I’ve located the EBR you sent; we have already a few bug reports on issues that look very similar so we will be collecting them all.

Can you send us a bug report, please, with the details on these NPCs and/or sites; and reply back with the EBR number?

What’s the EBR number of your bug report?

Please note that bug reports and support requests are two different things. If you need help from Customer Support, please visit the Help Center and open a chat or send in a support request. If you are reporting a bug then they are collected in our defect tracking system for the Dev Team, where they will be processed, but we may or may not respond to individual bug reports.

And were you then able to submit again once the 24 hour timer expired? Please send us a bug report.

Hello, everyone,

No Downtime is going well so far. Load on the memory and internal tasklets look healthier than the first time we ran this experiment, and no game-breaking issues have been identified thus far. This is all within the expectations set for the No Downtime Vol. 2!

We want to get as much data, sexy graphs, and learnings from this experiment as possible. Therefore, we will be pushing the daily maintenance by a few hours to get some extra time beyond the 48 hours of service that was initially planned. This will be another important step to gather data that will help drive this initiative closer to the future with less frequent downtimes.

EXPECTED DT DATE & TIME: FRIDAY, 10 SEPTEMBER AT 15:00 UTC
EXPECTED DT DURATION: 20 MINUTES

From a technical standpoint, this is completely uncharted territory for EVE Online. We are genuinely excited to see how our services and in-game chat will continue to perform and this will generate some great insights. The EVE team is well caffeinated and ready to tackle anything that might crop up!

We appreciate your understanding and thank you for your patience. It is amazing that we can conduct such essential experiments on live services together with encouragement from our fantastic community.

You guys are the best o7

At 15:00 EVE Time / UTC.