devtrackers.gg
THE flipping buff bar is broken? you broke the buff bar and instead of hot fixing the issue, u leave it to next week? wtf
External link →THE flipping buff bar is broken? you broke the buff bar and instead of hot fixing the issue, u leave it to next week? wtf
External link →Real answer, because not everything can be hotfixed. Some things require what's called a coldfix, which is where we need to take the whole system offline and reboot all the worlds in order to manage. That's a BIG ask and it's one we have to weigh up really carefully. If the issue is annoying, but not utterly destroying the game, then generally we feel waiting until the next update is best bet. If the issue is gamebreaking, then we'll coldfix and sometimes implement a rollback if we feel we need to.
Where we can, we hotfix. Even if it's where we temporarily add in a little bit of less than perfect code to fix the problem for now whilst we roll out the better fix when we update the game. But this isn't always possible, especially when it comes to things that deal with the client.
And sometimes the fix isn't immediately straightforward and takes time to solve. Ramen and I are currently pulling our hair out over a small bug (in test, not live) with Seedicide. An issue which sounds SOOOO simple, but in reality is nothing short of the dark sorcery of lovecraftian elder horrors (we've still not solved it because it is definitely witchcraft at this point).
Obviously ideally there would be no bugs on launch. But sometimes bugs just appear in the live environment and not in the test environments. Sometimes because of conflicting other updates, sometimes just because the spirits of the code are angry that we haven't performed the right sacrifices (seriously, sometimes bugs just appear and disappear with NO involvement from devs, the code is haunted). There's a reason that lots of programming humour includes things like "99 little bugs in the code, 99 little bugs, take one out and sort it out, 153 bugs in the code".
Someone said the rebooting of servers costs ungodly amounts of money. Is this true?
I'm honestly not that clued up on that part of the process, but I know rollbacks are certainly very costly.
The worst bugs are the ones you can't reproduce and then you have to leave logs everywhere in hopes that it happens and you get some useful information. Hate those!
Oh cthulhu yes, I HATE those!
How can you say that dying because of client crashes is not utterly destroying the game? How could anyone play hcim? Why am I losing mils for trying to do any normal boss because of dying from a game error.
Buff bar doesn't overly cause deaths so idc about that, but the client crashes are unacceptable
Client crashes are much trickier to track down and resolve than other bugs and uploading those fixes may require additional processes that I myself don't understand (not the part I work with).
So this is the power of 15 year old legacy code
In some part, yeah sure, but this is also just the power of code full stop. Sure there are more modern processes that help to limit these issues, but bugs happen in code, it's the nature of the beast and many of these modern processes simply cannot be combined with the runescape set up.
In the past Jagex would frequently do 2-3 coldfixes a week after updates in order to fix bugs. I don't get why Jagex went from doing that to being scared of taking the servers offline for 10 minutes where most people wouldn't notice.
So, realistically, this is turning into a discussion I cannot add to any more. We have done midweek coldfixes in the past, but they're a big deal and require some careful consideration, as I am not part of the discussion in this scenario you're not going to get the in-depth answers you want from me I'm afraid.
Coldfixes require a decision to be made and the costs weighed up. It's expensive (in both time and resources) to shut down and restart the game and shutting down the game disables the game for everyone not just the people suffering the issue. So a call needs to be made and this time I'm not part of that so can't give you any more answers there, sorry.
Just want to say Raven that we really appreciate your answers. There's a lot of misconceptions out there that have gone unchallenged in the past, keep up the good fight.
<3
If the issue is annoying, but not utterly destroying the game, then generally we feel waiting until the next update is best bet.
First off, respect for giving a real answer, but please see the customer's perspective here;
There's people that can't do group PvM like raids and AoD cause their bar gets completely screwed. Losing a streak mid telos cause you can't see the damn virus seems like a huge disruption to the game as well.
People don't want to take a break from the content because of something that's out of their control.
I 100% understand that neither you nor any of the other J Mods want bugs to persist ingame, but when bugs are persistently popping up and you have the choice between a cold fix; taking the game offline for a short while or delaying problems by a week, it just makes people not want to play. It's no wonder the player count is dropping every week, RS3 is under 17K concurrent players a day now, and still dropping.
This is considering the customer's perspective. Most bugs affect a very small percentage of players, so rebooting the servers, which would disrupt every player (kicking them out mid boss-fight, for instance) to fix an issue that is probably not even affecting them, is not desirable. Only if a bug is widespread, serious and not hotfix-able is it prudent to make an unscheduled update to coldfix it. Sure, we have update timers now, but that doesn't catch everyone, and even still disrupts gameplay. You have to weight up all the pros and cons for all players.
Ramen and I are currently pulling our hair out over a small bug (in test, not live) with Seedicide.
I've never even heard of any issue with seedicide. It might be nice to see more of these issues that you guys are fixing but no one ever hears of, and not just on your own personal twitter accounts - in a more compiled place. Maybe it'll be listed as a patch note? But even then, the patch notes never really describe or show what the issue was, they tend to just say an issue was fixed, which doesn't say anything about how difficult it was to find/fix, what was causing it, who all contributed to try to fix it, etc. I think it'd be pretty neat to see an extended version of the patch notes, or a page with further descriptions of each of the patch notes. Surely most of this is already tracked in jira, so basically just exporting the jira task for that bug into a public description after it's been fixed.
It might also be nice to do something similar for in-progress bugs too, so people aren't constantly asking what's going on with the buff bar, for example - they could check a page to see the current progress on that task without having to be following every jmod on twitter and try to figure out which one of them is working on the particular bug they want info on.
Well the issue wasn't a live one, it was with the content we're currently working on. So that would be why you weren't aware, I was just using it as a reference.
Honestly we don't have the time to write up that sort of detail for patch notes. What we write in jira would need to be edited, sanitised, have the code specifics removed and then rejigged to make it player facing. The time taken to do that for each bug would rack up really quickly and we'd be losing the ability to actually fix bugs or create new content to do it. So I don't think it's something we're ever going to do.
Out of morbid curiousity, what is this Seedicide bug that torements you so?
It was a really rather dumb one as it turns out. Checkboxes were not working properly for the new seeds we'd added. Something so simple on paper, literally the little ticks were not appearing on the screen.
Took 3 devs, several hours of debugging and pouring over the code, before we realised that there were duplicate entries that had snuck into a data file... that's it. Duplicate entries that our code addled brains had filtered out when we looked over it.
I may have screamed a little.