9 months ago - /u/moxjet200 - Direct link

Hello everyone! I wanted to share some thoughts and information about today’s launch, but before I do I want to thank all of you that showed us some patience and kindness in our first big release. Even though we felt we were prepared and came in with a high level of confidence - even feeling that we over prepared - we were shown that in a real-world situation things can go wrong that just simply do not in simulated scale testing. You can see a post I wrote yesterday about this that has of course “aged like milk” but I still think it’s informative and gives insight into our feelings the day before launch: https://www.reddit.com/r/LastEpoch/s/v87zrzUa3m

Within 20 minutes of launch 150,000 people had joined us. We are thrilled that so many people are excited about what we’ve poured our hearts and souls into and joined us for launch day. Truly and deeply. This did mean that all of our scale testing efforts were immediately put to the test, and unfortunately a service failed in a way that we didn’t suspect, and we immediately went to work to investigate and resolve it. Many people just played offline and this didn’t bother them, but many of course want to play online (me included).

What does that look like? As someone who was not in the games industry 6 years ago, I always wondered and now that I’m on the other side I can share with you all - at least what it looks like in our scenario. Launch day we had our senior engineers, backend team, leadership, infrastructure/server/services providers in the “war room”, which is just a silly name for a zoom/Google call where we monitor and address issues that crop up with all of our dashboards and tooling in front of us. Dashboards showing what’s happening with server connections, timeouts, regional data, player data, databasing calls, etc. People involved are calling out what they’re seeing, potentials of what may be causing a problem and potential solutions, determining if we should go down the route of trying a solution that may take X amount of time and solve an issue or leave us in the same position, etc. Then you have the rest of the internal team anxiously awaiting updates so we can communicate with you all what’s going on as that’s a lot of people who are pretty upset with you and many being quite vocal about it. “War room” makes a lot of sense after you’ve been in it during a launch.

One thing I wanted to ensure today is that if we did have problems, which we did, that we would stay as communicative with you all as possible. I tried hard to do this by keeping a log of updates and posting every 15-30 minutes in our Discord. I’ve been on the other side of this where as a player I just wanted communication - any communication - on what is going on. I can certainly see how large studios struggle with this as not every update is a PR win, but I’d rather stay transparent and hopefully there’s a net PR win by building trust between us and you all, knowing that we’ll communicate and care deeply when something isn’t going as intended.

Tomorrow as we aim to deploy another hotfix, reduce the too-often scene transition times being longer than they should be, and fix any other issues that crop up, I’ll be keeping everyone up to date with a log in the same way I did today. See the #news channel in our discord. https://discord.gg/lastepoch

I will also work with the team to sometime next week put out a more technical retrospective on today as it sounds like many of you are interested in that.

At the end of the day I’m very happy to see so many of you, mainly through Twitch, enjoying the game, even online,after we have things in a somewhat more stable state - and visiting Reddit to see some posts that put a much appreciated smile on my face after 14 hours in the “war room”. I even got to get a few levels on my Multishot Falconer before starting to write this.

Again, thank you all for your patience and excitement for the game. We are excited as heck that we have built a community of this size that we get to create content for, for years to come, and be a studio that you guys can trust from game design to communication and listening to your feedback. I look forward to seeing you all in game!

  • Judd
External link →
9 months ago - /u/moxjet200 - Direct link

Hello everyone! I wanted to share some thoughts and information about today’s launch, but before I do I want to thank all of you that showed us some patience and kindness in our first big release. Even though we felt we were prepared and came in with a high level of confidence - even feeling that we over prepared - we were shown that in a real-world situation things can go wrong that just simply do not in simulated scale testing. You can see a post I wrote yesterday about this that has of course “aged like milk” but I still think it’s informative and gives insight into our feelings the day before launch: https://www.reddit.com/r/LastEpoch/s/v87zrzUa3m

Within 20 minutes of launch 150,000 people had joined us. We are thrilled that so many people are excited about what we’ve poured our hearts and souls into and joined us for launch day. Truly and deeply. This did mean that all of our scale testing efforts were immediately put to the test, and unfortunately a service failed in a way that we didn’t suspect, and we immediately went to work to investigate and resolve it. Many people just played offline and this didn’t bother them, but many of course want to play online (me included).

What does that look like? As someone who was not in the games industry 6 years ago, I always wondered and now that I’m on the other side I can share with you all - at least what it looks like in our scenario. Launch day we had our senior engineers, backend team, leadership, infrastructure/server/services providers in the “war room”, which is just a silly name for a zoom/Google call where we monitor and address issues that crop up with all of our dashboards and tooling in front of us. Dashboards showing what’s happening with server connections, timeouts, regional data, player data, databasing calls, etc. People involved are calling out what they’re seeing, potentials of what may be causing a problem and potential solutions, determining if we should go down the route of trying a solution that may take X amount of time and solve an issue or leave us in the same position, etc. Then you have the rest of the internal team anxiously awaiting updates so we can communicate with you all what’s going on as that’s a lot of people who are pretty upset with you and many being quite vocal about it. “War room” makes a lot of sense after you’ve been in it during a launch.

One thing I wanted to ensure today is that if we did have problems, which we did, that we would stay as communicative with you all as possible. I tried hard to do this by keeping a log of updates and posting every 15-30 minutes in our Discord. I’ve been on the other side of this where as a player I just wanted communication - any communication - on what is going on. I can certainly see how large studios struggle with this as not every update is a PR win, but I’d rather stay transparent and hopefully there’s a net PR win by building trust between us and you all, knowing that we’ll communicate and care deeply when something isn’t going as intended.

Tomorrow as we aim to deploy another hotfix, reduce the too-often scene transition times being longer than they should be, and fix any other issues that crop up, I’ll be keeping everyone up to date with a log in the same way I did today. See the #news channel in our discord. https://discord.gg/lastepoch

I will also work with the team to sometime next week put out a more technical retrospective on today as it sounds like many of you are interested in that.

At the end of the day I’m very happy to see so many of you, mainly through Twitch, enjoying the game, even online,after we have things in a somewhat more stable state - and visiting Reddit to see some posts that put a much appreciated smile on my face after 14 hours in the “war room”. I even got to get a few levels on my Multishot Falconer before starting to write this.

Again, thank you all for your patience and excitement for the game. We are excited as heck that we have built a community of this size that we get to create content for, for years to come, and be a studio that you guys can trust from game design to communication and listening to your feedback. I look forward to seeing you all in game!

  • Judd
External link →
9 months ago - /u/moxjet200 - Direct link

Originally posted by wakasm

I can handle the technical issues, growing pains, I'll play your game when it's ready etc...

...but whatever you've done that has every other person trying to get bees or whatever by sharing codes. Please, just turn that off and give it to everyone and find a better, none intrusive way to get your social marketing on.

It's been such a distraction, and not for the positive.

Hopefully my thoughts on this doesn't send a brigade my way. Excited for the game long-term.

I hear you on this. I created another channel in Discord today specifically for people sharing codes. I’ll get someone on the team to consider solutions to this tomorrow

9 months ago - /u/moxjet200 - Direct link

Originally posted by JRPGFan_CE_org

reduce the too-often scene transition times being longer than they should be

Is that due the server load, so you make it longer to combat it and then put it back to normal once it cools off or is that a bug?

Being transparent, I don’t think we know what’s causing it just yet. It’s not something we saw in CT testing or scale testing but our day today was just trying to get servers/services back online. It’ll be the big focus tomorrow to investigate and address this

9 months ago - /u/moxjet200 - Direct link

Originally posted by MagiusPaulus

Respect for you all, especially with the communication!

One small advice. Discord is used only by the most hardcore of players. If you would have just copy / pasted the updates to Reddit (maybe even Steam discussion forums) it would have been seen by a lot of more players.

Having said that, I can imagine there is a balance between helping solve the problem and communication.

Yea this is sound advice. I’ll chat with the community team tomorrow and make sure that we have a better way to spread those updates to more platforms if they’re important.

9 months ago - /u/moxjet200 - Direct link

Originally posted by Nicetomitja

Is there a reason why the game suddenly has 30 fps less than before the patch? I had no problems with my GTX 1080ti until yesterday. Now the game is unplayable. With the same graphics settings. I can barely get above 50 fps. Apart from the fact that it is still impossible to play online. I think it sucks that the hardware requirements within a game are increased to such an extent by a patch that it is now no longer playable. At least for me. f**k that.

It sounds like you may need to verify your game files which is a Steam option. Performance has improved with the 1.0 updates as we’ve optimized zones and enemies. I think the only thing that went in the other direction was the startup memory consumption just a little bit which we’re working to bring down, but certainly moment to moment gameplay in zones should not have been negatively impacted

9 months ago - /u/moxjet200 - Direct link

Originally posted by Huitzilopoctlan

Hi. Can someone please fix the left mouse button? All it does is click to move and the attack option is missing.

Yes. Known issue that we’ll be hotfixing. Work already underway on that fix

9 months ago - /u/moxjet200 - Direct link

Originally posted by seqhawk

How much fighting was there in your war room? No fighting? A little bit of fighting that needed reminders about no fighting in the war room? All out brawls?

In all seriousness, it was a pretty frustrating launch to not even be able to play at all for indeterminate amounts of time (as opposed to something merely bad like being able to play for a bit and then crashing and then getting back in, or being stuck in a long queue but at least being able to play once through it), but I appreciate the communication, and it was pretty fun at the end of the day when things were more stable.

Happy to say there was zero fighting, and not even a small amount of tension, between team members even with the stressful situation. Truly a great group that is just focused on a common goal together. If anything, we do have an amazing team at EHG

9 months ago - /u/moxjet200 - Direct link

Originally posted by FakeChiBlast

Excellent! I'm having fun so far!

Why does stuff drop on the ground and I can't pick it up? I have no loot filter yet.

Hm that doesn’t sound right. Clicking on the label just doesn’t place it into your inventory? Send us a bug report with the in-game bug reporter. It gives us a lot of good diagnostic data that’s tremendously helpful when solving issues. Thanks for taking the time

9 months ago - /u/moxjet200 - Direct link

Originally posted by poet3322

There's an old joke about exactly this type of thing:

A QA engineer walks into a bar. Orders a beer. Orders 2 beers. Orders 10 beers. Orders 9999999999 beers. Orders 0 beers. Orders -1 beers. Orders a lizard. Orders a wteurgejeyth. 

The first actual customer walks into the bar and asks where the bathroom is. The bar immediately bursts into flames, killing everyone.

lol yes. We were even ordering -1 wteurgejeyths and were feeling confident!!

9 months ago - /u/moxjet200 - Direct link

Originally posted by Enter1ch

Stop excusing and buy the servers!

Even if you dont need them anymore in 1 week because playerbase droped by 50%

It certainly wasn’t a lack of spend on servers. For more insight check the post I linked to from yesterday.

9 months ago - /u/moxjet200 - Direct link

Originally posted by SixSixTrample

Honestly I just want to commend you for giving your team some time off to rest after today.

Obviously it was a long day, but a shitty leader would have been yelling and driving people into the ground rather than recognizing the need for rest.

Everything’s not perfect but no one can say you don’t care about the game, the customers, or your people.

I can’t say enough good things about them. What’s awesome is when you have a team that cares about what they’re working on you don’t have to force them to stay, you have to force them to leave.

9 months ago - /u/moxjet200 - Direct link

Originally posted by Ciri__witcher

Please let me rebind the ‘Y’ button on controller again. It’s bugged after 1.0 and is driving me crazy.

Oh if that’s true we’ll investigate and get that into a hotfix asap. Just posted in the internal channels

9 months ago - /u/moxjet200 - Direct link

Originally posted by Memphisrexjr

I'll never understand this for any game. It happens way too often from indie to AAA studios. If know you sold x amount of copies then why wouldn't you just get extra server space or whatever? I personally would take the hit on spending extra money on servers over lost players, refunds and negative reviews. This game has been funded by Kickstarter, in early access for six years, made money from sale copies, supporter packs and microtransactions. It's just puzzling how you couldn't just throw more money into extra servers for launch.

It wasn’t too long ago that I was just like you, a gamer seeing launch day issues, and didn’t thoroughly understand why so I wrote a bit about it in this post to give people some insight. https://www.reddit.com/r/LastEpoch/s/v87zrzUa3m

Cheers

9 months ago - /u/moxjet200 - Direct link

Originally posted by Henslock

It's very odd, but in a good way, seeing the game's founder be so open with communication, and in such a human way. Thank you!

It’s what I always wanted when I was on the other side as a gamer so it’s something I prioritize highly. You’re welcome and it’s truly my pleasure

9 months ago - /u/moxjet200 - Direct link

Originally posted by stamatov

At the end you did exactly what every other game developer did in the past... just wait for the storm to pass. I mean what else could you do? Buy new hardware... not enough time and don't want to put money into it. Rewrite the code...not enough time and don't want to put money into it. Sadly this was and always will be the "proper" way game developers deal with this. Just wait it out... people will give up, numbers will go down and it will fix itself. In the meantime just post updates and talk to the community or not, it really doesn't matter. The important thing is time to pass and the problem auto solves itself. And hey look, you save your corporation a lot of money in the process... same old... same old

I wish just a passage of time would alleviate these things! Unfortunately it requires a lot of smart and dedicated people’s efforts to address the types of issues we encountered today. If you’re new to our studio and view us as potentially some big evil corporation, this community made video about EHGs origins may be interesting to you. We started as an after hours passion project -> Kickstarter -> early access. https://youtu.be/JjVDwChiJdc?si=kXb0v6sDb8uy_kg9

Cheers

9 months ago - /u/moxjet200 - Direct link

Originally posted by Nimyron

So what happened ? I mean ok a service failed but any chance we get more details on that, like what service ?

Yea I’m going to try to get a technical retrospective out next week. But essentially a container app that hosted our matchmaking service died and we had to recreate the environment and migrate all traffic to the new container. There’s a lot of stuff that happens to match you to servers as there are regional servers, town servers, wild servers, servers that are pre warmed or need to spin up entirely, the list goes on. Then when you have a party it gets even more complex!

9 months ago - /u/moxjet200 - Direct link

Originally posted by garlicbreadmuncher

I still can't play in fully offline mode I keep getting stuck on the first loading screen and it never progresses, have tried every little fix possible and still nothing. I know other people are having this issue too. Is this a known issue and is there a fix? I was totally set on not playing online for a week while things got on track but this is really sad that the offline mode is also unplayable :( for context I have been playing the beta for about 3 years and never had issue like this before.

Hmmm have you verified your steam files? I actually haven’t heard of others having this issue, though I personally was pretty focused on the online issues today

9 months ago - /u/moxjet200 - Direct link

Originally posted by Nimyron

Oh you were not kidding about communication. Most studios go with some vague "this or that crashed", I wasn't expecting a proper answer.

Thanks for the reply and good luck with all this !

Cheers.

9 months ago - /u/moxjet200 - Direct link

Originally posted by Parachuteee

The game has 120k players right now and seems very fluid (I've just played for an hour, don't tell my boss!). So it wasn't the stress of 150k players but a bug that caused yesterday night's downtimes?

Correct-ish. There was an issue that caused a container app to crash which can pretty much be constituted as a bug or oversight. It goes a bit deeper than that but I’ll let our backend team speak to it when we put out the retrospective.

Glad you were able to get in and have some fun

9 months ago - /u/moxjet200 - Direct link

Originally posted by garlicbreadmuncher

Thanks for the reply, I just wanna say that you guys are awesome and all the hard work you have put in to this game and the launch is very appreciated! I hate to be one of those people complaining on Reddit but I wouldn't have come here if I wasn't completely stumped.

Yeah have tried all of those little things, even done a fresh install. Here's what I've tried: Verify files Uninstall and reinstall Restart computer Checked firewall settings Don't have a VPN Updated graphics card driver Created about 6 or 7 new characters always stuck on same loading screen Let the loading screen load for 30 minutes just to be sure it wasn't just slow Tried offline and fully offline, both identical issue Haven't tested in online, Aus que is huge

Have only seen a few others with this issue, and most them only had it online, seems like it's rare... Hope that helps, good luck!

It sounds like it could be some corrupt local character data, especially since you’ve been playing for years. Can you try navigating to your character data folder and backing it up and delete it and see if it resolves the issue?

I’m falling asleep pretty badly so I may not respond after this, but if that doesn’t work please reach out to our support team support.lastepoch.com.

9 months ago - /u/moxjet200 - Direct link

Originally posted by Armi5

Do you know if the Last Epoch banner was a paid marketing banner or Valve chose you and you had to make a banner for the launch for Steam store?

Unpaid but we were able to work with them to put it up as there were no other big releases today. I wasn’t directly involved in that effort though so I’m without much context.

9 months ago - /u/moxjet200 - Direct link

Originally posted by Silverneelse

Thats very good leadership. Have you seen the TED talk from Simon Sinek? I use that all the time in my courses about teamwork (I work at a highschool).

I don’t believe I have. Is this the one? https://youtu.be/qp0HIF3SfI4

9 months ago - /u/LeRoyRobenson - Direct link

Here's the thing. Game looks gorgeous. Tons of additions to 1.0. Everyone is in love with the bones you've made. The other stuff will work itself out. It took ggg a full decade to have stable launches!! The whole team should be proud. I'm excited to play a Falconer when I have some time :)