What's Been Going on Since 0.7.0.0 Released

Hey all, I’ve posted a bit on Discord already, but I wanted to put this in one place we could refer people to as well. This is a bit of a blog on the technical issues that have come about since releasing 7.0.0, what we’ve been doing to correct, and what still needs to be done.

Overview: How does everything work with the backend stuff I keep hearing about?

Right now, we have two backend servers you communicate with. One is a newer once that will eventually replace the old one. This is in addition to communicating with Steam and the servers that run the actual simulation of things like the lobby. Both the backend servers take advantage of certain steam authentication tools to make sure everything is on the level. These tools have rate limits however (about 1 use per minute), so when you first boot up the game and see the grey connection dot, don’t panic! It can take 2 to 3 minutes to finish connecting to the new server since we give the old one priority most of the time. Another thing to note is the old backend used this tool a lot, and due to the rate limit this can sometimes get in the way of transactions until everything is finished connecting (about a minute after). most of the time this isn’t an issue, but if you boot the game and then immediately try to sell something, it won’t go through for a while, again give it a minute after the green dot shows up and everything should work as normal.
Capture1

What’s been happening and what we’ve been doing

There’s been three major issues since the release, you’ve probably all felt the crashes, but there’s also been optimization and resource issues on the new backend system we’ve been working on. Unfortunately, these things weren’t distinct, they were all linked to the new backend system or achievements in some way. The first is unoptimized encryption. While the old system does take cryptographic measures to secure transactions, payouts, and more, the new system goes a bit further setting up an encrypted stream between you and itself which allows us to process things using less resources and much more quickly and securely over the course of play session, though this comes at the cost of a more complicated “handshake” so to speak between you and the server.

These handshakes took much longer than our previous tests lead us to believe and kept the server from handling other things in a timely manner such as achievements, so our initial guess was that the crashes and the resource issues were somehow linked with the backup of tasks. After a bunch of math, we got the handshake to be under a second in the average case, but the crashes and resource leaks continued.

The task then changed to investigating the client itself for the crash, but it took a bit to track down due to the nature of the bug. Unreal uses what’s called a Garbage Collector to clean up memory that’s no longer used. Normally this is fine, but if you don’t take certain steps to let it know that you’re still referencing something, it can delete the memory out from under you and you’ll not be any the wiser. This can lead to things error’ing in strange ways that are hard to pinpoint, and often it’ll crash in different places. As you might have guessed, the resource deleted was integral to how your client sends information regarding achievements to the backend.

The final issue, is a resource leak on the new backend that causes us to force a restart after it runs out. Basically, whenever people are using it, certain things are being returned to the OS when the program is done using them, and it slowly hoards more and more until it breaks. This is almost imperceptible with a small number of test connections. When you have 700+ people hitting it at once (we’re happy to have this problem don’t get me wrong :smiley: ), this resource hoarding happens very quickly.

Where are we at right now?

Currently we’ve fixed the first two issues, with the crash fix having come out as I’m typing this. We’re still investigating the resource issue, after which we shouldn’t need to restart the backend all the time (before the crash fix this could sometimes crash clients, sorry!) You should also have less issues with payouts and transactions. However, we’re starting to think their might be a 4th issue, relating to transactions and payouts separate from the other problems. It’s something we’ll be looking into as we try to track down the leak. Getting the game back at 100% is top priority for the tech team right now. I’ll be checking this thread for a bit so if you have any questions or concerns please comment down below :slight_smile:

46 Likes

Thanks for being transparent. Appreciate all of your work on getting this fixed! :blush:

5 Likes

It’s amazing how you guys figured all of these problems out and fixed them in a matter of days!
Keep it up, we are counting on you.
IMG_20190312_122120

11 Likes

I appreciate the communication and explanation here, and I’m excited for things to get back into ship shape.

Thanks for all the hard work devs! It’s honestly kinda amazing how quickly these major issues are being resolved :stuck_out_tongue:

Sweet! I think you guys should post this to the Steam news section, too. With the influx of new players some might not use the forums

5 Likes

I love unexpected Obi-Wans so much

10 Likes

They’re surprises to be sure, but welcome ones.

3 Likes

A small update: after investigating the payout/transaction issues we decided it would be faster to simply make the updates to move everything away from the transactions old server than to fruitlessly try to get the two to play nice. I’ve been working on this for the past day or so and am hoping to have a fix ready for testing tonight, if not tomorrow. I’ll post another update with an estimate on when we’ll be pushing the fix out once we’ve run some of those tests (I don’t want to promise Friday night, for example, only to have the test build fail for some weird reason).

17 Likes

Update Transactions and Payouts

Currently testing transactions and payouts fix. They have been moved to the new backend. There’s still a couple things on the old backend we’re relying on, so it’ll still take a couple minutes to startup as the servers fight over the aforementioned authentication tools, however as long as you’ve got the green dot you should be able to buy, sell, and receive unit payouts once we push a hotfix.

Coming Soon

Moving forward there’s still some stability issues I need to work out with the new backend (you’ve probably experienced the restarts) as well as improving reliability of unit payouts in the cases that outstanding circumstances causes a payout to fail partway through. There’s also some things we want to do to collate certain steam requests to help reduce overall server load and improve average response time (Steam will ignore you for a bit if you get too spammy).

I appreciate everyone’s patience and understanding as we’ve worked this out. Once we’ve got a couple more things done we’ll do a production build of the hotfix.

22 Likes

Update for Sunday the 17th:
We had a bit of a crazy weekend here, making the rush to move the transactions and payouts off the old server. Saturday night we had pushed a fix out, but then when we had to take the servers offline again. This was because an exploit had been found and people were able to get ridiculous amounts of units, we spent that night and most of the next morning fixing that and some other issues.

Today we woke up to server connection issues, essentially with the added load from transactions going out to valve and our own databases the server was dying. It would then restart and everyone would connect at once, bog it back down, and so on and so on. We spent today expanding our backend’s capacity as well as lowering the number of calls that need to be sent from the client to our servers. There was also an issue with people receiving milestone rewards that shouldn’t even be available yet, so we fixed that up at the same time. We’re continuing to monitor for errors and load issues, and as I’m typing this we’re finishing up testing and uploading both client and server hotfixes.

15 Likes

Example of said item
image

10 Likes

Also should mention all the items granted will be removed from the user’s inventory and they won’t have any representation in game.

5 Likes

that looks good, i like that

2 Likes

Backend servers are back online. We’ve added additional servers to help load balance and we’ve pushed out an update.

7 Likes

Another update, just to keep everyone in the loop. We currently have three backend servers handling all players right now. The backend servers will restart every 3 hours for 10 minutes to keep things stable (we’re still working on improving this). After the quick restarts, the servers will sync your progress with achievements and EXP.

We have been monitoring things all day and it seems much more stable now. We can add more servers if we need to without needing to push updates out and we can keep working on the backend without players really noticing that much, which is a great point to be at.

We’re very sorry about the issues in the past couple days, but this is an important step in the game and this new backend system was very much needed and the band-aid had to be torn off at some point.

22 Likes

Just to be clear, you mean you work on eventually not having to reboot them as a part of online operations, right?

Yes, he mentioned that we’re working on making them not have to do that.

3 Likes