Hey all, I’ve posted a bit on Discord already, but I wanted to put this in one place we could refer people to as well. This is a bit of a blog on the technical issues that have come about since releasing 7.0.0, what we’ve been doing to correct, and what still needs to be done.
Overview: How does everything work with the backend stuff I keep hearing about?
Right now, we have two backend servers you communicate with. One is a newer once that will eventually replace the old one. This is in addition to communicating with Steam and the servers that run the actual simulation of things like the lobby. Both the backend servers take advantage of certain steam authentication tools to make sure everything is on the level. These tools have rate limits however (about 1 use per minute), so when you first boot up the game and see the grey connection dot, don’t panic! It can take 2 to 3 minutes to finish connecting to the new server since we give the old one priority most of the time. Another thing to note is the old backend used this tool a lot, and due to the rate limit this can sometimes get in the way of transactions until everything is finished connecting (about a minute after). most of the time this isn’t an issue, but if you boot the game and then immediately try to sell something, it won’t go through for a while, again give it a minute after the green dot shows up and everything should work as normal.
What’s been happening and what we’ve been doing
There’s been three major issues since the release, you’ve probably all felt the crashes, but there’s also been optimization and resource issues on the new backend system we’ve been working on. Unfortunately, these things weren’t distinct, they were all linked to the new backend system or achievements in some way. The first is unoptimized encryption. While the old system does take cryptographic measures to secure transactions, payouts, and more, the new system goes a bit further setting up an encrypted stream between you and itself which allows us to process things using less resources and much more quickly and securely over the course of play session, though this comes at the cost of a more complicated “handshake” so to speak between you and the server.
These handshakes took much longer than our previous tests lead us to believe and kept the server from handling other things in a timely manner such as achievements, so our initial guess was that the crashes and the resource issues were somehow linked with the backup of tasks. After a bunch of math, we got the handshake to be under a second in the average case, but the crashes and resource leaks continued.
The task then changed to investigating the client itself for the crash, but it took a bit to track down due to the nature of the bug. Unreal uses what’s called a Garbage Collector to clean up memory that’s no longer used. Normally this is fine, but if you don’t take certain steps to let it know that you’re still referencing something, it can delete the memory out from under you and you’ll not be any the wiser. This can lead to things error’ing in strange ways that are hard to pinpoint, and often it’ll crash in different places. As you might have guessed, the resource deleted was integral to how your client sends information regarding achievements to the backend.
The final issue, is a resource leak on the new backend that causes us to force a restart after it runs out. Basically, whenever people are using it, certain things are being returned to the OS when the program is done using them, and it slowly hoards more and more until it breaks. This is almost imperceptible with a small number of test connections. When you have 700+ people hitting it at once (we’re happy to have this problem don’t get me wrong 
), this resource hoarding happens very quickly.
Where are we at right now?
Currently we’ve fixed the first two issues, with the crash fix having come out as I’m typing this. We’re still investigating the resource issue, after which we shouldn’t need to restart the backend all the time (before the crash fix this could sometimes crash clients, sorry!) You should also have less issues with payouts and transactions. However, we’re starting to think their might be a 4th issue, relating to transactions and payouts separate from the other problems. It’s something we’ll be looking into as we try to track down the leak. Getting the game back at 100% is top priority for the tech team right now. I’ll be checking this thread for a bit so if you have any questions or concerns please comment down below 
