1/18 @ 8:30pm EST - Huzzah! Helk just let me know they were able to improve server performance by about 30% due to some shifts to autoturrets and HAB saves!
1/18 @ 12:51pm EST - Good news on the performance front - we’re seeing some commits for profiling and physics performance today. Looks like some optimizations are on the way!
12:00am EST - Another week in Rust development and we’re seeing some new electrical components, more work on the Minicoptor, a fresh camera model, and more!
A new sensor has been added which creates up some interesting abilities. The Heartbeat, Breathing, Humidity, and Footstep Sensor looks very similar to a Land Mine, however, instead of blowing people up, it sends a pass-through of the number of people in a 20 meter radius (with line of sight). Simply hook it up to a counter to display how many people are nearby.
It’s also configurable with the hammer, allowing you to include or exclude authorized people. Opening some new possibilities with electricity, this sensor could have uses for monitoring, trap bases, raid protection, and more!
Also added, the RAND switch allows pass-through based on a random number. Each time the SET input receives power, it will roll a true or false to allow pass-through. Although its specific uses may not be immediately clear, this new switch creates interesting abilities for games, trap bases, or just fucking with people.
Tom has posted some new artwork for a detonator on his ArtStation page. It’s not clear exactly how this’ll be implemented in game - but it looks pretty Rusty and cool!
Work on the newest vehicle to take flight in Rust continues this week. We’ve seen progress on animations and poses, along with work on the core functionality of the Minicoptor. Testing in game isn’t quite possible yet as this work is still being done on a separate branch. That said, it does appear as though this new vehicle may make its way into the next update. Stay tuned over the coming weeks for more as this progresses.
New camera model in game
You’ll be a bit more stylish when snapping shots in game of your friends after the next update. The camera model has received a face-lift. Gone is the old Unity asset store model, a new custom design from Tom has made its way in game.
Let's talk about performance
It’s no secret, performance on our servers hasn’t been the greatest recently. If you play on or host a popular server, you know we’re not alone in this. Performance across the board has been slipping for months now, and it’s reached critical mass.
What took so long to say something?
We constantly monitor the performance of our servers and are in close communication with the Facepunch developers when we experience issues. Although I’m ever present to this, I don’t make it a habit of harping on it. It is not my intention to bust Facepunch’s balls - I greatly appreciate what they do and fucking love the game they have created. They are always very responsive when we bring things up, and I give them the benefit of the doubt that they will resolve issues in a reasonable amount of time.
I don’t like to leave the community in the dark as to what is going on though. Therefore, I’ve decided to fully flesh out where things are at, what we’re doing about it, what you can do about it, and where we go from here. This focuses on server performance, but we all know client performance could also use some work.
What’s causing this?
A server will run decently for a period of time, then at some point, performance will take a dip, and gameplay becomes either sluggish or completely unplayable. Very rarely does a server recover fully without a restart - and even then, restarts just kick the can down the proverbial road.
There doesn’t appear to be one single cause of these slowdowns. Rather, it’s likely a culmination of various changes and additions to the game over time that has gotten us to this point. Historically, AI has been a main culprit. Between the cargo ship, bandit town, compound, junkpile scientists, animals, and everything else, there is more AI in the game than ever before. The presence of the cargo ship in particular appears to be a catalyst in these dips. It’s not clear if this event is the main cause of the server slowing or simply the straw that's breaking the camel's back.
We also have electricity as a new part of the equation. Although Helk specifically throttled the amount of electrical requests which can be processed by the server at a specific time, there is definitely some overhead involved with this system. All this doesn’t even touch on the shortcomings of the underlying engine of the game, Unity.
In all fairness, Rust is really pushing the boundaries farther than pretty much any other game out there. Each server has potentially hundreds of people at a time gathering, hunting, building, raiding, and the like. It’s really incredible when you think about it, and the sheer massiveness of the scope is one of the things that makes Rust so special.
By the same token, when every action is delayed by seconds, or you end up dying and losing hours of work due to lag, the magic of the game is quickly replaced by anger and frustration. The very things which make the game amazing actually exacerbate the need for optimal performance.
What are we doing to combat this?
As a server owner, this is disheartening to say the least. When complaints of lag flood our Discord, I feel it in the pit of my gut; I cringe when I read posts referencing the poor performance on our servers.
Our Systems Team and I have been doing everything in our power to remedy the situation. We just purchased two new servers with Coffee Lake processors in our US data center. We also rented new hardware in the AU last month so each server in that region now has its own dedicated box. These are not the first times we’ve thrown more hardware at the situation and surely won’t be the last - we are always on the lookout for better hardware to be available from our providers.
As it stands, we run 23 dedicated boxes to host our 31 Rust servers. All of our larger servers have their own dedicated box to run the single instance of Rust. If you have any experience with hosting servers for other games, you know this is fucking insane.
In addition to this, we are running server restarts much more frequently and have set up special alerts to inform our Systems Team when a server’s FPS drops to single digits consistently. We also, as always, are working with the developers at Facepunch to provide them with any information we can to help identify the biggest culprits of low performance.
What should you do?
If you’re playing in our servers and performance drops consistently for 5 minutes or more, let us know in Discord and our Systems Team will look into it ASAP. Please keep in mind that a server restart isn’t a silver bullet and may not always be our immediate course of action.
If you’re looking for a Rust experience that is as smooth as possible, I recommend finding a very low population community server (of which there are numerous great ones). If you only like playing on high pop servers, you may want to consider taking a break from Rust all together till the performance pendulum swings in the other direction.
If you’re a server owner, I suggestion playing with disabling or limiting the amount of AI present on your server as this tends to be one of the largest performance hogs. Unfortunately, this is not an option for Rustafied servers as it fails to qualify as an official vanilla experience.
Where do we go from here?
The ball is in the developer’s court at this point. Thankfully, both Helk and Alistair have told me they are looking into it this month. Will we see some relief in the February update? That remains to be seen, but I sure as fuck hope so.
Thank you for your patience in this matter. I am a strong believer that the best days of Rust are still ahead of us and I am excited to be here, reporting on development and providing the best server experience we possibly can.
Shifts to water being worked on (including the addition of a wake for rowboats)
More progress on the Oil Rig monument
Work on lighting tweaks to Military Tunnels
No longer force full snapshots when spawning / respawning
Loading screen status tweaks when spawning / respawning
Add WorkshopSkin and MaterialReplacement components to skinned entities in prefab preprocessing to avoid runtime stutter (performance improvement?! Looks like it!)