DEV BLOG #165 - DEBUGGING, DEBUGGING & DEBUGGING
9 of February, 2018
This week all our programmers gathered together to take apart Crest's code piece by piece to find what's causing the random crashes. We finally did! But the journey there was rather adventurous as you'll find out below.
We dropped the price of Crest to $9.99 this week and started what is definitely not a cult. Find out more about The Inner Circle.
We will have a new patch next week but want to make sure first that everything is running flawlessly before we publish it. A good thing about all the debugging is, that we've been able to polish a lot of the code while hunting for the cause of the crashes.
Look, I'm sorry that my blog posts have been so cryptic for a while but that changes soon. Here's a last tease of what I'm doing.
This week all programmers including myself have been trying to fix the crash that keeps happening after playing the game for a while.
After working on it for a few days, we managed to fix a few major bugs having to do with object pooling, but the crash was still happening. We finally realized that the crash started happening when we upgraded to Unity 2017.3. We tried going back 5.6 which is the last version we used before upgrading, and the crash was gone. I tried a few versions and figured out that 2017.1 is the last version where the crash isn't happening, so that's what we've downgraded to now.
A very frustrating realization, but this weeks efforts have all been worth it anyway. We've fixed a lot of memory issues, and built some tools to help us catch problems faster in the future.
The name of the game this week has been debugging, debugging, and more debugging, and its been a journey I can tell you. The crashes have been our top priority and I have looked at more crash logs this week than I think I've done in my lifetime.
At first we could only narrow it down to being a memory issue, the crashes would occur at any place in the code where new objects where created. This lead us to believe that somewhere a loop might be creating a lot of objects during the same frame, but after some testing that didn't seem to be an issue. Looking at the memory usage over time showed a slow increase, building over time. This suggests a memory leak was occuring somewhere. So we started to look for references in static instances that was possibly keeping objects from being garbage collected. When we actually found some cases of this in our pooling of objects, we were feeling hopeful that fixing them would solve the crashes. But nope... Still crashes after a few minutes.
In a last effort we went back to the engine version where the game wasn't crashing, and lo and behold, no crashes! Apparently, there is a problem in our current engine version of Unity. In the end, we did fix a lot of issues in our own code so the time spent was definitely worth it, and now I can finally go back to working on completing the features in the game.
I completed fixing the bug where followers are overperforming and I made a change in the game mechanics. Each follower's action, like farming, picking berries, or eating their neighbor, will now have a fixed amount of stamina cost. Before, it was more dynamic but more prone to be bugged as well. For example, in some cases, a follower could do more actions than they had stamina for. This new mechanic I added will hopefully make designing and balancing more controllable without any unexpected bugs.
I have started adding Steam achievements to Crest now. We will see how long it will take, but I think the main functionality will be implemented early the next week. The goal is to make it easy for us to add or change achievements without too much effort!
This week has had it's ups and downs. I have been working closely with Johannes (our lead programmer) with creating tools for monitoring used memory in Crest. We then used the tool to find memory issues that could lead to Crest running out of memory. Thanks to the tool we have made some huge improvements. For example ending a game and returning to the main menu caused some memory to be stuck in limbo forever. This is now fixed.
Even though we made alot of improvements and tried to locate the main cause of the 'out of memory' error it still occured. It was a tough nut to crack but we finally found the problem. We are now working with correcting it and polishing up on some changes we made.
This week the majority of the team has been focused on finding and fixing the source of the crash bug we have suffered for a few weeks now. I have been keeping track of our progress on that and I have also been preparing for GDC this March.
Have a nice weekend!