Memory leaks in Adobe AIR Mobile / AS3 project
As many Adobe Flash / AIR / AS3 projects we have been using Event Dispatcher. This concept seemed to be too uncontrollable and lacking of many wanted information. Doubts about garbage collector made us to recreate the idea to take more of control and information about event listeners.
Keywords: Garbage Collector Which Sucks, Event Dispatcher, Event Bus, Event Listener, Object Pool, Disposable Objects
Story
We were developing a mobile game in company. Some day QA person noticed that the game crashes after long play (1 hour or so). In a short time after some profiling (using Adobe Scout) it was pretty obvious what was cause of it. It was constant memory leak. But why? Where did it came from?
Our general research was started by looking at GPU textures managememt. They were loading, reloading, unloading, everything seemed to be alright about GPU. The real problem appeared around numbers described “ActionScript Objects”. Most of them were OUR objects. What gives? Garbage collector doesn’t work?
Actually it happened. There is a plenty of sources talking about this problem.
Cleaning
What we needed to do was to just clear all connections between objects by “nulling” them. That was very rare to do so in our project which was about 100k+ lines of code then. In next 3 weeks I did some basic cleaning by implementing IDisposable interface in most places in the game next to our so-called “engine” (which seems to be sort of Starling equivalent).
interface IDisposable {
function dispose():Boolean;
}
Meanwhile, as the work was moving forward, many bugs appearead because of IDisposable implementations. Most of objects were “disposed of” (have nulled all reference type fields) and some of them gave us lessons. For example, there were some tweens (animated properties of objects) which were called even if some objects were not visible any more. Calling change of properties (class fields) or some “onEndCallbacks” caused many NullReferenceExceptions.
Those were to be fixed anyway but we couldn’t suddenly make disposing of all objects due to our game’s architecture. Error #1009 was everywhere. Nulling variables from references achieved some results - we have found some bugs, memory leaks were drastically rarer.
And now… Pool
Memory was more safe but in then game started to have worse performance. Garbage collector was clearing more objects because of IDisposable but it took more CPU cycles to perform deallocations. Solution seemed to be simple - implement Object Pooling - a concept of reusing frequently created objects.
It looked like this:
// Two ways to achieve new Signal object (the latter is faster)
var signal1:Signal = Pools.getPool("badass.engine.Signal").obtain() as Signal;
var signal2:Signal = Signal.pool.obtain();
// Two ways to free Signal object (the latter is faster)
Pools.free(signal);
Signal.pool.free(signal);
And Signal class with statically accessible pool:
public class Signal implements IDisposable {
private static var _signalPool:Pool;
public static const pool:Pool = new Pool(__instantiateSignal);
private static function __instantiateSignal():Signal {
return new Signal();
}
...
}
Object pooling made our engine creating less objects. It created other bugs because of object reuse. Later on we had to take care of all of those bugs due to pooling and nulling but still it wasn’t enough for memory leak issue - it was still there.
Object tracing
The actual problem was that I knew that there must exist many dependencies, so I started to think how to discover them.
So I decided to implement object pooling** **with tracing. I have implemented it with a static access. That mechanism gave me much better understanding about game objects creation and connections between them.
// Obtain pool and some object (save ownership!):
var pool:Pool = Pools.getPool("badass.engine.Signal", true);
var signal:Signal = pool.obtain() as Signal;
For example, some Window created many too many Tweens per second, which weren’t needed at all - and I discovered this one thanks to my new tool. Actually I could free all tweens for my window:
// Obtain pool and some object (save ownership!):
var pool:Pool = Pools.getPool("badass.engine.Signal", true);
var signal:Signal = pool.obtain() as Signal;
// alternatively:
Pools.freeAllOfOwner(this);
I have implemented pooling for Tweens, Clips (animations), Masks (needed for clipping in engine) and Signals (object-based Observer pattern implementation, used sometimes instead of events for performance). There were more pooled classes but those listed appeared to be most informing.
Those implementations gave me numbers about lost obtained objects but not full information. Often I didn’t know where objects were obtained from pool. I decided to implement stacktrace tracing.
public class DebugUtils {
public static function getStackTrace(shortenFilePaths:Boolean = true):String {
var stackTrace:String = new Error().getStackTrace()
stackTrace = cleanUpStackTraceString(stackTrace, shortenFilePaths);
return stackTrace;
}
[Inline]
public static function cleanUpStackTraceString(stackTrace:String, shortenFilePaths:Boolean = true):String {
stackTrace = stackTrace.replace("Error\n", "").replace(//g, "{anonymous}");
if (shortenFilePaths) {
var pathRegex:RegExp = /\[[a-zA-Z\/:0-9$\\_\.]*[\/\\]([a-zA-Z0-9.]+):([0-9]+)\]/;
var match:Array = null;
while ((match = stackTrace.match(pathRegex)) != null) {
stackTrace = stackTrace.replace(match[0], '[' + match[1] + ':' + match[2] + ']');
}
// remove first line which contains this function.
stackTrace = stackTrace.substring(stackTrace.indexOf("\n") + 1);
}
return stackTrace;
}
}
Function Pool.obtain() called DebugUtils.getStackTrace() and saved it somewhere inside the Pool. Later on I could read all stacktraces. Huge help.
Event Listeners -> Event Bus
After implementation of object tracing through object pooling, memory leaks were lower, performance was better but still it was not enough after a long play. In our framework we have many Flash concepts remade to our own classes - such as Stage, DisplayObject, Sprite, Clip etc. And there was EventDispatcher among these.
The next step was to change EventDispatcher to thing which I called Event Bus. Why? Flash-ish EventDispatcher is a base class for DisplayObject. Calling dispatchEvent(type) was easy and it could be called in every DisplayObject/Sprite derived class. Other objects could call addEventListener(type, function). In previous steps I have noticed that some functions were not unregistered through removeEventListener(type, function). It became clear that some objects are still connected through it’s listeners to other objects. And that’s why I decided to make Event Bus - statically accessible place to register event listeners and to remove them.
First, I have changed Event Dispatcher to call Event Bus and added some functions:
public class EventDispatcher {
public function EventDispatcher() {
}
public function dispatchEvent(event:Event):void {
EventBus.dispatchEvent(this, event);
}
public function hasEventListener(type:String):Boolean {
return EventBus.hasEventListener(this, type);
}
public function addEventListener(type:String, listenerOwner:*, listener:Function):void {
EventBus.addEventListener(this, type, listener, listenerOwner);
}
public function addEventListenerForOnce(type:String, listenerOwner:*, listener:Function):void {
EventBus.addEventListenerForOnce(this, type, listener, listenerOwner);
}
public function addSignalListenerForOnce(type:String, listenerOwner:*, listener:Function):void {
EventBus.addSignalListenerForOnce(this, type, listener, listenerOwner);
}
public function removeEventListener(type:String, listener:Function):void {
EventBus.removeEventListener(this, type, listener);
}
public function removeEventListeners(type:String = null):void {
EventBus.removeEventListeners(this, type);
}
public function removeAllEventListenersForOwner(listenerOwner:*):void {
EventBus.removeAllEventListenersForOwner(listenerOwner);
}
}
Thanks to this I didn’t have to change code for registering/unregistering event listeners in whole game. It could make produce conflicts in git and I really didn’t want to deal with those.
What did I achieve this way?
- all ever existing (non-flash) event listeners were in one place - static collection. I could see them in the debugger.
- I could trace when and by which object were listeners connected
- I could retrieve a list of all objects that were owners of functions which were connected to one chosen object
- I have statistics - how many listeners and dispatchers are currently existing in the game
All those features gave me a huge tool which was later used this way:
- CTRL+SHIFT+R to save current time
- open some window in the game
- close this window to go back
- CTRL+SHIFT+T to get all new listeners since step 1.
- Profit!
That’s how I have found more leaks in the game. It showed my listeners that were not cleared. I could make a breapoint in EventBus.update() (which didn’t do anything precious) to just see where are those listeners connected and who owns them (listeners’ owners).
Summary
Thanks to this whole research I was able to trace many objects in the game which helped us to reduce memory usage, memory leaks and crashes. There was even more - we also could fix more bugs due to calls of old listeners (duplicated listeners).
All that wasn’t really planned, it became a step by step solution. To summary those steps again - whole this evolution looked like this:
- game is crashing. Seems that doesn’t deallocate memory. Let’s null everything not-used - implement IDisposable.
- memory problems still existed but largely reduced. Garbage collector needed too much CPU cycles to deallocate memory - implement Object Pooling to not create objects so often.
- Some objects were not pushed back to pool. Added tracing to pooled objects.
- We have discovered that it may be because of event listeners - reimplement derivable Event Dispatcher to Event Bus. Event tried to put Pooling into Event Bus but that seemed to create too much in-game bugs.
Big lesson was learned - use memory profiler on your game!