Innovative Leisure

Imagine my delight at working with some of the people who created the iconic video games of the 1970s and 1980s. People like Ed Rotberg,1 Owen Rubin,2, and Ed Logg.3 These guys have more experience making games than just about anyone else. Wow.

But they wanted something besides just games; they wanted a whole service they could link multiple games to, even across platforms (with iOS and Android being their first targets). They wanted turn-based play, but in ways that didn’t quite fit the conventional turn-based models supported by Apple or Google. And they were aiming for large numbers, so the entire thing had to be able to scale up.

I like challenges.

Platform Choices

When I started with IL in the fall of 2012, they already had a proof-of-concept server implementation in PHP. This code was never intended to be production-level code, and I needed to be able to move quickly; the question I needed to settle on was whether to stick with PHP (with all its quirks) or switch to Python (with a completely different set of quirks). For this project, at this time, I chose Python;4 Django, MySQL, and Apache then fell into place. We ended up using Apache only during our early development work, and switched to nginx as our platform grew.5 We’re still using MySQL but not wedded to it.

As a small company, we didn’t want to spend a lot of capital on dedicated server hardware and co-location, nor did we want the distraction of maintaining our own hardware. Aside from an internal development server, we deployed our testing and production systems to Amazon Web Services as a way to get started and have room to grow. Nothing precludes us from switching to buying our own equipment later and setting up data centers, but these days that’s an option rather than a requirement, and I couldn’t be happier about that. I can manage servers just fine, but if I’m going to build a lot of stuff on my own for a while, I’m quite happy to have someone else mind the hardware aspects and let me focus on the part of the solution that’s my own contribution: the software.

Peculiar Challenges

Every interesting project has something about it that’s unique; that’s why it’s interesting. In this case, the interesting part is that although the platform is supporting turn-based games, they are games in which each player submits a move on each turn. That is, for a two-player game, each turn has two moves (one from each player); for a four-player game, each turn has four moves. And those moves can be submitted in any order. This makes the approach used for simple alternating-turns games completely fail; you have to do a lot more database locking to make this kind of thing work, and trying to get performance from a database that has a lot of locking going on is not really that much fun.

To figure out exactly how much database horsepower we were going to need, I wanted to test as close to a real-world scenario as possible. So I actually wrote, in Python, a robot game client that would connect to the server, invite other robot players to matches, and play those matches. Then I set up a cluster of servers to run the robot players and benchmarked the performance of the platform servers, increasing the load until I broke the platform. Then I identified the bottlenecks, fixed the code, and ran the tests again.6

Through several rounds of this testing we determined that our SQL tier was going to become a bottleneck first, and so even though we needed solid data integrity to avoid corrupted or hung games, we were going to have to do something to work around it. We migrated a lot of data out of the SQL tier into a Redis database (some via transparently caching it in Redis with some clever Django classes) and more by moving the locking process into Redis, too. With this setup, the SQL tier still proved to be the first choke point, but at a much higher capacity.

Client-Side Code

Rarely have I worked solely on server-side software, and although I started this project thinking it would be, it turned out not to be the case. I had only started with Unity3D and its JavaScript implementation, so I had very little experience with it directly, but one of the game teams (Draconus Rex) needed a client-side implementation of the service API I had built, and they didn’t have anyone readily available to do it. So I gave myself a crash-course in C# and built one.7

Unity3D is a very powerful tool, but it’s not perfect. It has a good number of quirks that require clever work-arounds. This becomes especially evident when you attempt to support multiple platforms. In theory, switching to a different platform is as simple as choosing it from a list and clicking “Build and Run” but of course in practice it isn’t quite this simple. At least, it isn’t when your platforms are iOS and Android. For iOS I ended up writing a couple of plugins; one to store account credentials in the secure keychain area, and another to directly access native iOS social functions for Twitter and Facebook. Those clearly don’t exist on Android, so alternative implementations had to be written there.8 9


Time and time again I have gone through product launches where we celebrate, and then the inevitable question comes: how well are we doing? Answering this question requires that you have both data collection and reporting already in place, but for some reason it seems like this piece of a project is never accounted for in release schedules, so it never seems to be ready from day 1.

In the case of Draconus Rex, we had most of what we needed already included within the application when it released, because so much of what it does is interacting with the server, leaving a data trail. There are analytics tools you can build into an app that will periodically push data out to remote servers, reporting on every little thing a user does within your app, but in our case we already had that data; we just needed to summarize it into a useful form.

Of course you still need to know what questions you’re trying to answer. The most interesting thing about metrics is that when you get your first set of answers, you typically discover you were asking the wrong questions. This results in a lot of iterating on the reporting, and with Draconus Rex it was no exception. The trickiest metric is “retention” which aims to represent how well an app remains interesting to a player over time. And the longer an interval you want to measure, the longer the time lag between making changes and measuring them. After our initial beta release and metrics analysis we saw how users weren’t staying as engaged as we wanted, so we were able to update the game to address those weaknesses, and we wouldn’t have known without the detailed metrics.

1 Battlezone

2 Major Havoc

3 Asteroids, Centipede

4 I toss aside a small saga in a single sentence. This choice is far from trivial and execution performance really wasn’t a major consideration. Mostly I was concerned about stability, security, maintainability, and the ability to hire really, really good programmers to work on the project. Some of these aspects worked out and some didn’t. But of all the platform decisions, this is the one that is hardest to undo later.

5 Apache is a workhorse and nginx is a racehorse. We still used Apache for our developers’ testing server but nginx has a much more predictable memory consumption profile, which is vastly preferable when running production systems.

6 This kind of testing isn’t cheap but it’s far, far less expensive to do on AWS than it is when you’re working with purchased hardware.

7 Unity3D allows you to write game code using JavaScript or C#. At the time I started dabbling with Unity I had never used C# (I am not really a Microsoft guy) but I had used plenty of JavaScript, so for experimentation that was fine. But here we were using it for “real” code, and C# is a much more robust language.

8 When building for iOS there is no option to access the host operating system except by writing an Objective-C component and interfacing it to Unity. The most challenging part of this was actually getting our build process to work without interfering with the other plugins we were trying to use; Unity does not provide plugin authors with robust mechanisms for modifying Xcode projects without stepping on each others’ toes. The Unity plugin scene is thus a complete mess.

9 For Android Unity3D actually has several options for accessing the host operating system, including allowing you to manipulate Java classes and objects directly from C# code (albeit at a performance penalty) rather than having to write a big blob of Java code and merge that into the project. Since most of what we needed was available with a few simple calls, we did this directly from the C# code without going to the trouble of writing a full plugin.

Photo Credits: snapshot of Draconus Rex running on iPad and Nexus 4; game: Innovative Leisure, image: Damien Jones