It still seems like yesterday when I announced the initial alpha of War Worlds is available on the Play Store. But it's actually been over one and a half years! In fact, it's been over three years since I started the project back in 2011! But, sadly, I feel I can no longer give the project the time and attention it requires and we must close our doors.

It's been an awesome time, and I've certainly learned a lot. Some highlights for me:

• My initial post about the game on the r/IndieGaming subreddit is where I got around 100 initial players, the game has since been downloaded from the Play Store over 45,000 times!
• At it's peak, War Worlds had over 2,000 active users!
• Some of my posts about the game reached the Hacker News front page, not once, not twice, not event thrice, but four times!
• I came to the attention of some Facebook employee for that last post about advertising, and they helped me tweak my campaign a bit. In fact, my post about App Engine also came to the attention of some people within Google as well. All in all, it seems like getting on to the front page of Hacker News is a pretty big deal.
• Perhaps a low-light, rather than a high-light, but getting hacked last month was actually kind of an honour. Perhaps that's when you know you've really gone mainstream...

Now, I still have about a week and a bit left of time before the billing cycle for my server resets, so I plan to keep the game running until 30th August, at which point, it will be shut down.

If you're interested, I've also made the complete source code for the game (both client and server) available on GitHub: https://github.com/codeka/wwmmo.

What's next for me? Well, I'm not really sure. I've got plenty of ideas for new games, but I think I'm going to take things easy for a month or two and take some time out with my family.

Thanks a lot to everyone who has made this game so special, it will always have a special place in my heart!

Below are instructions on how to get the code for War Worlds, how to build a local server, get it up and running, and also build a client that will connect to that server.

Prerequisites

You will need the following installed on your machine before you begin:

• Eclipse + Android SDK
• Postgresql (I use 9.3, but any recent version should work)
• Java JDK 1.7+

Getting the Code

The code is available from Github: https://github.com/codeka/wwmmo

If you plan on contributing code back, I recommend you start by forking the repository by clicking the "Fork" button on that page. For the purposes of this walkthrough, I'm going to assume you haven't done that, but if you do, simply replace all instances of "codeka" with your username below.

$ cd ~/src
$ mkdir wwmmo && cd wwmmo
$ git clone git@github.com:codeka/wwmmo.git code

This will take a while as it downloads the code, but once complete you'll be ready to create your database.

Creating a Database

I'm sure everyone who plays War Worlds regularly will have noticed that there was a bit of an "incident" over the last couple of weeks. This post is basically a quick postmortem of what went down, and what I've learnt from the experience (tl;dr: a lot!).

Background

Before we get started, first a little background about how the game actually works. The server-side is basically just a web server, that receives HTTP requests from clients. Everything is stored in a PostgreSQL database and the web server provides a layer of business logic on top of that database. The actual request/response is encoded using Protocol Buffers, which are an efficient binary encoding scheme (at some point, I'm thinking of also switching to SPDY instead of HTTP as well, for extra efficiency).

So for example, when you go to build a fleet of ships, say, the details of the request (such as the design you want to build, number of ships and so on) are serialized in a protocol buffer, and the request is posted to https://game.war-worlds.com/realms/beta/buildqueue.

Now, anyone who's ever built a web form will know, it doesn't matter whether you validate all your inputs in JavaScript in the form, you still need to validate them again on the server-side, because anyone can build their own form to post to your server. Well, the same is true here. Even though all inputs are validated in the game client, they also need to be validated by the server as well.

Generally, I think I do a pretty reasonable job of that, but I've been a somewhat lucky to date, since it's slightly more difficult to directly post requests to the server outside of the game client compared with a web form. Slightly more difficult, though not impossible!

Enter Proto Baggins

Proto Baggins is a game developer himself, he works for a major game company. So reverse-engineering the protocol buffer-based protocol is a pretty straight-forward affair. Initially, he was just using it to automate some tasks (presumably builds and expansions).

In addition to the automation, he also built some interesting tools to export visualizations of the game. For example, here's a visulation of his empire:

This is actually a pretty interesting visualization, since it shows the poisson distribution of stars pretty well (that's the algorithm I use to ensure stars are placed randomly, but never too close or too far apart). I go into the details of that algorithm a bit in this old post from a few years ago.

Here's another visualization Proto made, this time showing the entire War Worlds universe:

This time, you can see distribution of empires is roughly circular. This is because when a new player signs up, I first try to slot him in a star somewhere in the middle (usually where another empire joined, but then abandoned). If there's nowhere in the middle to add them, then new signups get added to a piece of empty space as close as possible to the "centre", leading to a roughly circular distribution. Also interesting here you can see all of the early players clustered around the middle, with some of the newer player's large empires around the outside. There's roughly 1.7 million stars represented here.

Negative Ships

Now, somehow, during Proto's probing of the game (which, I will say, is something I am not necessarily opposed to), he managed to get a fleet with a very large negative number of ships. I'm not entirely sure how this happened, but I suspect there was an integer overflow somewhere. An integer overflow happens when you have a very large integer value (> 2 billion) and it wraps around to a very large negative number instead. The issue here is that now, instead of costing money to send ships to other stars, you would be earning money to send ships to other stars!

Now, this doesn't actually get us infinite money, but it's quite easy to generate as much as you'd like. However, there was another glitch which did allow you to get infinite money, which I'm dubbing the Infinite Money glitch.

Infinite Money

This glitch was actually much more serious, in that it it actually allowed you to just generate money on demand (by depositing it into an alliance, and then having someone else take it out). I'm also not entirely sure how it was triggered, but it seems to have something to do with adding upgrades to fleets with "0" ships in them. Accelerating these builds seems to have caused the empire's "cash" value to go to NaN, which is a rather special number that tends to "infect" every calculation you make with it.

What Went Down

The exploits of Proto Baggins is what actually tipped people off to something going awry. People noticed his negative ship count, and that was quite suspicious. Next, they noticed that players who had only been in the game for a relatively short time were somehow able to amass massive armies of millions of ships, which they would launch against unsuspecting empires, who then had to scramble to deflect it.

What I discovered is that two empires, Carnage and Fatcow240, had hit the "Infinite Money" glitch and were depositing vast sums of cash into the alliance "Central Bank". They would then let their friends join that alliance, withdraw cash, and the cycle would continue.

Here's an example of what players would've seen before possibly getting completely wiped out:

Fallout

After I figured out what was going on, I decided to ban the players who were generating money. So that's Proto Baggins, Carnage and Fatcow240. Not everyone has been happy with that decision, but I felt it was warranted for two reasons:

1. While there are glitches from time to time, and glitches can sometime last for a while before I am able to fix them, this particular glitch was esspecially bad in that it only available to a few players who were "lucky" enough to have triggered the edge condition needed. Other glitches, such as the one that allowed players to create thousands of wormhole generators, were available to everyone: nobody was unfairly disadvantanged (at least, not once they were made aware of the glitch's existence), and
2. The advantage gained from the exploit was disproportionately large. Essentially infinite. The wormhole generator glitch was relatively harmless, in that once you've created the wormhole generators, there's not much you can do with them, since they're so expensive to move anyway. This exploit allowed a player to launch massive armadas of ships, which are basically impossible to defend against by anyone who was not also in control of an infinite supply of cash.

In addition, I went through the audit history of the "Central Bank" alliance, and anybody who withdrew cash from that alliance has had all their cash confiscated. I thought about also removing any ships that they accelerated the build of after getting their cash, but decided that it would be quite a lot of work to go through everybody's fleet list and remove the ones that seemed to be affected. I figure if you have no cash to actually move those ships, then it's not as bad.

What Went Well

There were a couple of things that helped resolve the situation.

Firstly, every single cash transaction in the game is stored in a special "audit" table. Every time you accelerate a build, move a fleet of ships, or deposit or withdraw to/from an alliance, a record is added to the "audit" table. This allowed me to easily go back and find out everyone who had withdrawn cash from the "Central Bank" alliance. It also means I could pinpoint when Carnage and Fatcow240 had managed to trigger the "infinite cash" bug, which then allowed me to narrow down the cause to an accelerate of a "0-ship" build.

Secondly, while my server-side code isn't perfect at filtering out bad requests, I think it's done a fairly decent job overall so that Proto Baggins, even though he was making requests directly to the server, wasn't able to do too much damage (apart from triggering the negative ship thing, as I noted above, which I consider to simply be a bug somewhere). This could still be somewhat improved, though.

The bugs themselves have now all been fixed, and I managed to fix a couple of other glitches which have been bugging people as well (such as the aforementioned wormhole generator bug, and another one which let you build boosters and such for cheap).

Finally, the players themselves have been quite supportive of some of the tough decisions I had to make, and also at reporting the issues. People sent me screenshots, snippets of conversations, and all sorts of details which greatly helped to track down the issues. Thanks to everyone who has reported an issue in the past!

What Went Wrong

Not everything was perfect, of course. The biggest problem is that I didn't realise anything was wrong until quite a long time after it started happening. This is mostly my own fault, because my email inbox receives quite a bit of mail from not just players, but also app store marketers (spammers, essentially), and all sorts of people, so things tend to slip through the cracks.

Also, my "audit" table is literally just a table in the database. So actually trawling through it look for evidence is a little time-consuming, concocting SQL queries and parsing the output into csv files and so on.

Finally, I'm just one person. I wish I was able to dedicate more time to community moderation and monitoring, but I generally only have a couple of hours per day that I am able to dedicate to War Worlds (usually after the kids are in bed), and I'd much rather be working on new features of fixing bugs than tracking down people exploiting bugs...

Action Items

Now, all of the actual issues should have already been fixed (i.e. the bugs that caused these glitches in the first place), but a couple of additional action items have come out of all this:

1. Spend more time in the chat. Keeping up with all of the email I get is a little tricky, but simply jumping on the in-game chat every now and then seems like a great way to keep on top of things (I do play the game as well, but I usually don't have time throughout the day to read chat as well),
2. Get a proper issue-tracking website. This one seems like a no-brainer, but some issues can linger for weeks or months in my inbox because I don't have a central way of tracking everything that people report. This has the added advantage that spammers marketers won't be contacting me there.
3. Monitoring and alerts! I need better monitoring for suspicious activity. If I'd been monitoring for things like spikes in requests per second, 400-errors per second and so on, I would have easily caught Proto Baggins before he'd been able to amass his large army.

So look for some of these changes coming in the next few weeks!

So after my last post dissing MySQL, I've had a few more issues with MySQL that's made me decide it's far more trouble than it's worth to keep it, so I'm ditching it for PostgreSQL.

The final straw was partially my own fault. I added a new table to the game's database, but forgot to add it to the Blitz database. So my blitz database started filling up with error messages (error messages get added to the database so I can analyze them later). It only took a couple of days for ibdata file to fill the entire disk. Now, you'd think this wasn't all that big of a deal, right? Just truncate the table, delete all the rows and it'll shrink back down again, right? Wrong, there is no way to shrink the the ibdata file! What you have to do, if you want to shrink your ibdata file, is export your data, blow the whole thing away, and re-import it again!

Worse, I can't just fix the table and leave the big file, because table definitions are stored in the same directory, and the disk was already full!

(There are other reasons for not using MySQL as well, but the general feeling of distrust towards Oracle also helps).

But in the end, since blowing the whole database away and starting again was my only option anyway, I decided that I'd finally migrate off MySQL.

Migrating the actual data

This was actually the simplest part, using pgloader, you just set up the mapping and it runs in no time. The biggest issue I faced was that the tool is written in lisp and the version of list that Debian runs is quite old, so I had to compile the latest version by hand. But with that out of the way, the migration is actually very quick (about 5 minutes for 4GB of data).

Migrating the code

When developing the server code, I was careful to limit the number of MySQL-isms, but it wasn't entirely possible to eliminate all of them. The first step in porting the code was to switch to PostgreSQL's JDBC driver, which is luckily not that difficult. It took me only a couple of hours one evening to port the rest of code away from MySQL-isms to PostgreSQL-isms, and another couple of evenings testing it all (this is where unit tests would have come in handy...)

Conclusion

I've only been running PostgreSQL for a few hours, but so far the performance has been pretty good. We'll see how it goes in the future.

So I promised to explain why I've been a little absent from War Worlds over the last few weeks, and now that Google I/O is on, I can finally reveal! At work, I work on the Google Maps for Mobile team, and we've been super busy getting Google Maps ready for the big reveal at I/O this year!

It's been a long couple of weeks, working hard, but now the big demo has gone off without a hitch, so hopefully I will be able to return to my regularly-scheduled role as benevolent overload and bringer-of-updates :)

If you want to learn more about Android Auto, here's the bit from the keynote where we got introduced. I'm so proud of my team!

(if it doesn't jump straight to the right second, you can fast-forward to 1h 35m in)

I'll be happy to answer any questions you have, though I'm not entirely sure how much I'm allowed to talk about just yet :)

Anyway, it's back to work on War Worlds for me! There's been a few updates over the last few days where I've tried to start pairing down some of the major crashes. I hope to have another update in the next few days that'll make a really big difference, but I think it'll be better if I spend a bit more time testing... we've had a few false starts over the last few days and don't want to repeat that!