The beauty of tools development – Pt 2/3 Beam

One of the first tools that we developed in-house was used to distribute test builds of DRG before release. We called it Beam, which naturally comes from the U.F.O. tractor beams which are used to move the build from the servers to the desktop. The tool was initially built in around two weeks over the Christmas period in 2016.

When I set about designing it, I had the goal of leveraging open source components as much as possible, this naturally saved development time, but also gave me access to an extended development community. Up until this point I’d made desktop tools in WPF for many years, but I’d also made many web tools and knew that the javascript open source community was much more vibrant. I couldn’t help but feel that I could build better front-ends with web due to the rapid iteration time and a larger community, so I set about finding a way to build a desktop tool with these technologies. I quickly found electron which combines browser with desktop and seemed perfect in all but one way… did I really want to write the business and back-end logic in javascript? Absolutely not, give me a strongly typed compiled language any day of the week. Initially, I was so keen to keep my C# roots that I began experimenting with electron-edge, a project which lets you (almost) seamlessly switch between js and cs in the same file, it’s pretty mind-blowing actually. By this point though I was getting a bit ahead of myself, I hadn’t even used electron properly yet, so I decided to get a grip on that first, using just javascript. I added semantic-ui for the front-end and before long I had a version 1:

I used a plethora of other open technologies for the distribution process, at the core was rsync which delivered the incremental build updates and was pretty essential, since, believe it or not, the internet wasn’t all that quick in a number of our working locations. This also made the server and security aspects easy, we simply deployed an EC2 Linux box with OpenSSH and I say “simply” because it was indeed simple using our already established automation pipeline powered by the open source Puppet server.

You might have spotted in the screenshot that we keep an archive of builds, which means that we can always switch back and forth between various versions and branches. As you might imagine though game builds can get pretty large, ours is actually only 7GB or so, but when you’re paying EBS by the gigabyte-month, you might think twice about how many builds to store. This is where our secret ingredient comes in, we used another tool “bup” to store the builds more efficiently. It’s actually designed as a backup system, but it provides an effective block level deduplication system which uses a rolling checksum for variable block sizes. The whole tool is based on the git packfile format (but not git itself so you can even store huge files if you like). Naturally it provides excellent versioning features. The final part of the puzzle was the ability to mount any build version to a FUSE filesystem which allowed it to be accessed transparently by rsync. Before long we had 100+ builds across multiple branches all stored in less than 30GB space, saving well over $70/month we could now even afford a few beers to celebrate.

The look and feel of the tool hasn’t actually changed all that much since version 1, we added some DRG branding, a game launcher and some metadata for the builds (stored in DynamoDB). Here’s what it looks like today:

The front-end definitely delivered, with minimal development time a tool was born. The original prototype was implemented mostly in a single HTML file and that was driven by jquery. The simplicity of it all was quite elegant but the engineering certainly wasn’t; it wouldn’t scale to more complicated tools and for a moment it was as though I’d forgotten all about MVVM, MVC, MVC-VM patterns, let alone something trendy like immutable application state or stateless components. In fairness, this was quite deliberate, I wanted to prove the concept without becoming bogged down with engineering but with the next tool, I decided to get serious… join me in part 3 where I’ll be talking about TypeScript and React and possibly trying to hold back my opinion of gulp.

Leave a Reply