Dev Update 02/25/2024: Building Blocks
Behind the Curtain on OpenTournament's Build Process
As February draws to a close, it’s time to give a quick update on Open Tournament development efforts. The past several months have marked a surge of interest and contributions to the project, which overall is great news, but highlighted some flaws in our development process.
Running on Duct-Tape & Bubblegum
Historically, Open Tournament’s development has been the story of tenacity and resourcefulness. The entire project has been developed with literally zero budget, using freely available resources and volunteer effort. Open Tournament has always been a labor of love geared towards build the best FPS shooter ever. But love only gets you so far. For as long as we’ve had a build pipeline our primary build machine has been the personal computer of an OT team member. Kicking off a build of the OT project could literally grind his PC to a halt as his system downloads and builds the game project in the background. Funny, but hardly ideal.
Late last year, we updated to Unreal Engine 5.3. Engine upgrades to the project also require a manual Unreal Engine update on every build machine. Since that team member was unavailable at the time, we weren’t able to perform the necessary updates. This put us in a bad spot. Because our primary build machine was no longer up to date, new commits to the project were not going through the full build process, which means that our commits might be breaking things in the game without even realizing it.
Responding to this, in January I recommissioned an HP Z620 workstation donated by a personal friend to act as another build machine. Equipped with a blazing-fast Nvidia Quadro K4000 GPU with 3GB of VRAM, it wasn’t clear if Unreal Engine 5 would even function on the decade-old box. Eventually, after a full weekend it took to update the machine with the latest Unreal Engine and its software dependencies, it was clear that the game engine would run, and, shockingly, Open Tournament was actually playable on such an antiquated machine.
With the initial setup out of the way, I followed up on some updates to the build pipeline itself. This took a little more engineering effort than expected due to the volume of changes to Open Tournament since our last engine update, but at the end of the process we had fully functional builds being generated on Windows for both server and clients.
Faster, Stronger, Better
Getting builds running again was a victory, but it wasn’t enough. Our repurposed build server was enough to keep things moving forward, but our incremental builds were taking over an hour to complete. Clean builds took over 3 hours. And full engine rebuilds took upwards of 24 hours. Those are just not acceptable wait times for developers volunteering their time to the project. We needed a better solution.
There are actually several possible approaches to this problem. Using a cloud service like Google Cloud, Microsoft Azure, or AWS could give us as much computing resources as we need to build the project efficiently. I have worked on game projects that do this, and it has some real advantages.
Alas, what these services bring in convenience and portability can’t outweigh the issue of ongoing costs. A project like Open Tournament has to run lean and fast, so I took the plunge and queued up some new hardware on Amazon. This actually represents a big step for Open Tournament, as it’s a real financial commitment for a project that has thus far run on free services and software.
Our new primary build machine is merely modest, but is a huge step up over the backup in every respect:
Intel i5 13400
ASRock 660M-HDV
16 GB (2 x 8 GB) DDR4 RAM
Nvidia 3060 12GB
970 EVO Plus 2TB - M.2 NVMe
The benchmark comparison on the GPU alone rates almost 700% improvement. The end result, after a few weekends of hardware assembly & build pipeline updates has been absolutely fantastic. Open Tournament is churning out incremental builds in under 15 minutes, and clean builds in about 90 minutes.
Looking Ahead
Now that our build pipeline has some dedicated hardware and a bit more resiliency to single points of failure, I’m moving back to more actively developing features on Open Tournament itself. As development accelerates over the next few months I am sure we will need to revisit the pipeline for further improvements, both for quality of life and to position ourselves for playtesting and releases later this year.
Cheers,
Wail