The Lumberyard engine ships with over 2000 unit tests, in this guide we’ll take a brief look at how to run these tests and debug them should they fail.
Why might you want to do this you ask? Well, one of the great things about Lumberyard is that the license grants you access to the source code so you can modify it to fully customise your game. Indeed, at TKG we have already made nearly 200 modifications to the engine, but it’s important to ensure that these modifications don’t break core functionality and the unit tests provide great coverage here. Our build servers pick up each modification we make and run the test suite to validate our changes against the expected behaviour and quickly isolate any unexpected breaks.
Building the Unit Tests
To build the test runner and compile the engine tests use the special “test” configuration.
Open a command window in your dev folder and build the Debug Test configuration with the following command.
lmbr_waf.bat build_win_x64_vs2015_debug_test -p all
Alternatively, from the Visual Studio solution, you can build the “[All] Debug Test, X64” configuration.
Setting up a Blacklist
To run the tests we use the “Amazon Test Scanner” which automatically finds and loads libraries in the bin folder, a side-effect of this approach is that it can pick up third party libraries which don’t have a test suite and can cause the scanner to crash. To solve this we exclude any incompatible libraries by creating a blacklist of known files which should not be scanned. In future Lumberyard releases this file may be included by default, but for 1.9 it must be added by hand.
The blacklist file is a newline separated list of wildcards, each wildcard lists the name of a library which should not be scanned for tests. A simple example looks like this:
opengl.* python.*
For a more complete example which works out the box with Lumberyard 1.9 download the following file: lmbr_test_blacklist.txt
This should be placed in the dev folder, e.g. “dev/lmbr_test_blacklist.txt”.
Running the Unit Tests
To run the tests use the lmbr_test tool, pointing it at the bin folder that you built earlier.
cd dev lmbr_test.cmd scan --dir Bin64vc140.Debug.Test
This will run through every enabled test in each library. Once the command completes you can find the results in an HTML report.
Running individual tests
To run the tests from a specific library use the –only flag and provide the name of the module. For example.
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only ResourceCompilerUiCanvas.dll
You can also separate multiple libraries with a comma
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only ResourceCompilerUiCanvas.dll,ResourceCompilerPC.dll
To run a specific test from a specific library user the –gtest_filter flag and supply the name of the test
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only SceneData.dll --gtest_filter=MeshDataPrimitiveUtils.CreateBox_BasicValues_BoxHasCorrectTopology
Or to run all tests from a particular test suite you can use a wildcard. E.g.
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only SceneData.dll --gtest_filter=MeshDataPrimitiveUtils.*
Debugging Tests
Debugging individual tests from Visual Studio
The easiest way to debug a failing test is by running it in isolation with the Visual Studio debugger.
- Generate the visual studio solution (if you’ve not already).
lmbr_waf configure
- Load the solution from “dev\Solutions” and change the configuration to “[All] Debug Test, X64”
- Set the AzTestRunner project as the default startup project.
- Right click the AZTestRunner project and select properties. From the Debugging menu update the Command Arguments to run your test.For example to run the “BehaviorContextTest.Test” from the “AzCoreTests.dll” library use the following arguments.
AzCoreTests.dll AzRunUnitTests --gtest_filter=BehaviorContextTest.Test --gtest_break_on_failure
I also recommend using the –gtest_break_on_failure flag. If you have a test which is crashing then setting this option will cause the test runner to break on the actual crash (without this option the debugger may instead break inside the Google test runner code).
Attaching a debugger to lmbr_test
It is also possible to attach the debugger after the test runner is launched, this can be useful if you’re running from the command line, simply add the –wait-for-debugger argument
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --wait-for-debugger