How to run the Lumberyard Unit Tests

The Lumberyard engine ships with over 2000 unit tests, in this guide we’ll take a brief look at how to run these tests and debug them should they fail.

Why might you want to do this you ask? Well, one of the great things about Lumberyard is that the license grants you access to the source code so you can modify it to fully customise your game. Indeed, at TKG we have already made nearly 200 modifications to the engine, but it’s important to ensure that these modifications don’t break core functionality and the unit tests provide great coverage here. Our build servers pick up each modification we make and run the test suite to validate our changes against the expected behaviour and quickly isolate any unexpected breaks.

Building the Unit Tests

To build the test runner and compile the engine tests use the special “test” configuration.

Open a command window in your dev folder and build the Debug Test configuration with the following command.

lmbr_waf.bat build_win_x64_vs2015_debug_test -p all

Alternatively, from the Visual Studio solution, you can build the “[All] Debug Test, X64” configuration.

 

Setting up a Blacklist

To run the tests we use the “Amazon Test Scanner” which automatically finds and loads libraries in the bin folder, a side-effect of this approach is that it can pick up third party libraries which don’t have a test suite and can cause the scanner to crash. To solve this we exclude any incompatible libraries by creating a blacklist of known files which should not be scanned. In future Lumberyard releases this file may be included by default, but for 1.9 it must be added by hand.

The blacklist file is a newline separated list of wildcards, each wildcard lists the name of a library which should not be scanned for tests. A simple example looks like this:

opengl.*
python.*

For a more complete example which works out the box with Lumberyard 1.9 download the following file: lmbr_test_blacklist.txt

This should be placed in the dev folder, e.g. “dev/lmbr_test_blacklist.txt”.

Running the Unit Tests

To run the tests use the lmbr_test tool, pointing it at the bin folder that you built earlier.

cd dev
lmbr_test.cmd scan --dir Bin64vc140.Debug.Test

This will run through every enabled test in each library. Once the command completes you can find the results in an HTML report.

 

Running individual tests

To run the tests from a specific library use the –only flag and provide the name of the module. For example.

lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only ResourceCompilerUiCanvas.dll

You can also separate multiple libraries with a comma

 lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only ResourceCompilerUiCanvas.dll,ResourceCompilerPC.dll

To run a specific test from a specific library user the  –gtest_filter flag and supply the name of the test

lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only SceneData.dll --gtest_filter=MeshDataPrimitiveUtils.CreateBox_BasicValues_BoxHasCorrectTopology

Or to run all tests from a particular test suite you can use a wildcard. E.g.

lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --only SceneData.dll --gtest_filter=MeshDataPrimitiveUtils.*

 

 

Debugging Tests

Debugging individual tests from Visual Studio

The easiest way to debug a failing test is by running it in isolation with the Visual Studio debugger.

1. Generate the visual studio solution (if you’ve not already).

   lmbr_waf configure

2. Load the solution from “dev\Solutions” and change the configuration to “[All] Debug Test, X64”
3. Set the AzTestRunner project as the default startup project.
   
4. Right click the AZTestRunner project and select properties. From the Debugging menu update the Command Arguments to run your test.For example to run the “BehaviorContextTest.Test” from the “AzCoreTests.dll” library use the following arguments.

   AzCoreTests.dll AzRunUnitTests --gtest_filter=BehaviorContextTest.Test --gtest_break_on_failure

   I also recommend using the –gtest_break_on_failure flag. If you have a test which is crashing then setting this option will cause the test runner to break on the actual crash (without this option the debugger may instead break inside the Google test runner code).

   

Attaching a debugger to lmbr_test

It is also possible to attach the debugger after the test runner is launched, this can be useful if you’re running from the command line, simply add the –wait-for-debugger argument

lmbr_test.cmd scan --dir Bin64vc140.Debug.Test --wait-for-debugger

 

 

Additional Reading