Organizing Tests in Julia

Comparing Julia test framework with popular test frameworks in Python, Go and Java

Maybe you are already familiar with test frameworks in other languages such as Python, Java and Go. Now you want to get up to speed with testing in Julia. In this story, we will explore how testing is similar and different from what you may already be used to.

Before discussing testing in Julia, I will give a quick overview of testing in popular testing frameworks from some mainstream programming languages. We will contrast with the testing framework bundled with Julia, which is based on the concept of nested test sets. This is a very flexible approach, but may be confusing if you have never encountered it before.

Pytest - Unit Testing in Python

Pytest is a frequently used testing framework for Python. Here, one simply prefixes functions that represent tests with test_ as shown below:

# Python - capitalize.py file
def capital_case(x):
    return x.capitalize()

def test_capital_case():
    assert capital_case('semaphore') == 'Semaphore'
    
# intentionally not working
def test_defective():
    assert "hello".upper() == "heLLO"

I have intentionally made one test fail, so you can see how failing tests are indicated. We can run pytest in a directory with multiple test files or specify a file to test directly, as in this example.

❯ pytest capitalize.py
===================== test session starts =====================
platform darwin -- Python 3.8.12, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/erikengheim/Development/Python/example
collected 2 items

capitalize.py .F                                        [100%]

========================== FAILURES ===========================
_______________________ test_defective ________________________

    def test_defective():
>       assert "hello".upper() == "heLLO"
E       AssertionError: assert 'HELLO' == 'heLLO'
E         - heLLO
E         + HELLO

capitalize.py:9: AssertionError
=================== short test summary info ===================
FAILED capitalize.py::test_defective - AssertionError: asser...
================= 1 failed, 1 passed in 0.11s =================

Unit Testing in Go

You can find lots of unit tests for most mainstream languages but I am focusing on well known and simple frameworks, so for Go will we look at the builtin unit testing framework which is still very popular among Go developers.

It is based on the simple idea of naming every test function as TestXXX(t *testing.T) where XXX is replaced with whatever you want to test. You need to accept an argument of type *testing.T to communicate with the testing framework.

// Go - foobar_test.go file
package main

import "testing"

func TestCapitalize(t *testing.T) {
    total := Sum(5, 5)
    if total != 10 {
       t.Errorf("Sum was incorrect, got: %d, want: %d.", 
                 total, 10)
    }
}

// deliberately made to fail
func TestDefective(t *testing.T) {
    total := Sum(5, 5)
    if total != 10 {
       t.Errorf("Sum was incorrect, got: %d, want: %d.",
                total, 10)
    }
}

For Go tests you put them in a file which ends with _test such as foobar_test.go. We made the last test fail, so you can see output from Go.

❯ go test
--- FAIL: TestDefective (0.00s)
    foobar_test.go:23: Uppercase incorrect, got: HELLO, want: heLLO.
FAIL
exit status 1
FAIL	github.com/ordovician/dummy	0.120s

This will look for the files ending with _test.go and treat every function with a Test prefix, taking a testing.T type argument as a test.

JUnit - Unit Testing in Java

JUnit dominates in the Java world. Each test is stored in a method on a test class. Notice how that is different from Python and Go, which uses free functions rather than test classes.

// Java - MyTests.java file
public class MyTests {

    @Test
    public void multiplyZeroIntsShouldReturnZero() {
        MyClass tester = new MyClass(); // MyClass is tested

        // assert statements
        assertEquals(0, tester.multiply(10, 0), 
                        "10 x 0 must be 0");
        assertEquals(0, tester.multiply(0, 10), 
                        "0 x 10 must be 0");
        assertEquals(0, tester.multiply(0, 0), 
                        "0 x 0 must be 0");
    }
}

Then these classes are into a test suite; thus you can avoid making enormous test classes:

// Java - JUnit test suite
@RunWith(Suite.class)
@SuiteClasses({
        MyClassTest.class,
        MySecondClassTest.class })

public class AllTests {

}

Testing with Julia Test Sets

In Julia, we do away with this artificial separation between test methods, test classes, and test suites. Instead, everything gets mashed into one concept called a testset. These are more flexible than what you find in other frameworks, which is why developers coming from other frameworks may feel uncomfortable with the much looser organization in Julia.

In Julia, everything is much more free form. You tailor the test framework more to your preferences and style. This is possible because test sets can be nested.

Here is an example from my Little Man Computer (LMC) assembler which you can find on Github to look more at the tests in detail.

@testset "Disassembler tests" begin
   
    @testset "Without operands" begin
        @test disassemble(901) == "INP"
        @test disassemble(902) == "OUT"
        @test disassemble(000) == "HLT"                                     
    end
    
    @testset "With operands" begin
        @test disassemble(105) == "ADD 5"
        @test disassemble(112) == "ADD 12"
        @test disassemble(243) == "SUB 43"
        @test disassemble(399) == "STA 99"
        @test disassemble(510) == "LDA 10"
        @test disassemble(600) == "BRA 0"
        @test disassemble(645) == "BRA 45"
        @test disassemble(782) == "BRZ 82"                                        
    end   
end

For every major software component, I define a test set. Let me give some examples: The assembler, disassembler, and simulator are all represented by a different test set. Each of these test sets have subtest sets which test aspects of that component.

Thus, a Julia @testset corresponds to both a test function and a test class, and a test suite.

What to Put in a Test Set

There are different ideas about what should be in a unit test. Many swear by the principle of one specific thing being tested per test. The Julia convention for test sets is that each test set has a collection of related tests. This is best understood by simply looking at real-world tests found in the Julia standard library.

The test sets I show as examples here will be shortened a bit by me, as there is no particular value in showing the full length of each test.

# Julia - abstractarray.jl tests

A = rand(5,4,3)
@testset "Bounds checking" begin
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
    @test checkbounds(Bool, A, 0, 1, 1) == false
    @test checkbounds(Bool, A, 1, 0, 1) == false
    @test checkbounds(Bool, A, 1, 1, 0) == false
end

@testset "vector indices" begin
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
    @test checkbounds(Bool, A, 1:5, 0:4, 1:3) == false
    @test checkbounds(Bool, A, 1:5, 1:4, 0:3) == false
end

Notice that tests defined for abstractarray.jl are designed to check several related things. Furthermore, there are no fixtures. We simply put the array A being tested against repeatedly outside the test sets. That makes it available in every test. We will explore alternatives to fixtures later.

Let us do another example by looking at the tests for individual characters represented by the Char type.

# Julia - char.jl tests
@testset "basic properties" begin

    @test typemin(Char) == Char(0)
    @test ndims(Char) == 0
    @test getindex('a', 1) == 'a'
    @test_throws BoundsError getindex('a', 2)
    # This is current behavior, but it seems questionable
    @test getindex('a', 1, 1, 1) == 'a'
    @test_throws BoundsError getindex('a', 1, 1, 2)

    @test 'b' + 1 == 'c'
    @test typeof('b' + 1) == Char
    @test 1 + 'b' == 'c'
    @test typeof(1 + 'b') == Char
    @test 'b' - 1 == 'a'
    @test typeof('b' - 1) == Char

    @test widen('a') === 'a'
    # just check this works
    @test_throws Base.CodePointError Base.code_point_err(UInt32(1))
end

@testset "issue #14573" begin
    array = ['a', 'b', 'c'] + [1, 2, 3]
    @test array == ['b', 'd', 'f']
    @test eltype(array) == Char

    array = [1, 2, 3] + ['a', 'b', 'c']
    @test array == ['b', 'd', 'f']
    @test eltype(array) == Char

    array = ['a', 'b', 'c'] - [0, 1, 2]
    @test array == ['a', 'a', 'a']
    @test eltype(array) == Char
end

Again, you can observe that related tests are grouped into test sets. I added the second test set to demonstrate that Julia developers following popular advice regarding bug fixing: Whenever you fix a bug, create a unit test which can catch any code modification which re-introduces the bug. With such practices in place, we can more easily avoid regressions in code quality.

Nesting Test Sets

A test set with code you are testing can also contain other test sets. Here is an example from the testing of the Dict type used to represent a dictionary in Julia.

# Julia - dict.jl tests

@testset "Dict" begin
    h = Dict()
    for i=1:10000
        h[i] = i+1
    end
    for i=1:10000
        @test (h[i] == i+1)
    end
    for i=1:2:10000
        delete!(h, i)
    end
    
    h = Dict{Any,Any}("a" => 3)
    @test h["a"] == 3
    h["a","b"] = 4
    @test h["a","b"] == h[("a","b")] == 4
    h["a","b","c"] = 4
    @test h["a","b","c"] == h[("a","b","c")] == 4

    @testset "eltype, keytype and valtype" begin
        @test eltype(h) == Pair{Any,Any}
        @test keytype(h) == Any
        @test valtype(h) == Any

        td = Dict{AbstractString,Float64}()
        @test eltype(td) == Pair{AbstractString,Float64}
        @test keytype(td) == AbstractString
        @test valtype(td) == Float64
        @test keytype(Dict{AbstractString,Float64}) === AbstractString
        @test valtype(Dict{AbstractString,Float64}) === Float64
    end
end

The code shows that the dictionary named h tested on inside the inner test set was actually defined first in the outer test set.

Repeat Tests with For Loops

It is easy in Julia to repeat tests across different kinds of data using loops.

let x = Dict(3=>3, 5=>5, 8=>8, 6=>6)
    pop!(x, 5)
    for k in keys(x)
        Dict{Int,Int}(x)
        @test k in [3, 8, 6]
    end
end

We can even loop on the whole test set itself. Here we are running the same set of test for both the == and isequal function:

@testset "equality" for eq in (isequal, ==)
    @test  eq(Dict(), Dict())
    @test  eq(Dict(1 => 1), Dict(1 => 1))
    @test !eq(Dict(1 => 1), Dict())
    @test !eq(Dict(1 => 1), Dict(1 => 2))
    @test !eq(Dict(1 => 1), Dict(2 => 1))
end

How to Write Fixtures in Julia

Man test frameworks have the concept of fixtures. Fixtures represent initialized data we wish to use repeatedly in multiple tests. Julia's developers believe it is over-engineering to create a unique concept just to handle this particular use-case. Instead, Julia developers believe in creating reusable data explicitly. Many testing frameworks, with support for fixtures, create these fixtures in an implicit manner.

Creating Immutable Fixtures

If you never modify your fixture, you can simply create the fixture as an object outside all the tests using that fixture. In this case, the A array is immutable (it doesn't change) and we used it in both the bounds checking tests and vector indices tests.

A = rand(5,4,3)
@testset "Bounds checking" begin
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
end

@testset "vector indices" begin
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end

Creating Mutable Fixtures

Sometimes functionality we are testing requires mutating an object. In this case, we cannot reuse the same object in multiple tests because each test can potentially modify the fixture object and leave it in an unknown state for the next test. In this case, we instead define a function to create the fixture object. Every test using the fixture will call the function creating the fixture. In this example, test_array, is a function creating a fixture.

test_array() = rand(5,4,3)

@testset "Bounds checking" begin
      A = test_array()
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
end

@testset "vector indices" begin
      A = test_array()
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end

Going Further

Want to learn more about testing in Julia? Read the official Julia documentation on testing: Unit Testing.

There are many features I did not cover which might be worth exploring further:

• Broken Tests @test_broken - Mark a test as consistently breaking.

• AbstractTestSet type - To customize how tests are recorded.