Chapter 11

Writing Automated Tests

In his 1972 essay “The Humble Programmer,” Edsger W. Dijkstra said that “program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” That doesn’t mean we shouldn’t try to test as much as we can!

Testing is a complex skill: one chapter cannot cover every detail of writing good tests, so this chapter covers the mechanics of Rust’s testing facilities — the annotations and macros available when writing tests, the default behavior and options for running them, and how to organize tests into unit tests and integration tests.

11.1 How to Write Tests
11.2 Controlling How Tests Are Run
11.3 Test Organization

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · Introduction

Correctness, and why the type system is not enough

Definition 11.1 (Correctness). The extent to which our code does what we intend it to do.

Rust is designed with a high degree of concern about the correctness of programs, but correctness is complex and not easy to prove. Rust’s type system shoulders a huge part of this burden, but it cannot catch everything. As such, Rust includes support for writing automated software tests.

Example 11.2. Say we write a function add_two that adds 2 to whatever number is passed to it: it accepts an integer parameter and returns an integer. When we compile it, Rust does all the type checking and borrow checking to ensure, for instance, that we aren’t passing a String value or an invalid reference to it. But Rust can’t check that it does precisely what we intend — return the parameter plus 2 rather than, say, the parameter plus 10 or the parameter minus 50! That’s where tests come in.

We can write tests that assert, for example, that when we pass 3 to add_two, the returned value is 5. We can run these tests whenever we change our code, to make sure any existing correct behavior has not changed.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

What a test is

Definition 11.3 (Test). A Rust function that verifies that the non-test code is functioning in the expected manner.

The bodies of test functions typically perform these three actions:

Set up any needed data or state.
Run the code you want to test.
Assert that the results are what you expect.

Rust provides features specifically for writing tests that take these actions:

the test attribute;
a few macros (assert!, assert_eq!, assert_ne!);
the should_panic attribute.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Structuring test functions: the test attribute

Definition 11.4 (Attribute). Metadata about pieces of Rust code — for example the derive attribute used with structs in Chapter 5.

At its simplest, a test is a function annotated with the test attribute: add #[test] on the line before fn. When you run cargo test, Rust builds a test runner binary that runs the annotated functions and reports whether each one passes or fails.

Whenever we make a new library project with Cargo, a test module with a test function in it is generated for us — a template, so you don’t have to look up the exact structure and syntax each time. You can add as many test functions and test modules as you want! We first experiment with the template, then write real tests of our own code. Create a library project adder that will add two numbers:

$ cargo new adder --lib
     Created library `adder` project
$ cd adder

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The generated test module

Listing 11.1: The code generated automatically by cargo new — src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

The file starts with an example add function so that we have something to test.

Focus on it_works. The #[test] annotation tells the test runner to treat this function as a test. The tests module might also hold non-test functions that set up common scenarios or perform common operations, so we always need to indicate which functions are tests.

The body uses the assert_eq! macro to assert that result — the result of calling add with 2 and 2 — equals 4. This assertion is an example of the format of a typical test.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Running the generated test with cargo test

The cargo test command runs all tests in our project:

Listing 11.2: The output from running the automatically generated test

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.57s
     Running unittests src/lib.rs (target/debug/deps/adder-01ad14159ff659ab)
 
running 1 test
test tests::it_works ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Reading the test output

Table 11.1: What each part of the output in Listing 11.2 means — Cargo compiled and ran the test

Output	Meaning
running 1 test	How many tests this run executes
test tests::it_works ... ok	The name of the generated test function, and the result of running it
test result: ok.	Overall summary: all the tests passed
1 passed; 0 failed	Totals of the tests that passed or failed
0 ignored	Tests marked as ignored so they don’t run in a particular instance (see 11.2)
0 measured	Benchmark tests that measure performance — as of this writing only in nightly Rust
0 filtered out	Tests not run because an argument to cargo test selected only names matching a string (see 11.2)
Doc-tests adder	Results of documentation tests: Rust can compile code examples in our API documentation, which keeps docs and code in sync (Chapter 14). Ignored for now.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Customizing the test: renaming it

src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn exploration() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

Let’s start to customize the test to our own needs. First, change the name of the it_works function to a different name, such as exploration.

Then run cargo test again. The output (next slide) now shows exploration instead of it_works.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The output now names exploration

The test line now reads test tests::exploration ... ok:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.59s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::exploration ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

A test that fails: calling panic!

Tests fail when something in the test function panics. Each test is run in a new thread, and when the main thread sees that a test thread has died, the test is marked as failed. The simplest way to panic is to call the panic! macro (Chapter 9). Add a test named another:

Listing 11.3: Adding a second test that will fail because we call the panic! macro — src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 

    #[test]
    fn exploration() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
 
    #[test]
    fn another() {
        panic!("Make this test fail");
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

One test passes, one test fails

Run the tests again using cargo test. The output shows that exploration passed and another failed: instead of ok, the line test tests::another shows FAILED.

Listing 11.4: Test results when one test passes and one test fails

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.72s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 2 tests
test tests::another ... FAILED
test tests::exploration ... ok
 
failures:
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The failure details and the failing-test list

Listing 11.4 (continued)

---- tests::another stdout ----
 
thread 'tests::another' (6019162) panicked at src/lib.rs:17:9:
Make this test fail
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::another
 
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

Two new sections appear between the individual results and the summary.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Reading a failure report

In Listing 11.4, the two sections between the individual results and the summary are:

The detailed reason for each test failure. Here, tests::another failed because it panicked with the message Make this test fail on line 17 in the src/lib.rs file.
Just the names of all the failing tests — useful when there are lots of tests and lots of detailed failing test output. We can use the name of a failing test to run just that test and debug it more easily (see “Controlling How Tests Are Run”, 11.2).

The summary line displays at the end: overall, our test result is FAILED. We had one test pass and one test fail.

Now that you’ve seen what the test results look like in different scenarios, let’s look at some macros other than panic! that are useful in tests.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Checking results with assert!

The assert! macro, provided by the standard library, ensures that some condition in a test evaluates to true. We give it an argument that evaluates to a Boolean. If the value is true, nothing happens and the test passes. If the value is false, assert! calls panic! to cause the test to fail. It helps us check that our code functions the way we intend.

Listing 11.5: The Rectangle struct and its can_hold method from Chapter 5 — src/lib.rs

#[derive(Debug)]
struct Rectangle {
    width: u32,
    height: u32,
}
 
impl Rectangle {
    fn can_hold(&self, other: &Rectangle) -> bool {
        self.width > other.width && self.height > other.height
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Testing can_hold with assert!

This is Listing 5-15 from Chapter 5, now in src/lib.rs. can_hold returns a Boolean, a perfect use case for assert!. The test creates a Rectangle of width 8 and height 7 and asserts that it can hold one of width 5 and height 1.

Listing 11.6: A test for can_hold that checks whether a larger rectangle can indeed hold a smaller rectangle — src/lib.rs

#[derive(Debug)]
struct Rectangle {
    width: u32,
    height: u32,
}
 
impl Rectangle {
    fn can_hold(&self, other: &Rectangle) -> bool {
        self.width > other.width && self.height > other.height
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The test module and use super::*

Listing 11.6 (continued)

 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn larger_can_hold_smaller() {
        let larger = Rectangle {
            width: 8,
            height: 7,

        };
        let smaller = Rectangle {
            width: 5,
            height: 1,
        };
 
        assert!(larger.can_hold(&smaller));
    }
}

Note the use super::*; line. The tests module is a regular module that follows the usual visibility rules (Chapter 7, “Paths for Referring to an Item in the Module Tree”). Because it is an inner module, we must bring the code under test in the outer module into its scope. We use a glob, so anything defined in the outer module is available to tests.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Running the can_hold test

We named the test larger_can_hold_smaller, created two Rectangles, and passed assert! the result of larger.can_hold(&smaller), which should be true. Let’s find out!

$ cargo test
   Compiling rectangle v0.1.0 (file:///projects/rectangle)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running unittests src/lib.rs (target/debug/deps/rectangle-6584c4561e48942e)
 
running 1 test
test tests::larger_can_hold_smaller ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests rectangle
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Asserting a negative: smaller cannot hold larger

It does pass! Now add a test asserting that a smaller rectangle cannot hold a larger rectangle:

src/lib.rs

#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn larger_can_hold_smaller() {
        // --snip--
    }
 
    #[test]
    fn smaller_cannot_hold_larger() {

        let larger = Rectangle {
            width: 8,
            height: 7,
        };
        let smaller = Rectangle {
            width: 5,
            height: 1,
        };
 
        assert!(!smaller.can_hold(&larger));
    }
}

The correct result of can_hold here is false, so we negate it before passing it to assert!. As a result, the test passes if can_hold returns false.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Two tests that pass

$ cargo test
   Compiling rectangle v0.1.0 (file:///projects/rectangle)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running unittests src/lib.rs (target/debug/deps/rectangle-6584c4561e48942e)
 
running 2 tests
test tests::larger_can_hold_smaller ... ok
test tests::smaller_cannot_hold_larger ... ok
 
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests rectangle
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Two tests that pass! Next: what happens to our test results when we introduce a bug.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Introducing a bug into can_hold

Now let’s see what happens to our test results when we introduce a bug. Change the implementation of can_hold by replacing the greater-than sign (>) with a less-than sign (<) when it compares the widths:

src/lib.rs

// --snip--
impl Rectangle {
    fn can_hold(&self, other: &Rectangle) -> bool {
        self.width < other.width && self.height > other.height
    }
}

The Rectangle struct and both tests stay exactly as before. Running the tests now produces the output on the next two slides.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The buggy can_hold: test output

Running the tests with the bug in place:

$ cargo test
   Compiling rectangle v0.1.0 (file:///projects/rectangle)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running unittests src/lib.rs (target/debug/deps/rectangle-6584c4561e48942e)
 
running 2 tests
test tests::larger_can_hold_smaller ... FAILED
test tests::smaller_cannot_hold_larger ... ok
 
failures:
 
---- tests::larger_can_hold_smaller stdout ----
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Our tests caught the bug

(output continued)

thread 'tests::larger_can_hold_smaller' (6020788) panicked at src/lib.rs:28:9:
assertion failed: larger.can_hold(&smaller)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::larger_can_hold_smaller
 
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

Our tests caught the bug! Because larger.width is 8 and smaller.width is 5, the comparison of the widths in can_hold now returns false: 8 is not less than 5.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Testing equality with assert_eq! and assert_ne!

A common way to verify functionality is to test for equality between the result of the code under test and the value you expect it to return. You could pass assert! an expression using the == operator — but this test is so common that the standard library provides a pair of macros to do it more conveniently:

assert_eq! compares two arguments for equality;
assert_ne! compares two arguments for inequality.

They also print the two values if the assertion fails, which makes it easier to see why the test failed. Conversely, assert! only indicates that it got a false value for the == expression, without printing the values that led to the false value.

In Listing 11.7 (next slide), we write a function named add_two that adds 2 to its parameter, and then test it using assert_eq!.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Testing add_two with assert_eq!

Listing 11.7: Testing the function add_two using the assert_eq! macro — src/lib.rs

pub fn add_two(a: u64) -> u64 {
    a + 2
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_adds_two() {
        let result = add_two(2);
        assert_eq!(result, 4);
    }
}

We create a variable named result that holds the result of calling add_two(2).

Then we pass result and 4 as the arguments to the assert_eq! macro.

Let’s check that it passes (next slide)!

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

assert_eq! passing

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.58s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::it_adds_two ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The output line for this test is test tests::it_adds_two ... ok, and the ok text indicates that our test passed!

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Making assert_eq! fail

src/lib.rs

pub fn add_two(a: u64) -> u64 {
    a + 3
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_adds_two() {
        let result = add_two(2);
        assert_eq!(result, 4);
    }
}

Let’s introduce a bug into our code to see what assert_eq! looks like when it fails.

Change the implementation of the add_two function to instead add 3, leaving the test unchanged.

Then run the tests again (output on the next two slides).

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

assert_eq! failing: test output

Run the tests again:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.61s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::it_adds_two ... FAILED
 
failures:
 
---- tests::it_adds_two stdout ----
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The failure shows the left and right values

thread 'tests::it_adds_two' (6020955) panicked at src/lib.rs:12:9:
assertion `left == right` failed
  left: 5
 right: 4
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::it_adds_two
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

Our test caught the bug! The message tells us the assertion that failed was left == right, and what the values are: the left argument, the result of add_two(2), was 5, but the right argument was 4. This helps us start debugging — especially helpful when we have a lot of tests going on.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Left and right, and when to use assert_ne!

Argument order does not matter

In some languages and test frameworks, the parameters to equality assertion functions are called expected and actual, and the order in which we specify the arguments matters. In Rust, they’re called left and right, and the order of the expected value and the value the code produces doesn’t matter. We could write assert_eq!(4, result), which would give the same failure message, assertion `left == right` failed.

assert_ne!

assert_ne! passes if the two values we give it are not equal and fails if they are equal. It is most useful when we’re not sure what a value will be, but we know what it definitely shouldn’t be.

Example 11.5. We test a function that is guaranteed to change its input in some way, but how the input changes depends on the day of the week we run our tests. The best thing to assert might be that the function’s output is not equal to its input.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

What assert_eq! and assert_ne! require: PartialEq and Debug

Under the surface, assert_eq! and assert_ne! use the operators == and !=, respectively. When the assertions fail, these macros print their arguments using debug formatting. So the values being compared must implement two traits:

PartialEq — to assert equality of the values;
Debug — to print the values when the assertion fails.

All primitive types and most of the standard library types implement these traits. For structs and enums that you define yourself, you’ll need to implement both. Because both are derivable traits (Listing 5-12 in Chapter 5), this is usually as straightforward as adding an annotation to your struct or enum definition:

#[derive(PartialEq, Debug)]

See Appendix C, “Derivable Traits,” for more details about these and other derivable traits.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Adding custom failure messages

You can add a custom message to be printed with the failure message as optional arguments to assert!, assert_eq!, and assert_ne!. Any arguments after the required ones are passed along to the format! macro (Chapter 8, “Concatenating with + or format!”), so you can pass a format string with {} placeholders and values to go in them. Custom messages document what an assertion means; when a test fails, you’ll have a better idea of what the problem is.

For example, a function greets people by name, and we test that the name we pass in appears in the output:

src/lib.rs

pub fn greeting(name: &str) -> String {
    format!("Hello {name}!")
}
 
#[cfg(test)]
mod tests {
    use super::*;

 
    #[test]
    fn greeting_contains_name() {
        let result = greeting("Carol");
        assert!(result.contains("Carol"));
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Why check contains, and a bug in greeting

src/lib.rs

pub fn greeting(name: &str) -> String {
    String::from("Hello!")
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn greeting_contains_name() {
        let result = greeting("Carol");
        assert!(result.contains("Carol"));
    }
}

The requirements for this program haven’t been agreed upon yet, and we’re pretty sure the Hello text at the beginning of the greeting will change. We don’t want to update the test when the requirements change, so instead of checking for exact equality with the value returned from greeting, we just assert that the output contains the text of the input parameter.

Now introduce a bug by changing greeting to exclude name (left), to see what the default test failure looks like.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The default failure message

Running this test produces the following:

$ cargo test
   Compiling greeter v0.1.0 (file:///projects/greeter)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.91s
     Running unittests src/lib.rs (target/debug/deps/greeter-170b942eb5bf5e3a)
 
running 1 test
test tests::greeting_contains_name ... FAILED
 
failures:
 
---- tests::greeting_contains_name stdout ----
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Only the assertion and its line

(output continued)

thread 'tests::greeting_contains_name' (6021143) panicked at src/lib.rs:12:9:
assertion failed: result.contains("Carol")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::greeting_contains_name
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

This result just indicates that the assertion failed and which line the assertion is on. A more useful failure message would print the value from the greeting function.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

A custom message with a format string

Add a custom failure message: a format string whose placeholder is filled with the actual value we got from greeting.

pub fn greeting(name: &str) -> String {
    String::from("Hello!")
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn greeting_contains_name() {
        let result = greeting("Carol");
        assert!(
            result.contains("Carol"),
            "Greeting did not contain name, value was `{result}`"
        );
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Running the test with the custom message

Now when we run the test, we get a more informative error message:

$ cargo test
   Compiling greeter v0.1.0 (file:///projects/greeter)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.93s
     Running unittests src/lib.rs (target/debug/deps/greeter-170b942eb5bf5e3a)
 
running 1 test
test tests::greeting_contains_name ... FAILED
 
failures:
 
---- tests::greeting_contains_name stdout ----
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The actual value appears in the output

(output continued)

thread 'tests::greeting_contains_name' (6021333) panicked at src/lib.rs:12:9:
Greeting did not contain name, value was `Hello!`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::greeting_contains_name
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

We can see the value we actually got in the test output, which helps us debug what happened instead of what we were expecting to happen.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Checking for panics with should_panic

In addition to checking return values, it’s important to check that our code handles error conditions as we expect.

Example 11.6. The Guess type from Chapter 9, Listing 9-13: other code that uses Guess depends on the guarantee that Guess instances contain only values between 1 and 100. We can write a test that ensures that attempting to create a Guess instance with a value outside that range panics.

Definition 11.7 (should_panic). An attribute added to a test function: the test passes if the code inside the function panics, and fails if the code inside the function doesn’t panic.

Listing 11.8 (next two slides) shows a test that checks that the error conditions of Guess::new happen when we expect them to.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The Guess type under test

Listing 11.8: Testing that a condition will cause a panic! — src/lib.rs

pub struct Guess {
    value: i32,
}
 
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 || value > 100 {
            panic!("Guess value must be between 1 and 100, got {value}.");
        }
 
        Guess { value }
    }
}

Guess::new panics whenever the value is below 1 or above 100. The test module follows on the next slide.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Placing #[should_panic]

Listing 11.8 (continued)

 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    #[should_panic]
    fn greater_than_100() {
        Guess::new(200);
    }
}

We place the #[should_panic] attribute after the #[test] attribute and before the test function it applies to.

The test calls Guess::new(200): 200 is outside the range 1 to 100, so the code should panic — and the test should therefore pass.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

A should_panic test that passes

The result when this test passes (note the - should panic marker) looks good!

$ cargo test
   Compiling guessing_game v0.1.0 (file:///projects/guessing_game)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.58s
     Running unittests src/lib.rs (target/debug/deps/guessing_game-57d70c3acb738f4d)
 
running 1 test
test tests::greater_than_100 - should panic ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests guessing_game
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Removing the upper-bound check

Now let’s introduce a bug in our code by removing the condition that the new function will panic if the value is greater than 100:

src/lib.rs

// --snip--
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 {
            panic!("Guess value must be between 1 and 100, got {value}.");
        }
 
        Guess { value }
    }
}

The struct and the test module stay as in Listing 11.8. When we run the test in Listing 11.8 now, it will fail (next two slides).

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

A should_panic test that fails

Running the test against the buggy Guess::new:

$ cargo test
   Compiling guessing_game v0.1.0 (file:///projects/guessing_game)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.62s
     Running unittests src/lib.rs (target/debug/deps/guessing_game-57d70c3acb738f4d)
 
running 1 test
test tests::greater_than_100 - should panic ... FAILED
 
failures:
 
---- tests::greater_than_100 stdout ----

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

“Test did not panic as expected”

(output continued)

note: test did not panic as expected at src/lib.rs:21:8
 
failures:
    tests::greater_than_100
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

We don’t get a very helpful message in this case, but the test function is annotated with #[should_panic], so the failure means that the code in the test function did not cause a panic.

Tests that use should_panic can be imprecise: a should_panic test would pass even if the test panics for a different reason from the one we were expecting.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Making should_panic precise with expected

To make should_panic tests more precise, add an optional expected parameter to the attribute: the test harness makes sure the failure message contains the provided text. In Listing 11.9, new panics with different messages depending on whether the value is too small or too large.

Listing 11.9: Testing for a panic! with a panic message containing a specified substring — src/lib.rs

// --snip--
 
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 {
            panic!(
                "Guess value must be greater than or equal to 1, got {value}."
            );
        } else if value > 100 {

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Two different panic messages

Listing 11.9 (continued)

            panic!(
                "Guess value must be less than or equal to 100, got {value}."
            );
        }
 
        Guess { value }
    }
}
 
#[cfg(test)]
mod tests {
    use super::*;
 

The value 200 takes the else if value > 100 branch, whose message says “less than or equal to 100”. The test itself follows on the next slide.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The expected substring

Listing 11.9 (continued)

    #[test]
    #[should_panic(expected = "less than or equal to 100")]
    fn greater_than_100() {
        Guess::new(200);
    }
}

This test passes because the value in the expected parameter is a substring of the message that Guess::new panics with. We could have specified the entire panic message, Guess value must be less than or equal to 100, got 200. What you choose depends on how much of the panic message is unique or dynamic and how precise you want your test to be. Here, a substring is enough to ensure that the code executes the else if value > 100 case.

To see a should_panic test with an expected message fail, we introduce a bug by swapping the bodies of the if value < 1 and else if value > 100 blocks.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Swapping the two panic messages

The bodies of the two branches are now swapped:

src/lib.rs

pub struct Guess {
    value: i32,
}
 
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 {
            panic!(
                "Guess value must be less than or equal to 100, got {value}."
            );
        } else if value > 100 {
            panic!(
                "Guess value must be greater than or equal to 1, got {value}."
            );
        }

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

The unchanged should_panic test

src/lib.rs (continued)

 
        Guess { value }
    }
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    #[should_panic(expected = "less than or equal to 100")]
    fn greater_than_100() {
        Guess::new(200);
    }
}

This time when we run the should_panic test, it will fail.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

An expected-message test that fails

$ cargo test
   Compiling guessing_game v0.1.0 (file:///projects/guessing_game)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running unittests src/lib.rs (target/debug/deps/guessing_game-57d70c3acb738f4d)
 
running 1 test
test tests::greater_than_100 - should panic ... FAILED
 
failures:
 
---- tests::greater_than_100 stdout ----
 

The failure details follow on the next slide.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Panic message vs expected substring

thread 'tests::greater_than_100' (6021675) panicked at src/lib.rs:12:13:
Guess value must be greater than or equal to 1, got 200.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: panic did not contain expected string
      panic message: "Guess value must be greater than or equal to 1, got 200."
 expected substring: "less than or equal to 100"
 
failures:
    tests::greater_than_100
 
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

The failure message indicates that this test did indeed panic as we expected, but the panic message did not include the expected string less than or equal to 100. The panic message we did get was Guess value must be greater than or equal to 1, got 200. Now we can start figuring out where our bug is!

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Using Result<T, E> in tests

All of our tests so far panic when they fail. We can also write tests that use Result<T, E>! Here’s the test from Listing 11.1, rewritten to use Result<T, E> and return an Err instead of panicking:

src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_works() -> Result<(), String> {
        let result = add(2, 2);
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.1 How to Write Tests

Returning Ok or Err from a test

src/lib.rs (continued)

        if result == 4 {
            Ok(())
        } else {
            Err(String::from("two plus two does not equal four"))
        }
    }
}

it_works now has the Result<(), String> return type. Rather than calling assert_eq!, the body returns Ok(()) when the test passes and an Err with a String inside when it fails.

Tests that return a Result<T, E> can use the question mark operator in their body — a convenient way to write tests that should fail if any operation within them returns an Err variant.

You can’t use #[should_panic] on tests that use Result<T, E>. To assert that an operation returns an Err variant, don’t use the question mark operator on the Result<T, E> value; instead, use assert!(value.is_err()).

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

What cargo test does by default

Now that you know several ways to write tests, let’s look at what happens when we run them and explore the options we can use with cargo test.

Just as cargo run compiles your code and then runs the resultant binary, cargo test compiles your code in test mode and runs the resultant test binary. By default, that binary runs all the tests in parallel and captures output generated during test runs, preventing it from being displayed and making the output related to the test results easier to read. Command line options change this default behavior.

Some options go to cargo test, and some go to the resultant test binary. To separate them, list the arguments for cargo test, then the separator --, then the ones for the test binary. Each command below displays the options you can use on its side of the separator; they are also documented in the “Tests” section of The rustc Book.

$ cargo test --help
$ cargo test -- --help

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Running tests in parallel: no shared state

When you run multiple tests, by default they run in parallel using threads, meaning they finish running more quickly and you get feedback sooner.

Because the tests are running at the same time, you must make sure your tests don’t depend on each other or on any shared state, including a shared environment, such as the current working directory or environment variables.

Example 11.8. Each of your tests runs some code that creates a file on disk named test-output.txt and writes some data to it. Then each test reads the data in that file and asserts that the file contains a particular value, which is different in each test. Because the tests run at the same time, one test might overwrite the file between when another test writes and reads it. The second test will then fail — not because the code is incorrect, but because the tests interfered with each other while running in parallel.

One solution is to make sure each test writes to a different file; another solution is to run the tests one at a time.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Running tests consecutively with --test-threads

If you don’t want to run the tests in parallel, or if you want more fine-grained control over the number of threads used, send the --test-threads flag and the number of threads you want to use to the test binary:

$ cargo test -- --test-threads=1

We set the number of test threads to 1, telling the program not to use any parallelism. Running the tests using one thread will take longer than running them in parallel, but the tests won’t interfere with each other if they share state.

Remark. --test-threads comes after the -- separator because it is an option of the test binary, not of cargo test.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Showing function output: captured by default

By default, if a test passes, Rust’s test library captures anything printed to standard output. If we call println! in a test and the test passes, we won’t see the println! output in the terminal; we’ll see only the line that indicates the test passed. If a test fails, we’ll see whatever was printed to standard output with the rest of the failure message.

Listing 11.10 has a silly function that prints the value of its parameter and returns 10, as well as a test that passes and a test that fails.

Listing 11.10: Tests for a function that calls println! — src/lib.rs

fn prints_and_returns_10(a: i32) -> i32 {
    println!("I got the value {a}");
    10
}
 
#[cfg(test)]
mod tests {
    use super::*;

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

A passing and a failing test that both print

Listing 11.10 (continued)

 
    #[test]
    fn this_test_will_pass() {
        let value = prints_and_returns_10(4);
        assert_eq!(value, 10);
    }
 
    #[test]
    fn this_test_will_fail() {
        let value = prints_and_returns_10(8);
        assert_eq!(value, 5);
    }
}

The first test prints I got the value 4 and passes; the second prints I got the value 8 and fails, because 10 is not 5.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Output of Listing 11.10 with cargo test

When we run these tests with cargo test, we see the following output:

$ cargo test
   Compiling silly-function v0.1.0 (file:///projects/silly-function)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.58s
     Running unittests src/lib.rs (target/debug/deps/silly_function-160869f38cff9166)
 
running 2 tests
test tests::this_test_will_fail ... FAILED
test tests::this_test_will_pass ... ok
 
failures:
 
---- tests::this_test_will_fail stdout ----
I got the value 8
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Only the failing test’s output is shown

thread 'tests::this_test_will_fail' (6019863) panicked at src/lib.rs:19:9:
assertion `left == right` failed
  left: 10
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::this_test_will_fail
 
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

Nowhere in this output do we see I got the value 4, printed when the passing test runs: that output has been captured. The output from the failing test, I got the value 8, appears in the test summary output, which also shows the cause of the failure.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Showing output of passing tests: --show-output

To see printed values for passing tests as well, tell Rust to also show the output of successful tests:

$ cargo test -- --show-output

Running the tests in Listing 11.10 again with the --show-output flag:

$ cargo test -- --show-output
   Compiling silly-function v0.1.0 (file:///projects/silly-function)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.60s
     Running unittests src/lib.rs (target/debug/deps/silly_function-160869f38cff9166)
 
running 2 tests
test tests::this_test_will_fail ... FAILED
test tests::this_test_will_pass ... ok
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

A successes section appears

Now the passing test’s I got the value 4 is shown too (output continued):

successes:
 
---- tests::this_test_will_pass stdout ----
I got the value 4
 
successes:
    tests::this_test_will_pass
 
failures:
 
---- tests::this_test_will_fail stdout ----
I got the value 8

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

--show-output: the failure section is unchanged

The rest of the output is the same failure report as before:

thread 'tests::this_test_will_fail' (6022313) panicked at src/lib.rs:19:9:
assertion `left == right` failed
  left: 10
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 
failures:
    tests::this_test_will_fail
 
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
error: test failed, to rerun pass `--lib`

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Running a subset of tests by name

Running a full test suite can sometimes take a long time. If you’re working on code in a particular area, you might want to run only the tests pertaining to that code. You can choose which tests to run by passing cargo test the name or names of the test(s) you want to run as an argument.

To demonstrate, we first create three tests for our add_two function and choose which ones to run.

Listing 11.11: Three tests with three different names — src/lib.rs

pub fn add_two(a: u64) -> u64 {
    a + 2
}
 
#[cfg(test)]
mod tests {
    use super::*;
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Three tests with three different names

Listing 11.11 (continued)

    #[test]
    fn add_two_and_two() {
        let result = add_two(2);
        assert_eq!(result, 4);
    }
 
    #[test]
    fn add_three_and_two() {
        let result = add_two(3);

        assert_eq!(result, 5);
    }
 
    #[test]
    fn one_hundred() {
        let result = add_two(100);
        assert_eq!(result, 102);
    }
}

The three tests are named add_two_and_two, add_three_and_two and one_hundred. If we run the tests without passing any arguments, as we saw earlier, all the tests run in parallel (next slide).

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Without arguments, all three tests run

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.62s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 3 tests
test tests::add_three_and_two ... ok
test tests::add_two_and_two ... ok
test tests::one_hundred ... ok
 
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Running single tests

We can pass the name of any test function to cargo test to run only that test:

$ cargo test one_hundred
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::one_hundred ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

Only the test named one_hundred ran; the other two didn’t match that name. The output tells us we had more tests that didn’t run by displaying 2 filtered out at the end.

We can’t specify the names of multiple tests in this way; only the first value given to cargo test will be used. But there is a way to run multiple tests.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Filtering to run multiple tests

Passing part of a test name runs every test whose name matches that value — filtering. Two of our tests’ names contain add, so cargo test add runs those two:

$ cargo test add
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.61s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 2 tests
test tests::add_three_and_two ... ok
test tests::add_two_and_two ... ok
 
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

This ran all tests with add in the name and filtered out one_hundred. The module in which a test appears becomes part of the test’s name, so we can run all the tests in a module by filtering on the module’s name.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Ignoring tests unless specifically requested

Some tests can be very time-consuming to execute, so you might want to exclude them during most runs of cargo test. Rather than listing all the tests you do want to run, annotate the time-consuming ones with the ignore attribute:

src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_works() {

        let result = add(2, 2);
        assert_eq!(result, 4);
    }
 
    #[test]
    #[ignore]
    fn expensive_test() {
        // code that takes an hour to run
    }
}

After #[test], we add the #[ignore] line to the test we want to exclude.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

The ignored test does not run

Now when we run our tests, it_works runs, but expensive_test doesn’t — it is listed as ignored:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.60s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 2 tests
test tests::expensive_test ... ignored
test tests::it_works ... ok
 
test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Running only the ignored tests: --ignored

If we want to run only the ignored tests, we can use cargo test -- --ignored:

$ cargo test -- --ignored
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.61s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::expensive_test ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.2 Controlling How Tests Are Run

Keeping test runs fast, and a recap of the options

By controlling which tests run, you can make sure your cargo test results are returned quickly. When it makes sense to check the results of the ignored tests and you have time to wait, run cargo test -- --ignored instead. To run all tests whether they’re ignored or not, run cargo test -- --include-ignored.

Table 11.2: Ways to control a test run covered in this section

Command	Effect
cargo test -- --test-threads=1	Run tests one at a time instead of in parallel
cargo test -- --show-output	Also show what passing tests printed
cargo test one_hundred	Run only tests whose name matches the string
cargo test add	Filter: run every test whose name contains add
cargo test -- --ignored	Run only tests marked #[ignore]
cargo test -- --include-ignored	Run all tests, ignored or not

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Two categories: unit tests and integration tests

As mentioned at the start of the chapter, testing is a complex discipline, and different people use different terminology and organization. The Rust community thinks about tests in terms of two main categories:

Definition 11.9 (Unit tests). Small, more focused tests that test one module in isolation at a time, and can test private interfaces.

Definition 11.10 (Integration tests). Tests entirely external to your library that use your code in the same way any other external code would — using only the public interface and potentially exercising multiple modules per test.

Writing both kinds of tests is important to ensure that the pieces of your library are doing what you expect them to, separately and together.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Unit tests and the #[cfg(test)] tests module

The purpose of unit tests is to test each unit of code in isolation from the rest of the code, to quickly pinpoint where code is and isn’t working as expected. You put unit tests in the src directory, in each file with the code that they’re testing. The convention is to create a module named tests in each file to contain the test functions and to annotate the module with cfg(test).

The #[cfg(test)] annotation on the tests module tells Rust to compile and run the test code only when you run cargo test, not when you run cargo build. This saves compile time when you only want to build the library, and saves space in the resultant compiled artifact because the tests are not included.

Integration tests go in a different directory, so they don’t need the #[cfg(test)] annotation.
Unit tests go in the same files as the code, so you use #[cfg(test)] to specify that they shouldn’t be included in the compiled result.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

What cfg(test) means

Generated by Cargo for adder — src/lib.rs

pub fn add(left: u64, right: u64) -> u64 {
    left + right
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

Recall the code Cargo generated for the new adder project in the first section of this chapter (left).

On the generated tests module, the attribute cfg stands for configuration: it tells Rust that the following item should only be included given a certain configuration option. Here the option is test, which Rust provides for compiling and running tests.

So Cargo compiles our test code only if we actively run the tests with cargo test — including any helper functions within this module, in addition to the functions annotated with #[test].

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Testing private functions

There’s debate within the testing community about whether private functions should be tested directly, and other languages make it difficult or impossible to test them. Regardless of which testing ideology you adhere to, Rust’s privacy rules do allow you to test private functions — such as internal_adder here.

Listing 11.12: Testing a private function — src/lib.rs

pub fn add_two(a: u64) -> u64 {
    internal_adder(a, 2)
}
 
fn internal_adder(left: u64, right: u64) -> u64 {
    left + right
}
 

#[cfg(test)]
mod tests {
    use super::*;
 
    #[test]
    fn internal() {
        let result = internal_adder(2, 2);
        assert_eq!(result, 4);
    }
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Why the private test compiles; integration tests

In Listing 11.12, internal_adder is not marked as pub. Tests are just Rust code, and the tests module is just another module. As discussed in “Paths for Referring to an Item in the Module Tree” (Chapter 7), items in child modules can use the items in their ancestor modules. The test brings all of the items of the tests module’s parent into scope with use super::*, and then it can call internal_adder. If you don’t think private functions should be tested, there’s nothing in Rust that will compel you to do so.

Integration tests

In Rust, integration tests are entirely external to your library. They use your library in the same way any other code would, so they can only call functions that are part of your library’s public API. Their purpose is to test whether many parts of your library work together correctly: units of code that work correctly on their own could have problems when integrated, so test coverage of the integrated code is important as well. To create integration tests, you first need a tests directory.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

The tests directory

We create a tests directory at the top level of our project directory, next to src. Cargo knows to look for integration test files in this directory. We can then make as many test files as we want, and Cargo will compile each of the files as an individual crate.

With the code in Listing 11.12 still in src/lib.rs, make a tests directory and create tests/integration_test.rs, then enter the code in Listing 11.13 into it.

Directory structure

adder
├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs
└── tests
    └── integration_test.rs

Listing 11.13: An integration test of a function in the adder crate — tests/integration_test.rs

use adder::add_two;
 
#[test]
fn it_adds_two() {
    let result = add_two(2);
    assert_eq!(result, 4);
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Each test file is its own crate

Each file in the tests directory is a separate crate, so we need to bring our library into each test crate’s scope — hence use adder::add_two; at the top, which we didn’t need in the unit tests. We don’t annotate any code in tests/integration_test.rs with #[cfg(test)]: Cargo treats the tests directory specially and compiles its files only when we run cargo test. Run cargo test now:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1.31s
     Running unittests src/lib.rs (target/debug/deps/adder-1082c4b063a8fbe6)
 
running 1 test
test tests::internal ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Three sections: unit, integration, doc tests

(output continued)

     Running tests/integration_test.rs (target/debug/deps/integration_test-1082c4b063a8fbe6)
 
running 1 test
test it_adds_two ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The three sections of output include the unit tests, the integration test, and the doc tests. If any test in a section fails, the following sections will not be run: if a unit test fails, there won’t be any output for integration and doc tests, because those tests only run if all unit tests pass.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Reading the integration test output

The first section, for the unit tests, is the same as we’ve been seeing: one line for each unit test (one named internal, added in Listing 11.12) and then a summary line for the unit tests.
The integration tests section starts with the line Running tests/integration_test.rs. Next comes a line for each test function in that integration test, and a summary line for its results just before the Doc-tests adder section starts.
Each integration test file has its own section, so if we add more files in the tests directory, there will be more integration test sections.

Running one integration test file

We can still run a particular integration test function by giving its name as an argument to cargo test. To run all the tests in a particular integration test file, use the --test argument of cargo test followed by the name of the file (next slide).

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

cargo test --test integration_test

$ cargo test --test integration_test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.64s
     Running tests/integration_test.rs (target/debug/deps/integration_test-82e7799c1bc62298)
 
running 1 test
test it_adds_two ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

This command runs only the tests in the tests/integration_test.rs file: there is no unit test section and no doc-test section in the output.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Submodules in integration tests

As you add more integration tests, you might want more files in the tests directory to organize them — for example, grouping test functions by the functionality they test. Each file in tests is compiled as its own separate crate, which is useful for creating separate scopes that more closely imitate the way end users will use your crate. But it means files in tests don’t share the same behavior as files in src do (Chapter 7, separating code into modules and files).

This is most noticeable with helper functions used by multiple integration test files. Following “Separating Modules into Different Files” (Chapter 7) to extract them into a common module, we create tests/common.rs with a function named setup, holding code we want to call from multiple test functions in multiple test files:

tests/common.rs

pub fn setup() {
    // setup code specific to your library's tests would go here
}

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

An unwanted section for common.rs

Running the tests again shows a new section for the common.rs file, even though it contains no test functions and we never called setup from anywhere:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.89s
     Running unittests src/lib.rs (target/debug/deps/adder-92948b65e88960b4)
 
running 1 test
test tests::internal ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
     Running tests/common.rs (target/debug/deps/common-92948b65e88960b4)
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

“running 0 tests” is not what we wanted

 
     Running tests/integration_test.rs (target/debug/deps/integration_test-92948b65e88960b4)
 
running 1 test
test it_adds_two ... ok
 
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 
   Doc-tests adder
 
running 0 tests
 
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Having common appear in the test results with running 0 tests displayed for it is not what we wanted: we just wanted to share some code with the other integration test files. To avoid it, instead of creating tests/common.rs, we’ll create tests/common/mod.rs.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Shared helpers in tests/common/mod.rs

The project directory now looks like this:

├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs
└── tests
    ├── common
    │   └── mod.rs
    └── integration_test.rs

This is the older naming convention that Rust also understands (“Alternate File Paths”, Chapter 7). Naming the file this way tells Rust not to treat the common module as an integration test file.

When we move the setup code into tests/common/mod.rs and delete tests/common.rs, the section in the test output no longer appears. Files in subdirectories of the tests directory don’t get compiled as separate crates or have sections in the test output.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Calling common::setup from an integration test

tests/integration_test.rs

use adder::add_two;
 
mod common;
 
#[test]
fn it_adds_two() {
    common::setup();
 
    let result = add_two(2);
    assert_eq!(result, 4);
}

After we’ve created tests/common/mod.rs, we can use it from any of the integration test files as a module. Here the it_adds_two test calls the setup function.

The mod common; declaration is the same as the module declaration demonstrated in Listing 7-21. Then, in the test function, we can call the common::setup() function.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · 11.3 Test Organization

Integration tests for binary crates

If our project is a binary crate that only contains a src/main.rs file and doesn’t have a src/lib.rs file, we can’t create integration tests in the tests directory and bring functions defined in src/main.rs into scope with a use statement. Only library crates expose functions that other crates can use; binary crates are meant to be run on their own.

This is one of the reasons Rust projects that provide a binary have a straightforward src/main.rs file that calls logic that lives in the src/lib.rs file.

Using that structure, integration tests can test the library crate with use to make the important functionality available.
If the important functionality works, the small amount of code in src/main.rs will work as well, and that small amount of code doesn’t need to be tested.

The Rust Programming Language · Ch.11 Writing Automated Tests

Chapter 11 · Summary

Chapter summary

Rust’s testing features provide a way to specify how code should function, to ensure that it continues to work as you expect, even as you make changes.

Unit tests exercise different parts of a library separately and can test private implementation details.
Integration tests check that many parts of the library work together correctly, and use the library’s public API to test the code the same way external code will use it.
Even though Rust’s type system and ownership rules help prevent some kinds of bugs, tests are still important to reduce logic bugs having to do with how your code is expected to behave.

Toolbox: #[test], assert!, assert_eq!, assert_ne!, custom messages, #[should_panic(expected = …)], Result<T, E> tests, #[ignore], #[cfg(test)], and the tests directory.

Remark. Next, Chapter 12 combines the knowledge from this chapter and previous chapters to work on a project.

The Rust Programming Language · Ch.11 Writing Automated Tests