Chapter 20
Advanced Features
By now you’ve learned the most commonly used parts of Rust. Before one more project in Chapter 21, this chapter looks at aspects of the language you might run into every once in a while but may not use every day. Use it as a reference when you encounter any unknowns: these features are useful in very specific situations, and you should have a grasp of all the features Rust has to offer. It’s a panoply of Rust features with something for everyone!
- 20.1 Unsafe Rust — opting out of some of Rust’s guarantees and taking responsibility for manually upholding them
- 20.2 Advanced Traits — associated types, default type parameters, fully qualified syntax, supertraits, and the newtype pattern in relation to traits
- 20.3 Advanced Types — more about the newtype pattern, type aliases, the never type, and dynamically sized types
- 20.4 Advanced Functions and Closures — function pointers and returning closures
- 20.5 Macros — ways to define code that defines more code at compile time
The Rust Programming Language · Ch.20 Advanced Features
1
Chapter 20 · 20.1 Unsafe Rust
A second language hidden inside Rust
All the code so far has had Rust’s memory safety guarantees enforced at compile time. However, Rust has a second language hidden inside it that doesn’t enforce these guarantees.
Definition 20.1 (Unsafe Rust). A part of Rust that works just like regular Rust but doesn’t enforce the memory safety guarantees, giving us extra superpowers.
Why it exists
- Static analysis is conservative. When the compiler tries to determine whether code upholds the guarantees, it’s better to reject some valid programs than to accept some invalid ones. Code that might be okay is rejected if the compiler lacks the information to be confident. Unsafe code tells the compiler, “Trust me, I know what I’m doing.” You use it at your own risk: used incorrectly, it can cause memory unsafety problems such as null pointer dereferencing.
- The hardware is inherently unsafe. Without unsafe operations you couldn’t do certain tasks. Rust needs to allow low-level systems programming — directly interacting with the operating system, or even writing your own operating system — which is one of the goals of the language.
The Rust Programming Language · Ch.20 Advanced Features
2
Chapter 20 · 20.1 Unsafe Rust
Performing unsafe superpowers
To switch to unsafe Rust, use the unsafe keyword and then start a new block that holds the unsafe code.
Definition 20.2 (Unsafe superpowers). The five actions you can take in unsafe Rust that you can’t in safe Rust.
- Dereference a raw pointer.
- Call an unsafe function or method.
- Access or modify a mutable static variable.
- Implement an unsafe trait.
- Access fields of unions.
unsafe doesn’t turn off the borrow checker or disable any of Rust’s other safety checks: if you use a reference in unsafe code, it will still be checked. The keyword only gives you access to these five features, which are then not checked by the compiler for memory safety. You still get some degree of safety inside an unsafe block.
The Rust Programming Language · Ch.20 Advanced Features
3
Chapter 20 · 20.1 Unsafe Rust
What unsafe does not mean
unsafe does not mean the code inside the block is necessarily dangerous or that it will definitely have memory safety problems. The intent is that you, the programmer, ensure that the code inside an unsafe block accesses memory in a valid way.
People are fallible and mistakes will happen, but because these five operations must be inside blocks annotated with unsafe, you’ll know that any errors related to memory safety must be within an unsafe block. Keep unsafe blocks small; you’ll be thankful later when you investigate memory bugs.
Safe abstractions
To isolate unsafe code as much as possible, enclose it within a safe abstraction and provide a safe API (shown later with unsafe functions and methods). Parts of the standard library are implemented as safe abstractions over unsafe code that has been audited. Wrapping unsafe code in a safe abstraction prevents uses of unsafe from leaking out into all the places that you or your users might want to use the functionality, because using a safe abstraction is safe.
Next: each of the five superpowers in turn, plus some abstractions that provide a safe interface to unsafe code.
The Rust Programming Language · Ch.20 Advanced Features
4
Chapter 20 · 20.1 Unsafe Rust
Superpower 1: dereferencing a raw pointer
Chapter 4 (“Dangling References”) noted that the compiler ensures references are always valid. Unsafe Rust has two new types, similar to references:
Definition 20.3 (Raw pointers). Pointer types written *const T (immutable) and *mut T (mutable). The asterisk isn’t the dereference operator; it’s part of the type name. Here, immutable means the pointer can’t be directly assigned to after being dereferenced.
Different from references and smart pointers, raw pointers:
- are allowed to ignore the borrowing rules by having both immutable and mutable pointers or multiple mutable pointers to the same location;
- aren’t guaranteed to point to valid memory;
- are allowed to be null;
- don’t implement any automatic cleanup.
By opting out of having Rust enforce these guarantees, you give up guaranteed safety in exchange for greater performance or the ability to interface with another language or hardware where Rust’s guarantees don’t apply.
The Rust Programming Language · Ch.20 Advanced Features
5
Chapter 20 · 20.1 Unsafe Rust
Creating raw pointers in safe code
Listing 20.1 creates an immutable and a mutable raw pointer.
Listing 20.1: Creating raw pointers with the raw borrow operators
fn main() {
let mut num = 5;
let r1 = &raw const num;
let r2 = &raw mut num;
}
- There is no unsafe keyword here. We can create raw pointers in safe code; we just can’t dereference raw pointers outside an unsafe block, as you’ll see in a bit.
- The raw borrow operators: &raw const num creates a *const i32 immutable raw pointer, and &raw mut num creates a *mut i32 mutable raw pointer.
- Because we created them directly from a local variable, we know these particular raw pointers are valid — but we can’t make that assumption about just any raw pointer.
The Rust Programming Language · Ch.20 Advanced Features
6
Chapter 20 · 20.1 Unsafe Rust
A raw pointer to an arbitrary address
Next, a raw pointer whose validity we can’t be so certain of: instead of a raw borrow operator, we use the keyword as to cast a value. Listing 20.2 creates a raw pointer to an arbitrary location in memory.
Listing 20.2: Creating a raw pointer to an arbitrary memory address
fn main() {
let address = 0x012345usize;
let r = address as *const i32;
}
Trying to use arbitrary memory is undefined: there might be data at that address or there might not, the compiler might optimize the code so that there is no memory access, or the program might terminate with a segmentation fault.
Remark. Usually there is no good reason to write code like this, especially where you can use a raw borrow operator instead — but it is possible.
The Rust Programming Language · Ch.20 Advanced Features
7
Chapter 20 · 20.1 Unsafe Rust
Dereferencing raw pointers in an unsafe block
Listing 20.3: Dereferencing raw pointers within an unsafe block
fn main() {
let mut num = 5;
let r1 = &raw const num;
let r2 = &raw mut num;
unsafe {
println!("r1 is: {}", *r1);
println!("r2 is: {}", *r2);
}
}
Reading the data a raw pointer points to needs the dereference operator *, which requires an unsafe block.
Creating a pointer does no harm; only accessing the value it points at might mean dealing with an invalid value.
In Listings 20.1 and 20.3, *const i32 and *mut i32 both point to where num is stored. With an immutable and a mutable reference to num, the code would not compile. With raw pointers we can change data through the mutable pointer, potentially creating a data race. Be careful!
Why use raw pointers at all? One major use case is interfacing with C code (next section). Another is building up safe abstractions that the borrow checker doesn’t understand — shown after unsafe functions are introduced.
The Rust Programming Language · Ch.20 Advanced Features
8
Chapter 20 · 20.1 Unsafe Rust
Superpower 2: calling an unsafe function or method
Definition 20.4 (Unsafe function). A function or method that looks exactly like a regular one but has an extra unsafe before the rest of the definition. The keyword indicates the function has requirements we need to uphold when we call it, because Rust can’t guarantee we’ve met them.
By calling an unsafe function within an unsafe block, we’re saying that we’ve read this function’s documentation and we take responsibility for upholding the function’s contracts. Here is an unsafe function named dangerous that doesn’t do anything in its body:
unsafe fn dangerous() {}
unsafe {
dangerous();
}
We must call dangerous within a separate unsafe block. Calling it without the unsafe block is an error (next slide).
The Rust Programming Language · Ch.20 Advanced Features
9
Chapter 20 · 20.1 Unsafe Rust
Calling an unsafe function without unsafe
$ cargo run
Compiling unsafe-example v0.1.0 (file:///projects/unsafe-example)
error[E0133]: call to unsafe function `dangerous` is unsafe and requires unsafe block
--> src/main.rs:4:5
|
4 | dangerous();
| ^^^^^^^^^^^ call to unsafe function
|
= note: consult the function's documentation for information on how to avoid undefined behavior
For more information about this error, try `rustc --explain E0133`.
error: could not compile `unsafe-example` (bin "unsafe-example") due to 1 previous error
With the unsafe block, we’re asserting to Rust that we’ve read the function’s documentation, we understand how to use it properly, and we’ve verified that we’re fulfilling the contract of the function.
To perform unsafe operations in the body of an unsafe function, you still need an unsafe block, just as within a regular function, and the compiler warns you if you forget. This keeps unsafe blocks as small as possible, as unsafe operations may not be needed across the whole function body.
The Rust Programming Language · Ch.20 Advanced Features
10
Chapter 20 · 20.1 Unsafe Rust
Creating a safe abstraction over unsafe code
A function that contains unsafe code doesn’t need to be marked unsafe as a whole. In fact, wrapping unsafe code in a safe function is a common abstraction.
Example: split_at_mut from the standard library, which requires some unsafe code. This safe method is defined on mutable slices: it takes one slice and makes it two by splitting the slice at the index given as an argument.
Listing 20.4: Using the safe split_at_mut function
fn main() {
let mut v = vec![1, 2, 3, 4, 5, 6];
let r = &mut v[..];
let (a, b) = r.split_at_mut(3);
assert_eq!(a, &mut [1, 2, 3]);
assert_eq!(b, &mut [4, 5, 6]);
}
We can’t implement this function using only safe Rust. An attempt might look like Listing 20.5, which won’t compile. For simplicity, it implements split_at_mut as a function rather than a method, and only for slices of i32 values rather than for a generic type T.
The Rust Programming Language · Ch.20 Advanced Features
11
Chapter 20 · 20.1 Unsafe Rust
An attempt using only safe Rust
Listing 20.5: An attempted implementation of split_at_mut using only safe Rust
fn split_at_mut(values: &mut [i32], mid: usize) -> (&mut [i32], &mut [i32]) {
let len = values.len();
assert!(mid <= len);
(&mut values[..mid], &mut values[mid..])
}
fn main() {
let mut vector = vec![1, 2, 3, 4, 5, 6];
let (left, right) = split_at_mut(&mut vector, 3);
}
- First get the slice’s total length, then assert that the index is ≤ the length: given a larger index, the function panics before it attempts to use it.
- Then return two mutable slices in a tuple: one from the start of the original slice to mid, another from mid to the end.
The Rust Programming Language · Ch.20 Advanced Features
12
Chapter 20 · 20.1 Unsafe Rust
Compiling Listing 20.5: two mutable borrows
$ cargo run
Compiling unsafe-example v0.1.0 (file:///projects/unsafe-example)
error[E0499]: cannot borrow `*values` as mutable more than once at a time
--> src/main.rs:6:31
|
1 | fn split_at_mut(values: &mut [i32], mid: usize) -> (&mut [i32], &mut [i32]) {
| - let's call the lifetime of this reference `'1`
...
6 | (&mut values[..mid], &mut values[mid..])
| --------------------------^^^^^^--------
| | | |
| | | second mutable borrow occurs here
| | first mutable borrow occurs here
| returning this value requires that `*values` is borrowed for `'1`
|
= help: use `.split_at_mut(position)` to obtain two mutable non-overlapping sub-slices
For more information about this error, try `rustc --explain E0499`.
error: could not compile `unsafe-example` (bin "unsafe-example") due to 1 previous error
The Rust Programming Language · Ch.20 Advanced Features
13
Chapter 20 · 20.1 Unsafe Rust
Reaching for unsafe: split_at_mut
The borrow checker can’t understand that we’re borrowing different parts of the slice; it only knows we’re borrowing from the same slice twice. Borrowing non-overlapping parts is fundamentally okay, but Rust isn’t smart enough to know this. When we know code is okay but Rust doesn’t, it’s time to reach for unsafe code: Listing 20.6 uses an unsafe block, a raw pointer, and some calls to unsafe functions.
Listing 20.6: Using unsafe code in the implementation of the split_at_mut function
use std::slice;
fn split_at_mut(values: &mut [i32], mid: usize) -> (&mut [i32], &mut [i32]) {
let len = values.len();
let ptr = values.as_mut_ptr();
assert!(mid <= len);
unsafe {
(
slice::from_raw_parts_mut(ptr, mid),
slice::from_raw_parts_mut(ptr.add(mid), len - mid),
The Rust Programming Language · Ch.20 Advanced Features
14
Chapter 20 · 20.1 Unsafe Rust
How the unsafe split_at_mut works
Listing 20.6 (continued)
)
}
}
fn main() {
let mut vector = vec![1, 2, 3, 4, 5, 6];
let (left, right) = split_at_mut(&mut vector, 3);
}
Recall from Chapter 4 (“The Slice Type”) that a slice is a pointer to some data and the length of the slice.
len gets the length of a slice; as_mut_ptr accesses its raw pointer. For a mutable slice of i32 values, as_mut_ptr returns a *mut i32, stored in ptr.
We keep the assertion that mid is within the slice. Then the unsafe code: slice::from_raw_parts_mut takes a raw pointer and a length and creates a slice. We use it to create a slice that starts from ptr and is mid items long. Then we call add on ptr with mid as an argument to get a raw pointer that starts at mid, and create a slice from that pointer with the remaining number of items after mid as the length.
The Rust Programming Language · Ch.20 Advanced Features
15
Chapter 20 · 20.1 Unsafe Rust
Why this is an appropriate use of unsafe
- slice::from_raw_parts_mut is unsafe because it takes a raw pointer and must trust that this pointer is valid.
- The add method on raw pointers is also unsafe, because it must trust that the offset location is also a valid pointer.
- Therefore we had to put an unsafe block around the calls to slice::from_raw_parts_mut and add so that we could call them.
By looking at the code and by adding the assertion that mid must be less than or equal to len, we can tell that all the raw pointers used within the unsafe block will be valid pointers to data within the slice. This is an acceptable and appropriate use of unsafe.
We don’t need to mark the resultant split_at_mut function as unsafe, and we can call it from safe Rust. We’ve created a safe abstraction to the unsafe code: the implementation uses unsafe code in a safe way, because it creates only valid pointers from the data this function has access to.
The Rust Programming Language · Ch.20 Advanced Features
16
Chapter 20 · 20.1 Unsafe Rust
In contrast: a slice from an arbitrary location
The use of slice::from_raw_parts_mut in Listing 20.7 would likely crash when the slice is used. This code takes an arbitrary memory location and creates a slice 10,000 items long.
Listing 20.7: Creating a slice from an arbitrary memory location
fn main() {
use std::slice;
let address = 0x01234usize;
let r = address as *mut i32;
let values: &[i32] = unsafe { slice::from_raw_parts_mut(r, 10000) };
}
We don’t own the memory at this arbitrary location, and there is no guarantee that the slice this code creates contains valid i32 values. Attempting to use values as though it’s a valid slice results in undefined behavior.
The Rust Programming Language · Ch.20 Advanced Features
17
Chapter 20 · 20.1 Unsafe Rust
Using extern functions to call external code
Sometimes Rust code needs to interact with code written in another language. For this, Rust has the keyword extern.
Definition 20.5 (Foreign Function Interface, FFI). A way for a programming language to define functions and enable a different (foreign) programming language to call those functions.
Listing 20.8 sets up an integration with the abs function from the C standard library.
Listing 20.8: Declaring and calling an extern function defined in another language — src/main.rs
unsafe extern "C" {
fn abs(input: i32) -> i32;
}
fn main() {
unsafe {
println!("Absolute value of -3 according to C: {}", abs(-3));
}
}
The Rust Programming Language · Ch.20 Advanced Features
18
Chapter 20 · 20.1 Unsafe Rust
The ABI, and safe functions in unsafe extern blocks
Functions declared within extern blocks are generally unsafe to call from Rust, so extern blocks must also be marked unsafe: other languages don’t enforce Rust’s rules and guarantees, and Rust can’t check them, so responsibility falls on the programmer.
Within the unsafe extern "C" block, we list the names and signatures of external functions from another language we want to call. The "C" part defines which ABI the external function uses.
Definition 20.6 (Application binary interface, ABI). The definition of how to call a function at the assembly level. The "C" ABI is the most common and follows the C programming language’s ABI. All the ABIs Rust supports are listed in the Rust Reference.
Every item declared within an unsafe extern block is implicitly unsafe. However, some FFI functions are safe to call. For example, abs from C’s standard library has no memory safety considerations, and we know it can be called with any i32.
The Rust Programming Language · Ch.20 Advanced Features
19
Chapter 20 · 20.1 Unsafe Rust
Marking an FFI function as safe
For a function like abs, the safe keyword says that this specific function is safe to call even though it is in an unsafe extern block. Once we make that change, calling it no longer requires an unsafe block.
Listing 20.9: Explicitly marking a function as safe within an unsafe extern block and calling it safely — src/main.rs
unsafe extern "C" {
safe fn abs(input: i32) -> i32;
}
fn main() {
println!("Absolute value of -3 according to C: {}", abs(-3));
}
Remark. Marking a function as safe does not inherently make it safe! Instead, it is like a promise you are making to Rust that it is safe. It is still your responsibility to make sure that promise is kept!
The Rust Programming Language · Ch.20 Advanced Features
20
Chapter 20 · 20.1 Unsafe Rust
Calling Rust functions from other languages
extern can also create an interface that lets other languages call Rust functions. Instead of a whole extern block, add the extern keyword and the ABI just before the fn keyword of the function, plus an #[unsafe(no_mangle)] annotation telling the compiler not to mangle its name.
Definition 20.7 (Mangling). A compiler changing the name we’ve given a function to a different name that contains more information for other parts of the compilation process to consume but is less human readable.
Every compiler mangles names slightly differently, so for a Rust function to be nameable by other languages, we must disable Rust’s name mangling. This is unsafe because there might be name collisions across libraries without the built-in mangling: it is our responsibility to make sure the name we choose is safe to export without mangling.
Here call_from_c becomes accessible from C code, after it’s compiled to a shared library and linked from C. This usage of extern requires unsafe only in the attribute, not on the extern block.
#[unsafe(no_mangle)]
pub extern "C" fn call_from_c() {
println!("Just called a Rust function from C!");
}
The Rust Programming Language · Ch.20 Advanced Features
21
Chapter 20 · 20.1 Unsafe Rust
Superpower 3: static variables
Definition 20.8 (Static variable). Rust’s name for a global variable. Rust supports them, but they can be problematic with Rust’s ownership rules: if two threads are accessing the same mutable global variable, it can cause a data race.
Listing 20.10: Defining and using an immutable static variable — src/main.rs
static HELLO_WORLD: &str = "Hello, world!";
fn main() {
println!("value is: {HELLO_WORLD}");
}
- Static variables are similar to constants (Chapter 3, “Declaring Constants”). Their names are in SCREAMING_SNAKE_CASE by convention.
- They can only store references with the 'static lifetime, so the compiler can figure out the lifetime and we needn’t annotate it.
- Accessing an immutable static variable is safe.
The Rust Programming Language · Ch.20 Advanced Features
22
Chapter 20 · 20.1 Unsafe Rust
Constants versus static variables
A subtle difference between constants and immutable static variables is that values in a static variable have a fixed address in memory: using the value will always access the same data. Constants, on the other hand, are allowed to duplicate their data whenever they’re used. Another difference is that static variables can be mutable — and accessing and modifying mutable static variables is unsafe.
Table 20.1: Constants and static variables compared
| Property | Constant (const) | Static variable (static) |
|---|---|---|
| Memory address | Data may be duplicated wherever it’s used | Fixed address; always the same data |
| Mutability | Never mutable | Can be mutable (static mut) |
| Accessing | Safe | Immutable: safe; mutable: unsafe |
| Naming | SCREAMING_SNAKE_CASE | SCREAMING_SNAKE_CASE |
Listing 20.11 (next slide) declares, accesses, and modifies a mutable static variable named COUNTER.
The Rust Programming Language · Ch.20 Advanced Features
23
Chapter 20 · 20.1 Unsafe Rust
A mutable static variable
Listing 20.11: Reading from or writing to a mutable static variable is unsafe. — src/main.rs
static mut COUNTER: u32 = 0;
/// SAFETY: Calling this from more than a single thread at a time is undefined
/// behavior, so you *must* guarantee you only call it from a single thread at
/// a time.
unsafe fn add_to_count(inc: u32) {
unsafe {
COUNTER += inc;
}
}
fn main() {
unsafe {
// SAFETY: This is only called from a single thread in `main`.
add_to_count(3);
println!("COUNTER: {}", *(&raw const COUNTER));
}
}
The Rust Programming Language · Ch.20 Advanced Features
24
Chapter 20 · 20.1 Unsafe Rust
Rules for mutable static variables
- As with regular variables, mutability is specified with mut. Any code that reads or writes COUNTER must be within an unsafe block.
- Listing 20.11 compiles and prints COUNTER: 3 as expected because it’s single threaded. Multiple threads accessing COUNTER would likely result in data races, so it is undefined behavior. Therefore the entire function is marked unsafe, with the safety limitation documented, so callers know what they are and are not allowed to do safely.
- Whenever we write an unsafe function, it is idiomatic to write a comment starting with SAFETY explaining what the caller needs to do to call it safely; whenever we perform an unsafe operation, a SAFETY comment explains how the safety rules are upheld.
- A compiler lint denies by default any attempt to create references to a mutable static. Either opt out with #[allow(static_mut_refs)] or access it via a raw pointer from a raw borrow operator — including where a reference is created invisibly, as in this listing’s println!. This makes the safety requirements more obvious.
Globally accessible mutable data makes data races hard to rule out — hence mutable statics are unsafe. Where possible, prefer Chapter 16’s concurrency techniques and thread-safe smart pointers, so the compiler checks cross-thread access.
The Rust Programming Language · Ch.20 Advanced Features
25
Chapter 20 · 20.1 Unsafe Rust
Superpower 4: implementing an unsafe trait
Definition 20.9 (Unsafe trait). A trait at least one of whose methods has some invariant that the compiler can’t verify. It is declared by adding unsafe before trait, and its implementation must be marked unsafe too.
Listing 20.12: Defining and implementing an unsafe trait
unsafe trait Foo {
// methods go here
}
unsafe impl Foo for i32 {
// method implementations go here
}
fn main() {}
By using unsafe impl, we’re promising that we’ll uphold the invariants that the compiler can’t verify.
Example: the Send and Sync marker traits (Chapter 16). The compiler implements them automatically for types composed entirely of Send and Sync types. To mark a type that contains a non-Send/Sync type, such as raw pointers, as Send or Sync, we must use unsafe: Rust can’t verify the guarantees, so we check them manually.
The Rust Programming Language · Ch.20 Advanced Features
26
Chapter 20 · 20.1 Unsafe Rust
Superpower 5: union fields — and checking with Miri
Accessing fields of a union
Definition 20.10 (Union). A type similar to a struct, but only one declared field is used in a particular instance at one time.
Unions are primarily used to interface with unions in C code. Accessing union fields is unsafe because Rust can’t guarantee the type of the data currently stored in the union instance (more: the Rust Reference).
Using Miri to check unsafe code
Definition 20.11 (Miri). An official Rust tool for detecting undefined behavior. Whereas the borrow checker is a static tool that works at compile time, Miri is a dynamic tool that works at runtime: it runs your program, or its test suite, and detects when you violate the rules it understands about how Rust should work.
Miri is one of the best ways to check that unsafe code you’ve written actually is safe and correct.
The Rust Programming Language · Ch.20 Advanced Features
27
Chapter 20 · 20.1 Unsafe Rust
Running Miri on Listing 20.7
Miri requires a nightly build of Rust (Appendix G). rustup +nightly component add miri installs both a nightly Rust and Miri; this doesn’t change the Rust version your project uses, it only adds the tool to your system. Run it with cargo +nightly miri run or cargo +nightly miri test. For an example of how helpful this can be, run it against Listing 20.7:
$ cargo +nightly miri run
Compiling unsafe-example v0.1.0 (file:///projects/unsafe-example)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
Running `file:///home/.rustup/toolchains/nightly/bin/cargo-miri runner
target/miri/debug/unsafe-example`
warning: integer-to-pointer cast
--> src/main.rs:5:13
|
5 | let r = address as *mut i32;
| ^^^^^^^^^^^^^^^^^^^ integer-to-pointer cast
|
...
The Rust Programming Language · Ch.20 Advanced Features
28
Chapter 20 · 20.1 Unsafe Rust
Miri’s warning: an integer-to-pointer cast
Miri output (continued): the help lines of the warning
= help: this program is using integer-to-pointer casts or (equivalently)
`ptr::with_exposed_provenance`, which means that Miri might miss pointer bugs in this program
= help: see https://doc.rust-lang.org/nightly/std/ptr/fn.with_exposed_provenance.html
for more details on that operation
= help: to ensure that Miri does not miss bugs in your program, use Strict Provenance APIs
(https://doc.rust-lang.org/nightly/std/ptr/index.html#strict-provenance,
https://crates.io/crates/sptr) instead
= help: you can then set `MIRIFLAGS=-Zmiri-strict-provenance` to ensure you are not relying
on `with_exposed_provenance` semantics
= help: alternatively, `MIRIFLAGS=-Zmiri-permissive-provenance` disables this warning
Miri correctly warns us that we’re casting an integer to a pointer, which might be a problem — but Miri can’t determine whether a problem exists, because it doesn’t know how the pointer originated.
The Rust Programming Language · Ch.20 Advanced Features
29
Chapter 20 · 20.1 Unsafe Rust
Miri’s error: undefined behavior
error: Undefined Behavior: constructing invalid value of type &mut [i32]: encountered a dangling
reference (0x1234[noalloc] has no provenance)
--> src/main.rs:7:35
|
7 | let values: &[i32] = unsafe { slice::from_raw_parts_mut(r, 10000) };
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| Undefined Behavior occurred here
|
= help: this indicates a bug in the program: it performed an invalid operation, and caused
Undefined Behavior
= help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html
for further information
note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace
error: aborting due to 1 previous error; 1 warning emitted
Then Miri returns an error where Listing 20.7 has undefined behavior, because we have a dangling pointer.
The Rust Programming Language · Ch.20 Advanced Features
30
Chapter 20 · 20.1 Unsafe Rust
What Miri can and cannot tell you
Thanks to Miri, we now know there is a risk of undefined behavior in Listing 20.7, and we can think about how to make the code safe. In some cases, Miri can even make recommendations about how to fix errors.
Miri doesn’t catch everything you might get wrong when writing unsafe code:
- It is a dynamic analysis tool, so it only catches problems with code that actually gets run. Use it in conjunction with good testing techniques to increase your confidence in your unsafe code.
- It does not cover every possible way your code can be unsound.
Put another way: if Miri does catch a problem, you know there’s a bug, but just because Miri doesn’t catch a bug doesn’t mean there isn’t a problem. It can catch a lot, though. Try running it on the other examples of unsafe code in this chapter and see what it says! More about Miri: its GitHub repository (github.com/rust-lang/miri).
The Rust Programming Language · Ch.20 Advanced Features
31
Chapter 20 · 20.1 Unsafe Rust
Using unsafe code correctly
Using unsafe to use one of the five superpowers isn’t wrong or even frowned upon, but it is trickier to get unsafe code correct because the compiler can’t help uphold memory safety.
- When you have a reason to use unsafe code, you can do so.
- The explicit unsafe annotation makes it easier to track down the source of problems when they occur.
- Whenever you write unsafe code, Miri can help you be more confident that it upholds Rust’s rules.
Remark. For a much deeper exploration of how to work effectively with unsafe Rust, read Rust’s official guide for unsafe, The Rustonomicon (doc.rust-lang.org/nomicon).
The Rust Programming Language · Ch.20 Advanced Features
32
Chapter 20 · 20.2 Advanced Traits
Defining traits with associated types
Traits were first covered in Chapter 10 (“Defining Shared Behavior with Traits”), without the more advanced details. Now we can get into the nitty-gritty.
Definition 20.12 (Associated type). A type placeholder connected with a trait, so that the trait’s method definitions can use it in their signatures. The implementor of the trait specifies the concrete type used in place of the placeholder for the particular implementation. That way a trait can use some types without needing to know exactly what they are until the trait is implemented.
Most of the advanced features in this chapter are rarely needed. Associated types are somewhere in the middle: they’re used more rarely than features explained in the rest of the book, but more commonly than many of the other features discussed in this chapter.
The Rust Programming Language · Ch.20 Advanced Features
33
Chapter 20 · 20.2 Advanced Traits
Example: Iterator and its associated type Item
The standard library’s Iterator trait has an associated type named Item, which stands in for the type of the values the implementing type is iterating over.
Listing 20.13: The definition of the Iterator trait that has an associated type Item
pub trait Iterator {
type Item;
fn next(&mut self) -> Option<Self::Item>;
}
The type Item is a placeholder, and next returns values of type Option<Self::Item>. Implementors of Iterator specify the concrete type for Item, and next returns an Option containing a value of that concrete type.
Associated types might seem similar to generics, which also let us define a function without specifying what types it can handle. To examine the difference, the next slide implements Iterator on a type named Counter that specifies the Item type is u32.
The Rust Programming Language · Ch.20 Advanced Features
34
Chapter 20 · 20.2 Advanced Traits
Implementing Iterator for Counter
src/lib.rs
struct Counter {
count: u32,
}
impl Counter {
fn new() -> Counter {
Counter { count: 0 }
}
}
impl Iterator for Counter {
type Item = u32;
fn next(&mut self) -> Option<Self::Item> {
// --snip--
if self.count < 5 {
self.count += 1;
Some(self.count)
} else {
None
}
}
}
This syntax seems comparable to that of generics. So why not just define the Iterator trait with generics, as in Listing 20.14?
The Rust Programming Language · Ch.20 Advanced Features
35
Chapter 20 · 20.2 Advanced Traits
Associated types versus generics
Listing 20.14: A hypothetical definition of the Iterator trait using generics
pub trait Iterator<T> {
fn next(&mut self) -> Option<T>;
}
With generics
We must annotate the types in each implementation. Because we could also implement Iterator<String> for Counter or any other type, there could be multiple implementations of Iterator for Counter, each with different concrete types. Using next on Counter would need type annotations to pick the implementation.
With associated types
We can’t implement a trait on a type multiple times, so no annotations are needed. With Listing 20.13 we choose the type of Item only once, because there can be only one impl Iterator for Counter; we don’t specify an iterator of u32 everywhere we call next.
Associated types are also part of the trait’s contract: implementors must provide the stand-in type. They often have a name describing how the type will be used; documenting them in the API docs is good practice.
The Rust Programming Language · Ch.20 Advanced Features
36
Chapter 20 · 20.2 Advanced Traits
Default generic parameters and operator overloading
When we use generic type parameters, we can specify a default concrete type for the generic type. This eliminates the need for implementors of the trait to specify a concrete type if the default type works. You specify a default type when declaring a generic type with the <PlaceholderType=ConcreteType> syntax.
A great example of where this technique is useful is operator overloading.
Definition 20.13 (Operator overloading). Customizing the behavior of an operator (such as +) in particular situations.
Rust doesn’t allow you to create your own operators or overload arbitrary operators. But you can overload the operations and corresponding traits listed in std::ops by implementing the traits associated with the operator. Listing 20.15 (next slide) overloads the + operator to add two Point instances together, by implementing the Add trait on a Point struct.
The Rust Programming Language · Ch.20 Advanced Features
37
Chapter 20 · 20.2 Advanced Traits
Overloading + for Point
Listing 20.15: Implementing the Add trait to overload the + operator for Point instances — src/main.rs
use std::ops::Add;
#[derive(Debug, Copy, Clone, PartialEq)]
struct Point {
x: i32,
y: i32,
}
impl Add for Point {
type Output = Point;
fn add(self, other: Point) -> Point {
Point {
x: self.x + other.x,
y: self.y + other.y,
}
}
}
fn main() {
assert_eq!(
Point { x: 1, y: 0 } + Point { x: 2, y: 3 },
Point { x: 3, y: 3 }
);
}
add sums the x values and the y values of two Point instances into a new Point. The Add trait’s associated type Output determines the type returned from add.
The Rust Programming Language · Ch.20 Advanced Features
38
Chapter 20 · 20.2 Advanced Traits
The Add trait and Rhs=Self
The default generic type in Listing 20.15 is within the Add trait. Here is its definition:
trait Add<Rhs=Self> {
type Output;
fn add(self, rhs: Rhs) -> Self::Output;
}
This should look generally familiar: a trait with one method and an associated type. The new part is Rhs=Self.
Definition 20.14 (Default type parameters). The syntax Rhs=Self, which gives a generic type parameter a default concrete type.
The Rhs generic type parameter (short for “right-hand side”) defines the type of the rhs parameter in add. If we don’t specify a concrete type for Rhs when implementing Add, it defaults to Self, the type we’re implementing Add on. For Point we used the default, because we wanted to add two Point instances.
The Rust Programming Language · Ch.20 Advanced Features
39
Chapter 20 · 20.2 Advanced Traits
Customizing Rhs: adding Millimeters and Meters
Listing 20.16: Implementing the Add trait on Millimeters to add Millimeters and Meters — src/lib.rs
use std::ops::Add;
struct Millimeters(u32);
struct Meters(u32);
impl Add<Meters> for Millimeters {
type Output = Millimeters;
fn add(self, other: Meters) -> Millimeters {
Millimeters(self.0 + (other.0 * 1000))
}
}
Now an example where we customize the Rhs type rather than using the default. Two structs, Millimeters and Meters, hold values in different units.
This thin wrapping of an existing type in another struct is the newtype pattern, described in more detail later in this section.
We want to add millimeters to meters and have Add do the conversion correctly. So we specify impl Add<Meters>, setting the Rhs type parameter instead of using the default of Self.
The Rust Programming Language · Ch.20 Advanced Features
40
Chapter 20 · 20.2 Advanced Traits
Why default type parameters; same-named methods
You’ll use default type parameters in two main ways:
- To extend a type without breaking existing code: if you add a type parameter to an existing trait, a default allows extending the trait’s functionality without breaking the existing implementation code.
- To allow customization in specific cases most users won’t need. The standard library’s Add is an example: usually you add two like types, but Add lets you customize beyond that. With the default, you don’t have to specify the extra parameter most of the time — a bit of implementation boilerplate isn’t needed, making the trait easier to use.
Disambiguating between identically named methods
Nothing in Rust prevents a trait from having a method with the same name as another trait’s method, nor implementing both traits on one type. A type can also have a method of its own with the same name. When calling methods with the same name, you need to tell Rust which one you want. In Listing 20.17, traits Pilot and Wizard both have a method fly; both are implemented on a type Human that already has its own fly. Each fly does something different.
The Rust Programming Language · Ch.20 Advanced Features
41
Chapter 20 · 20.2 Advanced Traits
Three methods named fly
Listing 20.17: Two traits are defined to have a fly method and are implemented on the Human type, and a fly method is implemented on Human directly. — src/main.rs
trait Pilot {
fn fly(&self);
}
trait Wizard {
fn fly(&self);
}
struct Human;
impl Pilot for Human {
fn fly(&self) {
println!("This is your captain speaking.");
}
}
impl Wizard for Human {
fn fly(&self) {
println!("Up!");
}
}
impl Human {
fn fly(&self) {
println!("*waving arms furiously*");
}
}
fn main() {}
The Rust Programming Language · Ch.20 Advanced Features
42
Chapter 20 · 20.2 Advanced Traits
Calling each fly
With the traits, struct and impls of Listing 20.17 unchanged, only main differs below.
Listing 20.18: Calling fly on an instance of Human — src/main.rs
fn main() {
let person = Human;
person.fly();
}
The compiler defaults to calling the method implemented directly on the type: this prints *waving arms furiously*.
Listing 20.19: Specifying which trait’s fly method we want to call — src/main.rs
fn main() {
let person = Human;
Pilot::fly(&person);
Wizard::fly(&person);
person.fly();
}
Specifying the trait name before the method name clarifies to Rust which implementation of fly we want. Human::fly(&person) is equivalent to person.fly(), but a bit longer to write if we don’t need to disambiguate.
The Rust Programming Language · Ch.20 Advanced Features
43
Chapter 20 · 20.2 Advanced Traits
Running Listing 20.19; functions without self
Running Listing 20.19 prints the following:
$ cargo run
Compiling traits-example v0.1.0 (file:///projects/traits-example)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.46s
Running `target/debug/traits-example`
This is your captain speaking.
Up!
*waving arms furiously*
Because fly takes a self parameter, if we had two types that both implement one trait, Rust could figure out which implementation to use based on the type of self.
But associated functions that are not methods have no self. When several types or traits define non-method functions with the same name, Rust doesn’t always know which type you mean unless you use fully qualified syntax. In Listing 20.20, an animal shelter names all baby dogs Spot: trait Animal has an associated non-method function baby_name, implemented for struct Dog, which also has its own baby_name.
The Rust Programming Language · Ch.20 Advanced Features
44
Chapter 20 · 20.2 Advanced Traits
Two associated functions named baby_name
Listing 20.20: A trait with an associated function and a type with an associated function of the same name that also implements the trait — src/main.rs
trait Animal {
fn baby_name() -> String;
}
struct Dog;
impl Dog {
fn baby_name() -> String {
String::from("Spot")
}
}
impl Animal for Dog {
fn baby_name() -> String {
String::from("puppy")
}
}
fn main() {
println!("A baby dog is called a {}", Dog::baby_name());
}
The Rust Programming Language · Ch.20 Advanced Features
45
Chapter 20 · 20.2 Advanced Traits
Dog::baby_name calls the type’s own function
Naming all puppies Spot is done by Dog’s own baby_name. Dog also implements Animal, which describes characteristics all animals have; baby dogs are called puppies, expressed in Animal’s baby_name on Dog.
In main, we call Dog::baby_name, which calls the associated function defined on Dog directly. This prints:
$ cargo run
Compiling traits-example v0.1.0 (file:///projects/traits-example)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.54s
Running `target/debug/traits-example`
A baby dog is called a Spot
This output isn’t what we wanted: we want the baby_name function that is part of the Animal trait implemented on Dog, so that the code prints A baby dog is called a puppy. Specifying the trait name, as in Listing 20.19, doesn’t help here: changing main to Listing 20.21 gives a compilation error.
The Rust Programming Language · Ch.20 Advanced Features
46
Chapter 20 · 20.2 Advanced Traits
The trait name alone isn’t enough
With Animal, Dog and both impls as in Listing 20.20, main becomes:
Listing 20.21: Attempting to call the baby_name function from the Animal trait, but Rust doesn’t know which implementation to use — src/main.rs
fn main() {
println!("A baby dog is called a {}", Animal::baby_name());
}
Because Animal::baby_name doesn’t have a self parameter, and there could be other types that implement the Animal trait, Rust can’t figure out which implementation of Animal::baby_name we want. We get compiler error E0790 (next slide): “cannot call associated function on trait without specifying the corresponding impl type.”
Remark. The compiler’s help already points at the fix: use the fully qualified path to the only available implementation, <Dog as Animal>::baby_name().
The Rust Programming Language · Ch.20 Advanced Features
47
Chapter 20 · 20.2 Advanced Traits
Compiling Listing 20.21: error E0790
$ cargo run
Compiling traits-example v0.1.0 (file:///projects/traits-example)
error[E0790]: cannot call associated function on trait without specifying the corresponding `impl` type
--> src/main.rs:20:43
|
2 | fn baby_name() -> String;
| ------------------------- `Animal::baby_name` defined here
...
20 | println!("A baby dog is called a {}", Animal::baby_name());
| ^^^^^^^^^^^^^^^^^^^ cannot call associated function of trait
|
help: use the fully-qualified path to the only available implementation
|
20 | println!("A baby dog is called a {}", <Dog as Animal>::baby_name());
| +++++++ +
For more information about this error, try `rustc --explain E0790`.
error: could not compile `traits-example` (bin "traits-example") due to 1 previous error
The Rust Programming Language · Ch.20 Advanced Features
48
Chapter 20 · 20.2 Advanced Traits
Using fully qualified syntax
To pick Animal for Dog over Animal for other types, use fully qualified syntax:
Listing 20.22: Using fully qualified syntax to specify that we want to call the baby_name function from the Animal trait as implemented on Dog — src/main.rs
fn main() {
println!("A baby dog is called a {}", <Dog as Animal>::baby_name());
}
The type annotation in angle brackets treats the Dog type as an Animal for this call, so Animal’s baby_name as implemented on Dog runs. It now prints what we want:
$ cargo run
Compiling traits-example v0.1.0 (file:///projects/traits-example)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.48s
Running `target/debug/traits-example`
A baby dog is called a puppy
The Rust Programming Language · Ch.20 Advanced Features
49
Chapter 20 · 20.2 Advanced Traits
Fully qualified syntax in general
Definition 20.15 (Fully qualified syntax). A call that names both the type and the trait whose implementation is meant, in the form below.
<Type as Trait>::function(receiver_if_method, next_arg, ...);
- For associated functions that aren’t methods, there is no receiver: only the list of other arguments.
- You could use fully qualified syntax everywhere you call functions or methods. However, you may omit any part of it that Rust can figure out from other information in the program.
- You only need this more verbose syntax where there are multiple implementations that use the same name and Rust needs help to identify which implementation you want to call.
The Rust Programming Language · Ch.20 Advanced Features
50
Chapter 20 · 20.2 Advanced Traits
Using supertraits
Sometimes a trait definition depends on another trait: for a type to implement the first trait, you want to require that it also implements the second, so that your trait definition can use the second trait’s associated items.
Definition 20.16 (Supertrait). The trait your trait definition is relying on.
Example: an OutlinePrint trait with an outline_print method that prints a value framed in asterisks. Given a Point that implements the standard library trait Display to result in (x, y), calling outline_print on a Point with 1 for x and 3 for y should print the output on the right.
**********
* *
* (1, 3) *
* *
**********
The implementation of outline_print needs Display’s functionality, so OutlinePrint must work only for types that also implement Display. We specify that in the trait definition as OutlinePrint: Display — similar to adding a trait bound to the trait.
The Rust Programming Language · Ch.20 Advanced Features
51
Chapter 20 · 20.2 Advanced Traits
OutlinePrint requires Display
Listing 20.23: Implementing the OutlinePrint trait that requires the functionality from Display — src/main.rs
use std::fmt;
trait OutlinePrint: fmt::Display {
fn outline_print(&self) {
let output = self.to_string();
let len = output.len();
println!("{}", "*".repeat(len + 4));
println!("*{}*", " ".repeat(len + 2));
println!("* {output} *");
println!("*{}*", " ".repeat(len + 2));
println!("{}", "*".repeat(len + 4));
}
}
fn main() {}
Because OutlinePrint requires the Display trait, we can use the to_string function that is automatically implemented for any type that implements Display.
Without the colon and Display after the trait name, using to_string would give an error saying that no method named to_string was found for the type &Self in the current scope.
The Rust Programming Language · Ch.20 Advanced Features
52
Chapter 20 · 20.2 Advanced Traits
Implementing OutlinePrint on a type without Display
What happens when we implement OutlinePrint on a type that doesn’t implement Display, such as the Point struct? Keeping use std::fmt; and the OutlinePrint trait of Listing 20.23, we add:
src/main.rs
struct Point {
x: i32,
y: i32,
}
impl OutlinePrint for Point {}
fn main() {
let p = Point { x: 1, y: 3 };
p.outline_print();
}
We get an error saying that Display is required but not implemented (next two slides).
The Rust Programming Language · Ch.20 Advanced Features
53
Chapter 20 · 20.2 Advanced Traits
E0277: Point doesn’t implement Display (1 of 2)
$ cargo run
Compiling traits-example v0.1.0 (file:///projects/traits-example)
error[E0277]: `Point` doesn't implement `std::fmt::Display`
--> src/main.rs:20:23
|
20 | impl OutlinePrint for Point {}
| ^^^^^ unsatisfied trait bound
|
help: the trait `std::fmt::Display` is not implemented for `Point`
--> src/main.rs:15:1
|
15 | struct Point {
| ^^^^^^^^^^^^
note: required by a bound in `OutlinePrint`
--> src/main.rs:3:21
|
3 | trait OutlinePrint: fmt::Display {
| ^^^^^^^^^^^^ required by this bound in `OutlinePrint`
The Rust Programming Language · Ch.20 Advanced Features
54
Chapter 20 · 20.2 Advanced Traits
E0277: Point doesn’t implement Display (2 of 2)
The second error is at the call site; its help block (elided as ...) repeats the first error’s.
error[E0277]: `Point` doesn't implement `std::fmt::Display`
--> src/main.rs:24:7
|
24 | p.outline_print();
| ^^^^^^^^^^^^^ unsatisfied trait bound
|
...
note: required by a bound in `OutlinePrint::outline_print`
--> src/main.rs:3:21
|
3 | trait OutlinePrint: fmt::Display {
| ^^^^^^^^^^^^ required by this bound in `OutlinePrint::outline_print`
4 | fn outline_print(&self) {
| ------------- required by a bound in this associated function
For more information about this error, try `rustc --explain E0277`.
error: could not compile `traits-example` (bin "traits-example") due to 2 previous errors
The Rust Programming Language · Ch.20 Advanced Features
55
Chapter 20 · 20.2 Advanced Traits
Satisfying the supertrait: implement Display
To fix this, implement Display on Point (src/main.rs; trait and struct unchanged):
impl OutlinePrint for Point {}
use std::fmt;
impl fmt::Display for Point {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "({}, {})", self.x, self.y)
}
}
fn main() {
let p = Point { x: 1, y: 3 };
p.outline_print();
}
Now implementing OutlinePrint on Point compiles, and outline_print displays a Point within an outline of asterisks.
The Rust Programming Language · Ch.20 Advanced Features
56
Chapter 20 · 20.2 Advanced Traits
Implementing external traits with the newtype pattern
Chapter 10 (“Implementing a Trait on a Type”) introduced the orphan rule: we may implement a trait on a type only if the trait or the type, or both, are local to our crate. The newtype pattern gets around this restriction.
Definition 20.17 (Newtype pattern). Creating a new type in a tuple struct (Chapter 5) with one field: a thin wrapper around the type for which we want to implement a trait. The wrapper type is local to our crate, so we can implement the trait on it. Newtype is a term that originates from the Haskell programming language.
- There is no runtime performance penalty for using this pattern: the wrapper type is elided at compile time.
- Example: we want to implement Display on Vec<T>, which the orphan rule prevents directly because Display and Vec<T> are both defined outside our crate.
- Instead we make a Wrapper struct that holds an instance of Vec<T>, then implement Display on Wrapper and use the Vec<T> value (Listing 20.24, next slide).
The Rust Programming Language · Ch.20 Advanced Features
57
Chapter 20 · 20.2 Advanced Traits
A Wrapper around Vec<String>
Listing 20.24: Creating a Wrapper type around Vec<String> to implement Display — src/main.rs
use std::fmt;
struct Wrapper(Vec<String>);
impl fmt::Display for Wrapper {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "[{}]", self.0.join(", "))
}
}
fn main() {
let w = Wrapper(vec![String::from("hello"), String::from("world")]);
println!("w = {w}");
}
Display uses self.0 to reach the inner Vec<T>: Wrapper is a tuple struct and Vec<T> is the item at index 0. Then we can use Display’s functionality on Wrapper.
The Rust Programming Language · Ch.20 Advanced Features
58
Chapter 20 · 20.2 Advanced Traits
The downside of a newtype
Wrapper is a new type, so it doesn’t have the methods of the value it’s holding. Three options:
- Implement all the methods of Vec<T> directly on Wrapper, delegating to self.0 — then we can treat Wrapper exactly like a Vec<T>.
- If the new type should have every method the inner type has, implement the Deref trait on Wrapper to return the inner type (Chapter 15, “Treating Smart Pointers Like Regular References”).
- If Wrapper should not have all the inner type’s methods — for example, to restrict its behavior — implement manually just the methods we do want.
The newtype pattern is also useful even when traits are not involved. Next, the focus switches to some advanced ways to interact with Rust’s type system.
The Rust Programming Language · Ch.20 Advanced Features
59
Chapter 20 · 20.3 Advanced Types
Type safety and abstraction with newtypes
The Rust type system has some features mentioned but not yet discussed. This section starts with newtypes in general, as types; then type aliases, a feature similar to newtypes but with slightly different semantics; then the ! type and dynamically sized types.
Building on “Implementing External Traits with the Newtype Pattern”, newtypes are useful for more tasks:
- Statically enforcing that values are never confused, and indicating units. In Listing 20.16, Millimeters and Meters wrapped u32 values. A function with a Millimeters parameter won’t compile if called by accident with a Meters or a plain u32.
- Abstracting away implementation details. The new type can expose a public API that is different from the API of the private inner type.
- Hiding internal implementation. A People type could wrap a HashMap<i32, String> storing a person’s ID with their name. Code using People only uses our public API, such as a method to add a name string; it needn’t know we assign an i32 ID to names internally. A lightweight way to achieve encapsulation (Chapter 18).
The Rust Programming Language · Ch.20 Advanced Features
60
Chapter 20 · 20.3 Advanced Types
Type synonyms and type aliases
Definition 20.18 (Type alias). Another name for an existing type, declared with the type keyword. The alias is a synonym, not a separate, new type.
The alias Kilometers to i32:
type Kilometers = i32;
let x: i32 = 5;
let y: Kilometers = 5;
println!("x + y = {}", x + y);
Unlike the Millimeters and Meters types of Listing 20.16, Kilometers is not a separate, new type: values of type Kilometers are treated the same as values of type i32.
Because Kilometers and i32 are the same type, we can add values of both types and pass Kilometers values to functions that take i32 parameters. But we don’t get the type-checking benefits of the newtype pattern: if we mix up Kilometers and i32 values somewhere, the compiler will not give us an error.
The Rust Programming Language · Ch.20 Advanced Features
61
Chapter 20 · 20.3 Advanced Types
Type aliases reduce repetition
The main use case for type synonyms is to reduce repetition. For example, we might have a lengthy type like Box<dyn Fn() + Send + 'static>.
Writing it in function signatures and as type annotations all over the code can be tiresome and error-prone. Imagine a project full of code like Listing 20.25.
Listing 20.25: Using a long type in many places
let f: Box<dyn Fn() + Send + 'static> = Box::new(|| println!("hi"));
fn takes_long_type(f: Box<dyn Fn() + Send + 'static>) {
// --snip--
}
fn returns_long_type() -> Box<dyn Fn() + Send + 'static> {
// --snip--
}
The Rust Programming Language · Ch.20 Advanced Features
62
Chapter 20 · 20.3 Advanced Types
The Thunk alias
A type alias makes this code more manageable by reducing the repetition. Listing 20.26 introduces an alias named Thunk for the verbose type and replaces all uses of the type with it.
Listing 20.26: Introducing a type alias, Thunk, to reduce repetition
type Thunk = Box<dyn Fn() + Send + 'static>;
let f: Thunk = Box::new(|| println!("hi"));
fn takes_long_type(f: Thunk) {
// --snip--
}
fn returns_long_type() -> Thunk {
// --snip--
}
This code is much easier to read and write! A meaningful alias name also communicates intent: thunk is a word for code to be evaluated at a later time, so it’s an appropriate name for a closure that gets stored.
The Rust Programming Language · Ch.20 Advanced Features
63
Chapter 20 · 20.3 Advanced Types
Aliases for Result<T, E>: the problem
Type aliases are also common with Result<T, E>. In std::io, I/O operations often return a Result<T, E> to handle failures; the std::io::Error struct represents all possible I/O errors, and many std::io functions return Result<T, E> with E = std::io::Error, such as these in the Write trait:
use std::fmt;
use std::io::Error;
pub trait Write {
fn write(&mut self, buf: &[u8]) -> Result<usize, Error>;
fn flush(&mut self) -> Result<(), Error>;
fn write_all(&mut self, buf: &[u8]) -> Result<(), Error>;
fn write_fmt(&mut self, fmt: fmt::Arguments) -> Result<(), Error>;
}
The Result<..., Error> is repeated a lot.
The Rust Programming Language · Ch.20 Advanced Features
64
Chapter 20 · 20.3 Advanced Types
std::io::Result<T>
So std::io has this type alias declaration, and the Write trait’s signatures end up looking like this:
use std::fmt;
type Result<T> = std::result::Result<T, std::io::Error>;
pub trait Write {
fn write(&mut self, buf: &[u8]) -> Result<usize>;
fn flush(&mut self) -> Result<()>;
fn write_all(&mut self, buf: &[u8]) -> Result<()>;
fn write_fmt(&mut self, fmt: fmt::Arguments) -> Result<()>;
}
Because this declaration is in the std::io module, we can use the fully qualified alias std::io::Result<T>: a Result<T, E> with the E filled in as std::io::Error.
It makes code easier to write and gives a consistent interface across std::io. As an alias it’s just another Result<T, E>: its methods and special syntax like the ? operator work.
The Rust Programming Language · Ch.20 Advanced Features
65
Chapter 20 · 20.3 Advanced Types
The never type that never returns
Definition 20.19 (Never type). The special type !, known in type theory lingo as the empty type because it has no values. Rust prefers the name never type because it stands in the place of the return type when a function will never return.
fn bar() -> ! {
// --snip--
panic!();
}
This is read as “the function bar returns never.” We can’t create values of the type !, so bar can never possibly return.
Definition 20.20 (Diverging function). A function that returns never.
But what use is a type you can never create values for? Recall the number-guessing game of Listing 2.5; a bit of it is reproduced in Listing 20.27 (next slide).
The Rust Programming Language · Ch.20 Advanced Features
66
Chapter 20 · 20.3 Advanced Types
What does continue return?
Listing 20.27: A match with an arm that ends in continue
let guess: u32 = match guess.trim().parse() {
Ok(num) => num,
Err(_) => continue,
};
Code that doesn’t work: arms of different types
fn main() {
let guess = "3";
let guess = match guess.trim().parse() {
Ok(_) => 5,
Err(_) => "hello",
};
}
Listing 20.27 is a bit of the guessing game; at the time, some details were skipped. Chapter 6 (“The match Control Flow Construct”) said that match arms must all return the same type, so the code on the right doesn’t work: the type of guess would have to be an integer and a string, and Rust requires that guess have only one type.
So what does continue return? How were we allowed to return a u32 from one arm and have another arm that ends with continue in Listing 20.27?
The Rust Programming Language · Ch.20 Advanced Features
67
Chapter 20 · 20.3 Advanced Types
continue has the type !
As you might have guessed, continue has a ! value. When Rust computes the type of guess, it looks at both match arms: the former with a value of u32, the latter with a ! value. Because ! can never have a value, Rust decides that the type of guess is u32.
Definition 20.21 (Coercion of !). The formal description of this behavior: expressions of type ! can be coerced into any other type.
We’re allowed to end this match arm with continue because continue doesn’t return a value; instead, it moves control back to the top of the loop. So in the Err case, we never assign a value to guess.
The never type is useful with the panic! macro as well. Recall the unwrap function that we call on Option<T> values to produce a value or panic (next slide).
The Rust Programming Language · Ch.20 Advanced Features
68
Chapter 20 · 20.3 Advanced Types
panic! has the type !
The definition of unwrap on Option<T>:
impl<T> Option<T> {
pub fn unwrap(self) -> T {
match self {
Some(val) => val,
None => panic!("called `Option::unwrap()` on a `None` value"),
}
}
}
The same thing happens as in the match in Listing 20.27: Rust sees that val has the type T and panic! has the type !, so the result of the overall match expression is T.
This code works because panic! doesn’t produce a value; it ends the program. In the None case, we won’t be returning a value from unwrap, so this code is valid.
The Rust Programming Language · Ch.20 Advanced Features
69
Chapter 20 · 20.3 Advanced Types
loop has the type !; dynamically sized types
One final expression that has the type ! is a loop:
fn main() {
print!("forever ");
loop {
print!("and ever ");
}
}
Here, the loop never ends, so ! is the value of the expression. However, this wouldn’t be true if we included a break, because the loop would terminate when it got to the break.
Dynamically sized types and the Sized trait
Rust needs to know certain details about its types, such as how much space to allocate for a value of a particular type. This leaves one corner of its type system a little confusing at first:
Definition 20.22 (Dynamically sized type). Also called a DST or unsized type: a type that lets us write code using values whose size we can know only at runtime.
The Rust Programming Language · Ch.20 Advanced Features
70
Chapter 20 · 20.3 Advanced Types
str is a dynamically sized type
Consider str, used throughout the book. That’s right: not &str, but str on its own, is a DST. In many cases, such as when storing text entered by a user, we can’t know how long the string is until runtime. So we can’t create a variable of type str, nor take an argument of type str. This code does not work:
fn main() {
let s1: str = "Hello there!";
let s2: str = "How's it going?";
}
Rust needs to know how much memory to allocate for any value of a particular type, and all values of a type must use the same amount of memory. If Rust allowed this code, these two str values would need to take up the same amount of space. But they have different lengths: s1 needs 12 bytes of storage and s2 needs 15. This is why it’s not possible to create a variable holding a dynamically sized type.
The Rust Programming Language · Ch.20 Advanced Features
71
Chapter 20 · 20.3 Advanced Types
Putting DSTs behind a pointer
So what do we do? You already know the answer: make the type of s1 and s2 a string slice (&str) rather than str.
- Recall from Chapter 4 (“String Slices”) that the slice data structure only stores the starting position and the length of the slice.
- Although &T is a single value that stores the memory address of where the T is located, a string slice is two values: the address of the str and its length.
- So we can know the size of a string slice value at compile time: it’s twice the length of a usize. We always know the size of a string slice, no matter how long the string it refers to is.
- In general, this is how dynamically sized types are used in Rust: they have an extra bit of metadata that stores the size of the dynamic information.
Remark. The golden rule of dynamically sized types: we must always put values of dynamically sized types behind a pointer of some kind.
The Rust Programming Language · Ch.20 Advanced Features
72
Chapter 20 · 20.3 Advanced Types
Traits as DSTs, and the Sized trait
We can combine str with all kinds of pointers: for example, Box<str> or Rc<str>. You’ve seen this before with a different DST: traits. Every trait is a DST we can refer to by using the name of the trait. Chapter 18 (“Using Trait Objects to Abstract over Shared Behavior”) put trait objects behind a pointer, such as &dyn Trait or Box<dyn Trait> (Rc<dyn Trait> would work too).
Definition 20.23 (Sized trait). The trait Rust provides to determine whether or not a type’s size is known at compile time. It is automatically implemented for everything whose size is known at compile time.
In addition, Rust implicitly adds a bound on Sized to every generic function. A generic function definition like the one on the left is actually treated as though we had written the one on the right:
fn generic<T>(t: T) {
// --snip--
}
fn generic<T: Sized>(t: T) {
// --snip--
}
The Rust Programming Language · Ch.20 Advanced Features
73
Chapter 20 · 20.3 Advanced Types
Relaxing the bound with ?Sized
By default, generic functions work only on types that have a known size at compile time. This special syntax relaxes the restriction:
fn generic<T: ?Sized>(t: &T) {
// --snip--
}
- A trait bound on ?Sized means “T may or may not be Sized,” and this notation overrides the default that generic types must have a known size at compile time.
- The ?Trait syntax with this meaning is only available for Sized, not any other traits.
- The type of the t parameter switched from T to &T: because the type might not be Sized, we need to use it behind some kind of pointer. Here we’ve chosen a reference.
Next: functions and closures!
The Rust Programming Language · Ch.20 Advanced Features
74
Chapter 20 · 20.4 Advanced Functions and Closures
Function pointers
This section explores some advanced features related to functions and closures: function pointers and returning closures.
We’ve passed closures to functions; you can also pass regular functions to functions! This is useful when you want to pass a function you’ve already defined rather than defining a new closure.
Definition 20.24 (Function pointer). The type fn (with a lowercase f) that functions coerce to — not to be confused with the Fn closure trait. Passing functions with function pointers lets you use functions as arguments to other functions.
The syntax for a parameter that is a function pointer is similar to that of closures. In Listing 20.28, add_one adds 1 to its parameter. do_twice takes two parameters: a function pointer to any function that takes an i32 parameter and returns an i32, and one i32 value. It calls f twice, passing it arg, then adds the two results together. main calls do_twice with the arguments add_one and 5.
The Rust Programming Language · Ch.20 Advanced Features
75
Chapter 20 · 20.4 Advanced Functions and Closures
Passing add_one to do_twice
Listing 20.28: Using the fn type to accept a function pointer as an argument — src/main.rs
fn add_one(x: i32) -> i32 {
x + 1
}
fn do_twice(f: fn(i32) -> i32, arg: i32) -> i32 {
f(arg) + f(arg)
}
fn main() {
let answer = do_twice(add_one, 5);
println!("The answer is: {answer}");
}
This code prints The answer is: 12.
The parameter f in do_twice is an fn that takes one parameter of type i32 and returns an i32. We can then call f in the body of do_twice. In main, we pass the function name add_one as the first argument.
Unlike closures, fn is a type rather than a trait, so we specify fn as the parameter type directly rather than declaring a generic type parameter with one of the Fn traits as a trait bound.
The Rust Programming Language · Ch.20 Advanced Features
76
Chapter 20 · 20.4 Advanced Functions and Closures
Function pointers and the closure traits
Function pointers implement all three of the closure traits (Fn, FnMut, and FnOnce), so you can always pass a function pointer as an argument for a function that expects a closure. It’s best to write functions using a generic type and one of the closure traits, so your functions can accept either functions or closures.
That said, one case where you’d want to accept only fn and not closures is interfacing with external code that doesn’t have closures: C functions can accept functions as arguments, but C doesn’t have closures.
Where you could use either a closure defined inline or a named function: the map method of the Iterator trait. To turn a vector of numbers into a vector of strings, we could use a closure, as in Listing 20.29.
Listing 20.29: Using a closure with the map method to convert numbers to strings
let list_of_numbers = vec![1, 2, 3];
let list_of_strings: Vec<String> =
list_of_numbers.iter().map(|i| i.to_string()).collect();
The Rust Programming Language · Ch.20 Advanced Features
77
Chapter 20 · 20.4 Advanced Functions and Closures
Named functions and enum initializers in place of closures
Or we could name a function as the argument to map instead of the closure:
Listing 20.30: Using the String::to_string function with the map method to convert numbers to strings
let list_of_numbers = vec![1, 2, 3];
let list_of_strings: Vec<String> =
list_of_numbers.iter().map(ToString::to_string).collect();
We must use the fully qualified syntax from “Advanced Traits”, because there are multiple functions available named to_string. Here we use the to_string defined in the ToString trait, which the standard library implements for any type that implements Display.
Recall from Chapter 6 (“Enum Values”) that the name of each enum variant also becomes an initializer function. These initializer functions are function pointers that implement the closure traits, so we can pass them to methods that take closures (Listing 20.31, next slide).
The Rust Programming Language · Ch.20 Advanced Features
78
Chapter 20 · 20.4 Advanced Functions and Closures
An enum initializer as the argument to map
Listing 20.31: Using an enum initializer with the map method to create a Status instance from numbers
enum Status {
Value(u32),
Stop,
}
let list_of_statuses: Vec<Status> = (0u32..20).map(Status::Value).collect();
Here we create Status::Value instances using each u32 value in the range that map is called on, by using the initializer function of Status::Value.
Remark. Some people prefer this style and some people prefer to use closures. They compile to the same code, so use whichever style is clearer to you.
The Rust Programming Language · Ch.20 Advanced Features
79
Chapter 20 · 20.4 Advanced Functions and Closures
Returning closures
Closures are represented by traits, so you can’t return closures directly. Where you might want to return a trait, you can usually return the concrete type that implements it — but closures don’t have a concrete type that is returnable. For example, you’re not allowed to use the function pointer fn as a return type if the closure captures any values from its scope.
Instead, you will normally use the impl Trait syntax from Chapter 10. You can return any function type, using Fn, FnOnce, and FnMut. Listing 20.32 compiles just fine.
Listing 20.32: Returning a closure from a function using the impl Trait syntax
fn returns_closure() -> impl Fn(i32) -> i32 {
|x| x + 1
}
However, as Chapter 13 (“Inferring and Annotating Closure Types”) noted, each closure is also its own distinct type. To work with multiple functions that have the same signature but different implementations, you need a trait object for them. Consider Listing 20.33.
The Rust Programming Language · Ch.20 Advanced Features
80
Chapter 20 · 20.4 Advanced Functions and Closures
A Vec of closures returned as impl Fn
Listing 20.33: Creating a Vec<T> of closures defined by functions that return impl Fn types — src/main.rs
fn main() {
let handlers = vec![returns_closure(), returns_initialized_closure(123)];
for handler in handlers {
let output = handler(5);
println!("{output}");
}
}
fn returns_closure() -> impl Fn(i32) -> i32 {
|x| x + 1
}
fn returns_initialized_closure(init: i32) -> impl Fn(i32) -> i32 {
move |x| x + init
}
Both return impl Fn(i32) -> i32, but the closures they return differ. Compiling fails (next slide).
The Rust Programming Language · Ch.20 Advanced Features
81
Chapter 20 · 20.4 Advanced Functions and Closures
E0308: mismatched opaque types (1 of 2)
$ cargo build
Compiling functions-example v0.1.0 (file:///projects/functions-example)
error[E0308]: mismatched types
--> src/main.rs:2:44
|
2 | let handlers = vec![returns_closure(), returns_initialized_closure(123)];
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected opaque type,
| found a different opaque type
...
9 | fn returns_closure() -> impl Fn(i32) -> i32 {
| ------------------- the expected opaque type
...
13 | fn returns_initialized_closure(init: i32) -> impl Fn(i32) -> i32 {
| ------------------- the found opaque type
|
The Rust Programming Language · Ch.20 Advanced Features
82
Chapter 20 · 20.4 Advanced Functions and Closures
E0308: mismatched opaque types (2 of 2)
= note: expected opaque type `impl Fn(i32) -> i32`
found opaque type `impl Fn(i32) -> i32`
= note: distinct uses of `impl Trait` result in different opaque types
For more information about this error, try `rustc --explain E0308`.
error: could not compile `functions-example` (bin "functions-example") due to 1 previous error
Definition 20.25 (Opaque type). The unique type Rust creates whenever we return an impl Trait: a type where we cannot see into the details of what Rust constructs for us, nor can we guess the type Rust will generate to write ourselves.
So even though these functions return closures that implement the same trait, Fn(i32) -> i32, the opaque types Rust generates for each are distinct. (This is similar to how Rust produces different concrete types for distinct async blocks even when they have the same output type — Chapter 17, “The Pin Type and the Unpin Trait”.) The solution, seen a few times now: a trait object (Listing 20.34).
The Rust Programming Language · Ch.20 Advanced Features
83
Chapter 20 · 20.4 Advanced Functions and Closures
Returning Box<dyn Fn> trait objects
Listing 20.34: Creating a Vec<T> of closures defined by functions that return Box<dyn Fn> so that they have the same type
fn main() {
let handlers = vec![returns_closure(), returns_initialized_closure(123)];
for handler in handlers {
let output = handler(5);
println!("{output}");
}
}
fn returns_closure() -> Box<dyn Fn(i32) -> i32> {
Box::new(|x| x + 1)
}
fn returns_initialized_closure(init: i32) -> Box<dyn Fn(i32) -> i32> {
Box::new(move |x| x + init)
}
This code compiles just fine (more on trait objects: Chapter 18). Next: macros!
The Rust Programming Language · Ch.20 Advanced Features
84
Chapter 20 · 20.5 Macros
Macros: a family of features
We’ve used macros like println! throughout this book, but haven’t fully explored what a macro is and how it works. The term macro refers to a family of features in Rust: declarative macros with macro_rules! and three kinds of procedural macros.
Table 20.2: The kinds of macros in Rust
| Kind | What it does |
|---|---|
| Declarative (macro_rules!) | Matches the code passed in against patterns and replaces it with code |
| Custom #[derive] (procedural) | Specifies code added with the derive attribute used on structs and enums |
| Attribute-like (procedural) | Defines custom attributes usable on any item |
| Function-like (procedural) | Looks like a function call but operates on the tokens specified as its argument |
We’ll talk about each of these in turn, but first: why do we even need macros when we already have functions?
The Rust Programming Language · Ch.20 Advanced Features
85
Chapter 20 · 20.5 Macros
The difference between macros and functions
Definition 20.26 (Metaprogramming). Writing code that writes other code — fundamentally, what macros are.
Appendix C discusses the derive attribute, which generates an implementation of various traits for you; we’ve also used the println! and vec! macros throughout the book. All of these macros expand to produce more code than the code you’ve written manually.
Metaprogramming is useful for reducing the amount of code you have to write and maintain, which is also one of the roles of functions. However, macros have some additional powers that functions don’t have:
- A function signature must declare the number and type of its parameters. Macros can take a variable number of parameters: println!("hello") with one argument or println!("hello {}", name) with two.
- Macros are expanded before the compiler interprets the meaning of the code, so a macro can, for example, implement a trait on a given type. A function can’t, because it gets called at runtime and a trait needs to be implemented at compile time.
The Rust Programming Language · Ch.20 Advanced Features
86
Chapter 20 · 20.5 Macros
Downsides of macros; declarative macros
- Macro definitions are more complex than function definitions, because you’re writing Rust code that writes Rust code. Due to this indirection, they are generally more difficult to read, understand, and maintain.
- You must define macros or bring them into scope before you call them in a file, as opposed to functions, which you can define anywhere and call anywhere.
Declarative macros for general metaprogramming
Definition 20.27 (Declarative macro). The most widely used form of macros in Rust, also called “macros by example,” “macro_rules! macros,” or just plain “macros.” At its core, it lets you write something similar to a Rust match expression.
A match expression (Chapter 6) takes an expression, compares the resultant value to patterns, and runs the code associated with the matching pattern. Macros also compare a value to patterns associated with code: here the value is the literal Rust source code passed to the macro; the patterns are compared with the structure of that source code; and the code associated with each pattern, when matched, replaces the code passed to the macro. This all happens during compilation.
The Rust Programming Language · Ch.20 Advanced Features
87
Chapter 20 · 20.5 Macros
Defining a macro with macro_rules!: vec!
To define a macro, you use the macro_rules! construct. Let’s look at how vec! is defined. Chapter 8 used it to create a new vector with particular values, for example three integers:
let v: Vec<u32> = vec![1, 2, 3];
We could also use vec! to make a vector of two integers or a vector of five string slices. A function couldn’t do the same, because we wouldn’t know the number or type of values up front. Listing 20.35 is a slightly simplified definition of vec!.
Listing 20.35: A simplified version of the vec! macro definition — src/lib.rs
#[macro_export]
macro_rules! vec {
( $( $x:expr ),* ) => {
{
let mut temp_vec = Vec::new();
$(
temp_vec.push($x);
)*
temp_vec
}
};
}
The Rust Programming Language · Ch.20 Advanced Features
88
Chapter 20 · 20.5 Macros
Anatomy of the vec! definition
Remark. The actual definition of vec! in the standard library includes code to pre-allocate the correct amount of memory up front. That code is an optimization left out here to make the example simpler.
- #[macro_export] indicates that this macro should be made available whenever the crate in which it is defined is brought into scope. Without it, the macro can’t be brought into scope.
- The definition starts with macro_rules! and the name of the macro without the exclamation mark — here vec — followed by curly brackets denoting the body of the macro definition.
- The body’s structure is similar to a match expression. Here there is one arm with the pattern ( $( $x:expr ),* ), followed by => and the block of code associated with this pattern. If the pattern matches, the associated block of code is emitted.
- This is the only pattern in this macro, so there is only one valid way to match; any other pattern results in an error. More complex macros have more than one arm.
The Rust Programming Language · Ch.20 Advanced Features
89
Chapter 20 · 20.5 Macros
Reading the macro pattern
Macro pattern syntax differs from the pattern syntax of Chapter 19, because macro patterns are matched against Rust code structure rather than values. The pieces of the pattern ( $( $x:expr ),* ) in Listing 20.35 (full syntax: the Rust Reference):
- A set of parentheses encompasses the whole pattern.
- A dollar sign ($) declares a variable in the macro system that will contain the Rust code matching the pattern; it makes clear this is a macro variable, not a regular Rust variable.
- Next, a set of parentheses captures values that match the pattern within them, for use in the replacement code. Within $() is $x:expr, which matches any Rust expression and gives it the name $x.
- The comma following $() means a literal comma separator must appear between each instance of the code matching $(). The * means the pattern matches zero or more of whatever precedes it.
Calling vec![1, 2, 3];, the $x pattern matches three times with the three expressions 1, 2, and 3.
The Rust Programming Language · Ch.20 Advanced Features
90
Chapter 20 · 20.5 Macros
What vec![1, 2, 3] expands to
In the arm’s body, temp_vec.push() within $()* is generated for each part matching $() — zero or more times — with $x replaced by each expression matched. vec![1, 2, 3]; is replaced by:
{
let mut temp_vec = Vec::new();
temp_vec.push(1);
temp_vec.push(2);
temp_vec.push(3);
temp_vec
}
We’ve defined a macro that can take any number of arguments of any type and can generate code to create a vector containing the specified elements.
Remark. To learn more about writing macros, consult the online documentation or other resources, such as “The Little Book of Rust Macros” (veykril.github.io/tlborm), started by Daniel Keep and continued by Lukas Wirth.
The Rust Programming Language · Ch.20 Advanced Features
91
Chapter 20 · 20.5 Macros
Procedural macros for generating code from attributes
Definition 20.28 (Procedural macro). The second form of macros, which acts more like a function (and is a type of procedure): it accepts some code as an input, operates on that code, and produces some code as an output — rather than matching against patterns and replacing the code with other code as declarative macros do. Its three kinds — custom derive, attribute-like, and function-like — all work in a similar fashion.
When creating procedural macros, the definitions must reside in their own crate with a special crate type. This is for complex technical reasons that the Rust team hopes to eliminate in the future. Listing 20.36 shows how to define a procedural macro, where some_attribute is a placeholder for using a specific macro variety.
Listing 20.36: An example of defining a procedural macro — src/lib.rs
use proc_macro::TokenStream;
#[some_attribute]
pub fn some_name(input: TokenStream) -> TokenStream {
}
The Rust Programming Language · Ch.20 Advanced Features
92
Chapter 20 · 20.5 Macros
TokenStream in, TokenStream out
- The function that defines a procedural macro takes a TokenStream as an input and produces a TokenStream as an output.
- The TokenStream type is defined by the proc_macro crate that is included with Rust, and represents a sequence of tokens.
- This is the core of the macro: the source code that the macro is operating on makes up the input TokenStream, and the code the macro produces is the output TokenStream.
- The function also has an attribute attached to it that specifies which kind of procedural macro we’re creating. We can have multiple kinds of procedural macros in the same crate.
Next: a custom derive macro, then the small dissimilarities that make the other forms different.
The Rust Programming Language · Ch.20 Advanced Features
93
Chapter 20 · 20.5 Macros
Custom derive macros: the goal
We’ll create a crate named hello_macro that defines a trait HelloMacro with one associated function hello_macro.
Rather than making users implement HelloMacro for each of their types, we provide a procedural macro so they can annotate their type with #[derive(HelloMacro)] to get a default implementation of hello_macro. It prints Hello, Macro! My name is TypeName!, where TypeName is the name of the type on which the trait has been defined.
Listing 20.37: The code a user of our crate will be able to write when using our procedural macro — src/main.rs
use hello_macro::HelloMacro;
use hello_macro_derive::HelloMacro;
#[derive(HelloMacro)]
struct Pancakes;
fn main() {
Pancakes::hello_macro();
}
When we’re done, this code will print Hello, Macro! My name is Pancakes!
The Rust Programming Language · Ch.20 Advanced Features
94
Chapter 20 · 20.5 Macros
The hello_macro crate and its trait
The first step is to make a new library crate:
$ cargo new hello_macro --lib
Next, in Listing 20.38, we define the HelloMacro trait and its associated function.
Listing 20.38: A simple trait that we will use with the derive macro — src/lib.rs
pub trait HelloMacro {
fn hello_macro();
}
We have a trait and its function. At this point, our crate user could implement the trait to achieve the desired functionality, as in Listing 20.39 (next slide).
The Rust Programming Language · Ch.20 Advanced Features
95
Chapter 20 · 20.5 Macros
Why a manual implementation isn’t enough
Listing 20.39: How it would look if users wrote a manual implementation of the HelloMacro trait — src/main.rs
use hello_macro::HelloMacro;
struct Pancakes;
impl HelloMacro for Pancakes {
fn hello_macro() {
println!("Hello, Macro! My name is Pancakes!");
}
}
fn main() {
Pancakes::hello_macro();
}
- Users would need this implementation block for every type used with hello_macro; we want to spare them.
- No default implementation can print the type’s name: Rust has no reflection, so it can’t look up the name at runtime. We need a macro to generate code at compile time.
The Rust Programming Language · Ch.20 Advanced Features
96
Chapter 20 · 20.5 Macros
A separate crate for the procedural macro
The next step is to define the procedural macro. At the time of this writing, procedural macros need to be in their own crate; eventually, this restriction might be lifted. The convention: for a crate named foo, a custom derive procedural macro crate is called foo_derive. Start a new crate called hello_macro_derive inside the hello_macro project:
$ cargo new hello_macro_derive --lib
- The two crates are tightly related, so the procedural macro crate lives within the directory of hello_macro. If we change the trait definition in hello_macro, we’ll have to change the procedural macro in hello_macro_derive as well.
- The two crates will need to be published separately, and programmers using them will need to add both as dependencies and bring both into scope.
- Alternatively, hello_macro could use hello_macro_derive as a dependency and re-export the procedural macro code. But this structure lets programmers use hello_macro even if they don’t want the derive functionality.
The Rust Programming Language · Ch.20 Advanced Features
97
Chapter 20 · 20.5 Macros
Declaring a proc-macro crate
We need to declare the hello_macro_derive crate as a procedural macro crate. We’ll also need functionality from the syn and quote crates, so we add them as dependencies. Add the following to the Cargo.toml file for hello_macro_derive:
hello_macro_derive/Cargo.toml
[lib]
proc-macro = true
[dependencies]
syn = "2.0"
quote = "1.0"
To start defining the procedural macro, place the code of Listing 20.40 (next slide) into the src/lib.rs file of the hello_macro_derive crate. This code won’t compile until we add a definition for the impl_hello_macro function.
The Rust Programming Language · Ch.20 Advanced Features
98
Chapter 20 · 20.5 Macros
The outer function: parsing the TokenStream
Listing 20.40: Code that most procedural macro crates will require in order to process Rust code — hello_macro_derive/src/lib.rs
use proc_macro::TokenStream;
use quote::quote;
#[proc_macro_derive(HelloMacro)]
pub fn hello_macro_derive(input: TokenStream) -> TokenStream {
// Construct a representation of Rust code as a syntax tree
// that we can manipulate.
let ast = syn::parse(input).unwrap();
// Build the trait implementation.
impl_hello_macro(&ast)
}
hello_macro_derive parses the TokenStream; impl_hello_macro transforms the syntax tree — a split that makes writing a procedural macro more convenient. The outer function is the same for almost every procedural macro crate; the inner body depends on the macro’s purpose.
The Rust Programming Language · Ch.20 Advanced Features
99
Chapter 20 · 20.5 Macros
Three crates: proc_macro, syn, and quote
Table 20.3: The crates used by hello_macro_derive
| Crate | Role |
|---|---|
| proc_macro | Comes with Rust (no Cargo.toml entry needed). The compiler’s API that lets us read and manipulate Rust code from our code. |
| syn | Parses Rust code from a string into a data structure that we can perform operations on. |
| quote | Turns syn data structures back into Rust code. |
These crates make it much simpler to parse any sort of Rust code we might want to handle: writing a full parser for Rust code is no simple task.
The hello_macro_derive function will be called when a user of our library specifies #[derive(HelloMacro)] on a type. This is possible because we’ve annotated it with proc_macro_derive and specified the name HelloMacro, which matches our trait name — the convention most procedural macros follow.
The Rust Programming Language · Ch.20 Advanced Features
100
Chapter 20 · 20.5 Macros
From TokenStream to DeriveInput
hello_macro_derive first converts the input from a TokenStream to a data structure we can interpret and operate on — this is where syn comes in. Its parse function takes a TokenStream and returns a DeriveInput struct representing the parsed Rust code.
Listing 20.41: The DeriveInput instance we get when parsing the code that has the macro’s attribute in Listing 20.37
DeriveInput {
// --snip--
ident: Ident {
ident: "Pancakes",
span: #0 bytes(95..103)
},
data: Struct(
DataStruct {
struct_token: Struct,
fields: Unit,
semi_token: Some(
Semi
)
}
)
}
The parsed code is a unit struct with the ident (identifier, meaning the name) of Pancakes. More fields describe all sorts of Rust code; see the syn documentation for DeriveInput.
The Rust Programming Language · Ch.20 Advanced Features
101
Chapter 20 · 20.5 Macros
Output TokenStream, and panicking on errors
Soon we’ll define impl_hello_macro, where we build the new Rust code we want to include. Note first that the output of our derive macro is also a TokenStream. The returned TokenStream is added to the code our crate users write, so when they compile their crate, they get the extra functionality that we provide in the modified TokenStream.
We call unwrap so that hello_macro_derive panics if the call to syn::parse fails. Our procedural macro must panic on errors, because proc_macro_derive functions must return TokenStream rather than Result to conform to the procedural macro API.
Remark. The example is simplified by using unwrap; in production code, provide more specific error messages about what went wrong by using panic! or expect.
Now that we can turn the annotated Rust code from a TokenStream into a DeriveInput instance, let’s generate the code that implements the HelloMacro trait on the annotated type (Listing 20.42).
The Rust Programming Language · Ch.20 Advanced Features
102
Chapter 20 · 20.5 Macros
Implementing HelloMacro from the parsed code
Listing 20.42: Implementing the HelloMacro trait using the parsed Rust code — hello_macro_derive/src/lib.rs
use proc_macro::TokenStream;
use quote::quote;
#[proc_macro_derive(HelloMacro)]
pub fn hello_macro_derive(input: TokenStream) -> TokenStream {
// Construct a representation of Rust code as a syntax tree
// that we can manipulate
let ast = syn::parse(input).unwrap();
// Build the trait implementation
impl_hello_macro(&ast)
}
The outer function is as in Listing 20.40; the new part, impl_hello_macro, continues on the next slide.
The Rust Programming Language · Ch.20 Advanced Features
103
Chapter 20 · 20.5 Macros
The inner function: impl_hello_macro
Listing 20.42 (continued)
fn impl_hello_macro(ast: &syn::DeriveInput) -> TokenStream {
let name = &ast.ident;
let generated = quote! {
impl HelloMacro for #name {
fn hello_macro() {
println!("Hello, Macro! My name is {}!", stringify!(#name));
}
}
};
generated.into()
}
ast.ident gives an Ident holding the annotated type’s name (identifier). Per Listing 20.41, for Listing 20.37 its ident field is "Pancakes", so name holds an Ident that prints as "Pancakes".
The Rust Programming Language · Ch.20 Advanced Features
104
Chapter 20 · 20.5 Macros
quote! and stringify!
- The quote! macro lets us define the Rust code that we want to return. The compiler expects something different from the direct result of quote!, so we convert it to a TokenStream by calling into, which consumes this intermediate representation and returns a value of the required TokenStream type.
- quote! also provides templating: we can enter #name, and quote! replaces it with the value in the variable name. It can even do repetition similar to regular macros (see the quote crate’s docs).
- The macro generates an implementation of HelloMacro for the annotated type, #name. Its one function, hello_macro, prints Hello, Macro! My name is and then the name of the annotated type.
- stringify! is built into Rust: it takes a Rust expression, such as 1 + 2, and at compile time turns it into a string literal, such as "1 + 2". Unlike format! or println!, it doesn’t evaluate the expression first. The #name input might be an expression to print literally, so we use stringify!; it also saves an allocation by converting #name to a string literal at compile time.
The Rust Programming Language · Ch.20 Advanced Features
105
Chapter 20 · 20.5 Macros
Using the derive macro in a binary crate
At this point, cargo build should complete successfully in both hello_macro and hello_macro_derive. To see the procedural macro in action, create a new binary project in your projects directory with cargo new pancakes, and add hello_macro and hello_macro_derive as dependencies in the pancakes crate’s Cargo.toml.
If you publish your versions of hello_macro and hello_macro_derive to crates.io, they would be regular dependencies; if not, specify them as path dependencies:
[dependencies]
hello_macro = { path = "../hello_macro" }
hello_macro_derive = { path = "../hello_macro/hello_macro_derive" }
Put the code of Listing 20.37 into src/main.rs and run cargo run: it should print Hello, Macro! My name is Pancakes! The implementation of HelloMacro from the procedural macro was included without the pancakes crate needing to implement it; #[derive(HelloMacro)] added the trait implementation.
The Rust Programming Language · Ch.20 Advanced Features
106
Chapter 20 · 20.5 Macros
Attribute-like macros
Attribute-like macros are similar to custom derive macros, but instead of generating code for the derive attribute, they allow you to create new attributes. They’re also more flexible: derive only works for structs and enums; attributes can be applied to other items as well, such as functions.
Example: a web framework’s route attribute on a function, and the signature of its procedural macro definition:
#[route(GET, "/")]
fn index() {
#[proc_macro_attribute]
pub fn route(attr: TokenStream, item: TokenStream) -> TokenStream {
There are two parameters of type TokenStream. The first is for the contents of the attribute: the GET, "/" part. The second is the body of the item the attribute is attached to: here, fn index() {} and the rest of the function’s body. Other than that, attribute-like macros work the same way as custom derive macros: create a crate with the proc-macro crate type and implement a function that generates the code you want!
The Rust Programming Language · Ch.20 Advanced Features
107
Chapter 20 · 20.5 Macros
Function-like macros
Function-like macros define macros that look like function calls. Like macro_rules! macros, they’re more flexible than functions; for example, they can take an unknown number of arguments. But macro_rules! macros can only be defined using the match-like syntax of “Declarative Macros for General Metaprogramming”. Function-like macros take a TokenStream parameter, and their definition manipulates that TokenStream using Rust code, as the other two types of procedural macros do. Example: an sql! macro that might be called like so:
let sql = sql!(SELECT * FROM posts WHERE id=1);
This macro would parse the SQL statement inside it and check that it’s syntactically correct — much more complex processing than a macro_rules! macro can do. It would be defined like this:
#[proc_macro]
pub fn sql(input: TokenStream) -> TokenStream {
This is similar to the custom derive macro’s signature: we receive the tokens inside the parentheses and return the code we wanted to generate.
The Rust Programming Language · Ch.20 Advanced Features
108
Chapter 20 · Summary
Chapter summary
Whew! Now you have some Rust features in your toolbox that you likely won’t use often, but you’ll know they’re available in very particular circumstances. These complex topics were introduced so that when you encounter them in error message suggestions or in other people’s code, you’ll recognize the concepts and syntax. Use this chapter as a reference to guide you to solutions:
- unsafe unlocks five superpowers (raw pointers, unsafe functions, mutable statics, unsafe traits, union fields); keep it small, wrap it in safe abstractions, and check it with Miri.
- Advanced traits: associated types, default type parameters, fully qualified syntax, supertraits, and newtypes for external traits.
- Advanced types: newtypes, type aliases, the never type !, and dynamically sized types with Sized/?Sized.
- Function pointers (fn) and returning closures with impl Trait or Box<dyn Fn>.
- Macros: declarative macro_rules! and procedural custom derive, attribute-like and function-like macros.
Remark. Next, Chapter 21 puts everything discussed throughout the book into practice with one more project.
The Rust Programming Language · Ch.20 Advanced Features
109