Naming is hard

These days saw the release of nolife 0.4, a crate that offers an intuitive fix for borrow errors by letting you conveniently store a scope of execution containing multiple borrows inside of your struct without a lifetime. I want to seize this opportunity to write a bit about what goes into making a library that juggles self-referential code like nolife does.

A taste of `nolife`

nolife 0.4 allows you to turn the sad error E0597: borrowed value does not live long enough:

use std::{fs::File, io::Read};

use zip::{read::ZipFile, ZipArchive};

pub fn zip_streamer(file_name: &str, member_name: &str) -> impl Read {
    let file = File::open(file_name).unwrap();
    let mut archive = ZipArchive::new(file).unwrap();
    let file : ZipFile<'_> = archive.by_name(member_name).unwrap();
    file
}

into the happily compiling:

use std::{fs::File, io::Read};

use zip::{read::ZipFile, ZipArchive};

pub fn zip_streamer(file_name: String, member_name: String) -> impl Read {
    let zip_scope =
        // freeze the scope in place, not closing it, and keeping the file borrowed
        nolife::BoxScope::new_dyn(nolife::scope!({
            let file = File::open(file_name).unwrap();
            let mut archive = ZipArchive::new(file).unwrap();
            let mut file = archive.by_name(&member_name).unwrap();
            freeze_forever!(&mut file)
        }));
    ZipStreamer { zip_scope }
}

struct ZipStreamer {
    // Store your scope without a lifetime
    zip_scope: nolife::BoxScope<ZipFamily>,
}

impl Read for ZipStreamer {
    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        // access the file that was frozen inside of the scope and read from it
        self.zip_scope.enter(|zip_file| zip_file.read(buf))
    }
}

// nolife boilerplate
struct ZipFamily;
impl<'a> nolife::Family<'a> for ZipFamily {
    type Family = ZipFile<'a>;
}

fn main() {
    let mut output = String::new();
    zip_streamer(
        std::env::args().nth(1).unwrap(),
        std::env::args().nth(2).unwrap(),
    )
    .read_to_string(&mut output)
    .unwrap();
    println!("{}", output);
}

The 0.4 version brings breaking changes, most notably a new macro API that is mandatory for safety. Find all the changes in the changelog.

🐲 Thou shall face the dragons

Unsafe Rust is very difficult to write correctly, so much so that I think the unsafe keyword could as well be replaced by the proverbial "here be dragons", you know, for clarity.

The best way to tame these dragons is to talk to them: whenever you're using the unsafe keyword in the "oath taker" position, you're making the promise that your code verifies the preconditions stated in the documentation of items for which the unsafe keyword is used in the "oath recipient" position. To not lose track of the contract, the preconditions have to be spelled out explicitly at the "oath recipient" position and then verified explicitly at the "oath taker" position.


// in <https://github.com/dureuill/nolife/blob/v0.4.0/src/raw_scope.rs>
impl<T, F: ?Sized> RawScope<T, F>
where
    T: for<'a> Family<'a>,
    F: Future<Output = Never>,
{
    // 👇 Oath recipient position
    /// # Safety
    ///
    /// 1. `this` points to a properly aligned, fully initialized `RawScope<T, F>`.
    /// 2. `this` verifies the guarantees of `Pin` (one of its fields is pinned in this function)
    /// 3. No other exclusive reference to the frozen value. In particular, no concurrent calls to this function.
    pub(crate) unsafe fn enter<'borrow, Output, G>(this: NonNull<Self>, f: G) -> Output
    where
        G: for<'a> FnOnce(&'borrow mut <T as Family<'a>>::Family) -> Output,
    {
        todo!()
    }
}


// in <https://github.com/dureuill/nolife/blob/v0.4.0/src/box_scope.rs>
impl<T, F: ?Sized> BoxScope<T, F>
where
    T: for<'a> Family<'a>,
    F: Future<Output = Never>,
{
    /// Enters the scope, making it possible to access the data frozen inside of the scope.
    ///
    /// # Panics
    ///
    /// - If the passed function panics.
    /// - If the underlying future panics.
    /// - If the underlying future awaits for a future other than the [`crate::FrozenFuture`].
    pub fn enter<'borrow, Output, G>(&'borrow mut self, f: G) -> Output
    where
        G: for<'a> FnOnce(&'borrow mut <T as Family<'a>>::Family) -> Output,
    {
        // 👇 Oath taker position
        // SAFETY:
        // 1. `self.0` is valid as a post-condition of `new`.
        // 2. The object pointed to by `self.0` did not move and won't before deallocation.
        // 3. `BoxScope::enter` takes an exclusive reference and the reference passed to `f` cannot escape `f`.
        unsafe { RawScope::enter(self.0, f) }
    }
}

The other best way to tame the dragons is to trial them with dedicated tooling. nolife has automated tests that run in CI that use miri to check for Undefined Behavior.

🚫📜 Thou shall not compile

One trick I'm proud of for nolife is the "counterexamples" module. It uses the code example features from rustdoc to make sure that some code examples that shouldn't compile, don't.

Here's an example of non-compiling example:

//! # Attempting to save the frozen future in an async block
//!
//! ```compile_fail,E0767,E0267
//! use nolife::{scope, SingleFamily, TopScope};
//! fn forcing_inner_async() {
//!     fn some_scope(x: u32) -> impl TopScope<Family = SingleFamily<u32>> {
//!         scope!({
//!             let fut = async {
//!                 freeze!(&mut 0);
//!             };
//!             // poll future
//!             // bang!
//!             panic!()
//!         })
//!     }
//! }
//! ```

When using the nightly toolchain, the specified error numbers are even verified, and the test fails if the compilation errors don't match.

🚫😋 Thou shall refrain from gluttony

The API of nolife is intentionally small, it provides the minimal amount of code to implement its main use cases. This is because having more API surface, and more implementation code creates more opportunities to introduce errors.

In time as the crate matures I'll be adding more convenience and support ancillary use cases.

🚫📦 Thou shall not use `Box`

This one is more specific to a crate that uses self-referential structures like nolife.

nolife's BoxScope type is an abstraction for a boxed scope, yet it doesn't use Box internally. This is because Box carries "noalias" semantics, which is incompatible with a self-referential structure such as BoxScope.

I took the utmost care to check that any access to a variable through a pointer would not get afoul of the borrow invalidation rules that proceed from the aliasing rules.

All test cases in nolife currently pass miri with both the tree borrow and stacked borrow models.

🌈 Thou shall believe in thy friends 💖

Armed with the commandments above, I released nolife 0.3 back in January, together with an article to announce it. Despite my best efforts at making nolife sound, I eventually woke up on a Saturday to find two soundness issues had been filed to the issue tracker on GitHub. While #7 had a simple fix, it turned out that #8 basically required the new macro API (I'll let interested parties read the discussions in that issue).

@steffahn, the contributor who opened these issues, has also been pivotal to resolve them.

@steffahn is used to contributing to self-referential crates, with issues in ouroboros, yoke and self_cell. More broadly, he has experience reporting soundness issues to many projects in the Rust ecosystem, including Rust itself, qcell, or PyO3.

When I asked him over email about his interest in self-referential crates, here's what he replied:

As for my interest in self-referential crates, I love the premise of Rust's safety guarantees that they are supposed to hold up even for the most complex (safe!) Rust code imaginable, and I enjoy learning to understand how it all fits together.

Whenever something doesn't seem to fit my expectations, this means there's either more nuance to learn in the details, or there's a soundness hole. One of Rust's strengths in their safety story is that unsafe code can be used to build new safe abstractions, the most general ones of which can even serve de-facto as extensions to the language features, fundamentally extending what's possible in safe Rust for everyone else. Self-referencing crates are a prime example of this, with a lot of design space but also a lot lot lot that can go wrong.

@steffahn also stresses that ensuring that the crates building such safe abstractions are sound is a community effort. Other people are also reporting plenty of soundness issues, the resolution of which guarantees a rock solid foundation to these libraries.

Indeed, I remember people on the users.rust-lang.org forum patiently walking me through the leakpocalypse as I was discussing a naive prior idea for a... lifetime-erased reference 😄. That forum is a great place to actively seek a review of your latest unsafe crimes, with many experienced Rustaceans kindly dwelling in the land 🦀

Back to @steffahn, he is now the first external contributor to nolife. I asked him what he thought of nolife, here are his words:

I find it interesting because it's a take on self-referential types with an API that approaches it from a direction I hadn't seen before. I'm curious what kind of use-cases might emerge where this offers strong usability improvements over the existing – let's say "traditional" – self-referencing crates. It comes with high versatility by allowing execution of additional code between calls to 'enter', which gives it almost a flavor of generator-style framework, with a "lending iterators" twist; on the other hand, it's harder to do certain things with it such as regaining ownership of the original owned data (the owner in "traditional" self-referential crates) and the use of a Future::poll call every time the scope is re-entered, even when freeze_forever! is used, could have slight performance downsides, too, as well as the need (on current Rust) for using trait objects in order to get a nameable type out of it.

I am very grateful to him for his dedication, it was a pleasure to work with him on nolife, and I love open-source ❤️

Conclusion

If you want to write an unsafe crate, keep the above advice in mind. If you think that's too much work, and your problem looks like the motivating example above, then 🛒 go grab nolife! It is sounder than ever, and has a brand new, convenient API ✨

A taste of nolife