Crate blake3_servil

Expand description

blake3-servil is a fast BLAKE3 implementation for 64-bit Arm and x86-64. It produces the same digests as official BLAKE3, with a call for each way programs hash (the table below).

§Examples

// Once per process, before any other call (see Startup self-test).
blake3_servil::initialize();

// Hash an input all at once.
let hash1 = blake3_servil::hash(b"foobarbaz");

// Hash an input incrementally.
let mut hasher = blake3_servil::Hasher::new();
hasher.update(b"foo");
hasher.update(b"bar");
hasher.update(b"baz");
let hash2 = hasher.finalize();
assert_eq!(hash1, hash2);

// Extended output. OutputReader also implements Read and Seek.
let mut output = [0; 1000];
let mut output_reader = hasher.finalize_xof();
output_reader.fill(&mut output);
assert_eq!(hash1, output[..32]);

// Print a hash as hex.
println!("{}", hash1);

§Which call to use

Your data	Call
One message in memory	`hash_multithreaded`
One message arriving in pieces	`Hasher::update`
Many messages of one length, in one buffer	`hash_many_multithreaded`
Many messages of different lengths, in memory	`hash_each_multithreaded_with`
Many messages arriving at once (a server’s uploads)	`Hasher::update_each`
A stream of messages read, made, or arriving	`stream`
A message read in verified ranges (Bao, iroh-blobs)	[`outboard_with`], [`verify_range_with`], [`Verifier`]

Each call returns once its hash is done, and uses other CPU cores where that is faster (the forms without multithreaded stay on the calling thread). A stream is different: your program submits messages and goes on to read or make the next ones while other threads hash them; each digest arrives in the place your program named, a little later. Choose it when reading or making the data keeps your thread busy, such as files read from memory or a fast disk: it hashes the most bytes per second.

Short messages hash fastest together: 1024 messages of 64 bytes in one hash_many call take about a fifth of the time of 1024 hash calls.

Keyed hashing and key derivation run at the plain hash’s speed through a Mode, passed to a call’s _with form (hash_with, hash_many_with, …), to Hasher::new_with, or to a stream.

Hashing now and then (rather than nonstop) is slower per byte on every hash function, since its code and data must come back into the CPU’s caches and a resting core runs slower at first: hashing 64 KiB after a pause takes about three times as long as nonstop on an Apple M4 Max.

Apple M4 and later have one matrix unit that this crate uses for inputs of 16 KiB and more; when several threads of a program hash such inputs at the same moment, one runs at full speed and the others at about two thirds of it. The multithreaded calls share the work among the cores themselves, so a program gets the most from one thread making them.

§Memory

A Hasher is about 18 KiB; box it where that matters.
Single-threaded calls allocate nothing: hash, hash_with, hash_many, hash_many_with, Hasher::update and Hasher::finalize, and OutputReader. Hasher::update_reader uses a 1 MiB buffer while it reads past 64 KiB, and hash_each_with 32 bytes for each chunk of its messages of 2 to 15 chunks, each freed when it returns. [outboard_with] returns the outboard it allocates (64 bytes for each 16 KiB), and [verify_range_with] allocates 32 bytes for each group of its range, freed when it returns. A [Verifier] keeps up to 16 KiB, for a group that arrives in pieces, and 48 bytes for each level of the tree; each [Verifier::update] allocates about 100 bytes for each node its piece holds, and its groups’ working space (about 60 bytes a chunk), freed when it returns.
Multithreaded calls allocate nothing below 512 KiB, where they hash on the calling thread. Above it they keep a list of the pieces they hand out, freed when they return: 48 bytes for each 128 KiB of input, and at most 200 bytes for each CPU (1 GiB on 16 CPUs: 390 KiB).
A stream takes its storage (its 2 MiB ring and its entries) from a pool the process keeps: a thread’s first stream allocates it and later streams reuse it, never freed.
Once per process: the startup self-test’s buffers, about 45 KiB of the calling thread’s stack while it runs (it allocates nothing), and the threads: by default a worker per CPU beyond the first, each with the standard library’s stack, started by initialize_multithreaded.

§Startup self-test

Every program calls initialize (or initialize_multithreaded) once, before any other call of this crate: a call before it panics, in every build, saying so. The check costs nothing: it is part of the platform load every call makes. A program that makes multithreaded calls or streams calls initialize_multithreaded: a multithreaded call or a stream before it panics (one load per call).

initialize checks the build and the CPU before the process first hashes, in well under a millisecond. It runs every assembly kernel built for this machine on fixed inputs and compares the outputs with the reference implementation’s. A mismatch stops the program with a panic naming the code path. Other threads that call initialize meanwhile wait for it, and stop with it if it fails.

§Cargo Features

The std feature (the only feature enabled by default) enables the Write implementation and the update_reader method for Hasher, and also the Read and Seek implementations for OutputReader.

The zeroize feature (off by default, on for docs.rs) implements Zeroize for this crate’s types.

The serde feature (off by default, on for docs.rs) implements serde::Serialize and serde::Deserialize for Hash.

Structs§

Hash: An output of the default size, 32 bytes, which provides constant-time equality checking.
Hasher: A message’s hash, fed in pieces: update with each piece, in order, then finalize. The pieces may be of any lengths; the hash is the same as hash of them all at once.
HexError: The error type for Hash::from_hex.
OutputReader: An incremental reader for extended output, returned by Hasher::finalize_xof.
Stream: The messages of one stream call, and the places for their digests, borrowed for its scope; D is the stream’s handler.

Enums§

Mode: Which of BLAKE3’s three functions a call computes: the plain hash (hash), the keyed hash, or key derivation. Every call that takes a mode runs each one at the same speed, but for DeriveKey, which first hashes its context string (once per call; DeriveKeyMaterial takes it hashed once for many calls).

Constants§

BLOCK_LEN: The number of bytes in a block, 64.
CHUNK_LEN: The number of bytes in a chunk, 1024.
KEY_LEN: The number of bytes in a key, 32.
OUT_LEN: The number of bytes in a Hash, 32.

Functions§

hash: The hash of input, on the calling thread.
hash_derive_key_context: Hash a key-derivation context string and return a ContextKey.
hash_each_multithreaded_with: hash_each_with, using other CPU cores where that is faster: the same hashes, never slower. Collections of 512 KiB and more in all use them, as hash_multithreaded does. Requires out.len() == items.len().
hash_each_with: The hash of each message, of any lengths, on the calling thread: out[i] is hash_with(mode, items[i]). Short messages hash together, so a collection of small items (a directory’s files, a repository’s objects) takes less time than one call each. Requires out.len() == items.len().
hash_many: The hashes of many messages of one length, laid out in one buffer, on the calling thread: out[i] becomes the hash of message i.
hash_many_multithreaded: hash_many, using other CPU cores where that is faster: the same hashes, never slower.
hash_many_multithreaded_with: hash_many_multithreaded in any Mode, at its speed.
hash_many_with: hash_many in any Mode, at its speed.
hash_multithreaded: The hash of input, using other CPU cores where that is faster: the same Hash as hash, never slower.
hash_multithreaded_with: hash_multithreaded in any Mode, at its speed.
hash_with: hash in any Mode: the plain hash, the keyed hash, or key derivation, at hash’s speed.
initialize: Run the startup self-test and detect the CPU: every program calls this (or initialize_multithreaded) once, before any other call of this crate, and a call before it panics. Under 0.2 ms, once per process (see the crate’s documentation); later calls return at once.
initialize_multithreaded: initialize, and start the threads that the multithreaded calls and stream use: every program that makes such calls calls this once, before them, in place of initialize. It starts threads - 1 workers, and each multithreaded call hashes on its calling thread and the workers (calls made at once share the workers), and on CPUs with SME2 (Apple M4) the SME2 thread too; None means one thread a CPU (std::thread::available_parallelism, which follows the CPU affinity the program runs under). With 1, calls hash on their calling thread alone: no workers start, nor the SME2 thread. Otherwise it starts the first worker and returns; the first worker starts the others (each 20-30 us on Linux x86-64), and calls run on the workers started so far. The threads wait asleep whenever no call has work for them (their memory: see Memory). Later calls return at once, and must ask for the same number: another Some panics. A panic on a worker thread (a bug in this crate) ends the process, its message printed, rather than leaving the call that waits for that worker’s piece waiting forever.
stream: Hash a stream of messages on this crate’s threads, in place: body submits messages that live at least as long as the call (Stream::submit), each with the place its digest goes, in the program’s memory (a batch’s, the place for all its digests). The threads write the digests there, and done receives each place back, in submission order, once its digests are in it: from a submission (when the stream is full), collect and finish, on this thread. body calls finish last (a scope that ends with places not handed back panics). A message of any length; this crate’s threads read it where it is, and stream returns only after they are done, also when body panics.

Type Aliases§

ContextKey: An alias to distinguish hash_derive_key_context outputs from other keys.