IV.3: Persistent threadpool
II.4 put rayon under the big matmuls and cut a 511-token prefill from 19.4 s to 3.8 s. Decode got nothing, by design: the MIN_ROWS_FOR_PARALLEL threshold sends every single-row matmul straight to one core, because a one-row matmul has "nothing to split." That framing was half the truth. A decode matvec does have something to split: its output rows. The Q8 matvec from II.6 computes p independent output elements (one dot product each), and nothing stops four cores from taking a quarter of them apiece.
We tried exactly that during Act 2, quietly, with rayon: hand the matvec's output rows to par_chunks_mut and let the thread pool do its thing. Measured result: zero gain. Not a small gain, zero. This chapter is about why that happened and what llama.cpp does instead, and it ends with decode at 109 tokens/sec on four cores, up from 46 on one.
Why rayon lost
Count the work first. One decode token runs 7 quantized matvecs per layer (Q, K, V, output projection, gate, up, down) across 28 layers, plus the lm_head: 197 matvecs per token. At the ~46 tokens/sec the simd backend measures after IV.1 and IV.2, a token takes ~21.8 ms, so the average matvec is on the order of a hundred microseconds. That is the entire job: read one to three megabytes of weights, do the integer dots, write a few thousand floats.
Now count rayon's costs. rayon's workers sleep when there is no work; that's the polite thing for a general-purpose library to do. So every par_chunks_mut call on a cold pool pays to wake worker threads through the OS, inject the task, and join at the end. Between our matvecs the workers have nothing to do for a few dozen microseconds (the serial glue between ops), which is exactly long enough for them to start going back to sleep. The fork-join overhead lands in the tens-to-hundreds of microseconds range, the same order as the matvec itself. Splitting a 100-microsecond job four ways saves at most 75 microseconds; if publishing the job costs about that much, the split buys nothing. 197 times per token, it bought us exactly the zero we measured.
For prefill none of this mattered: a 511-row matmul is hundreds of milliseconds of work, and a one-time wakeup disappears into it. That is why II.4's threshold was the right call given rayon. The fix is not to stop splitting; it is to stop paying the wakeup.
What llama.cpp does: threads that never sleep
llama.cpp's CPU backend spawns its worker threads once, at startup, and they never die and never sleep while a generation is running. They sit in a tight loop watching a single atomic counter. Publishing a job to the whole pool is one atomic increment of that counter; each worker notices the counter change, runs its slice of the job, and increments a second atomic to say it finished. No locks, no channels, no condition variables, no OS wakeups anywhere in the hot path. The cost of dispatching work to N cores drops from tens of microseconds to tens of nanoseconds, and suddenly 100-microsecond jobs are worth splitting.
The trade is naked: spinning threads burn CPU while they wait. For a general-purpose library that would be rude. For an inference engine that dispatches 197 jobs per token with microseconds between them, it is the whole point; the wait is short, and the cores would otherwise idle through it anyway.
This chapter builds that pool in about 140 lines, then a fifth CPU backend, pool, that uses it for the decode matvec and keeps rayon for prefill.
The pool
src/backend/pool.rs, from the top:
use std::cell::UnsafeCell;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread::{self, JoinHandle};
type JobFn = unsafe fn(*const (), usize, usize);
struct Shared {
generation: AtomicUsize,
done: AtomicUsize,
stop: AtomicBool,
job: UnsafeCell<Option<(JobFn, *const ())>>,
}
unsafe impl Sync for Shared {}
unsafe impl Send for Shared {}Shared is the entire coordination state, one struct the caller and every worker hold through an Arc. generation is the job counter the workers watch: it only ever goes up, and each increment means "a new job is posted." done counts workers that have finished the current job. stop tells workers to exit at shutdown. job is the mailbox: a function pointer plus an opaque context pointer, which together are a type-erased closure.
UnsafeCell and the two unsafe impls are Rust's way of making us sign for what we're doing: we are sharing a mutable slot and raw pointers across threads with no lock, and the compiler cannot verify that's sound, so we assert it. The safety argument is short and worth stating precisely, because the whole file rests on it:
- The caller writes
jobbefore incrementinggeneration(a release store), and workers readjobonly after observing the increment (an acquire load). The release/acquire pair makes the write visible. - The caller cannot return from
dispatch, and therefore cannot invalidate the closure the pointers point into, until every worker has bumpeddone. Workers bumpdoneonly after they finish running the job. So the pointers are never dereferenced outside the closure's lifetime. - The job itself writes only disjoint slices of the output (we'll see that in
pool_cpu.rs), so concurrent execution races on nothing.
pub struct ThreadPool {
shared: Arc<Shared>,
workers: Vec<JoinHandle<()>>,
n_threads: usize,
}
impl ThreadPool {
pub fn new(n_threads: usize) -> Self {
let n_threads = n_threads.max(1);
let shared = Arc::new(Shared {
generation: AtomicUsize::new(0),
done: AtomicUsize::new(0),
stop: AtomicBool::new(false),
job: UnsafeCell::new(None),
});
let mut workers = Vec::new();
for id in 1..n_threads {
let s = shared.clone();
workers.push(thread::spawn(move || worker_loop(s, id, n_threads)));
}
Self {
shared,
workers,
n_threads,
}
}Note the loop bounds: a pool of n_threads spawns n_threads - 1 workers, with ids 1..n_threads. Id 0 is reserved for the caller. When dispatch runs a job, the thread that called it doesn't stand around waiting; it participates as thread 0 and computes its own share. A 4-thread pool is three spinning workers plus you.
pub fn dispatch<F: Fn(usize, usize) + Sync>(&self, f: F) {
unsafe fn call<F: Fn(usize, usize)>(ctx: *const (), id: usize, n: usize) {
let f = unsafe { &*(ctx as *const F) };
f(id, n);
}
if self.n_threads == 1 {
f(0, 1);
return;
}
let ctx = &f as *const F as *const ();
unsafe {
*self.shared.job.get() = Some((call::<F>, ctx));
}
self.shared.done.store(0, Ordering::Relaxed);
self.shared.generation.fetch_add(1, Ordering::Release);
f(0, self.n_threads);
let target = self.n_threads - 1;
let mut spins = 0u32;
while self.shared.done.load(Ordering::Acquire) != target {
std::hint::spin_loop();
spins += 1;
if spins > 1 << 16 {
thread::yield_now();
}
}
}
}dispatch takes a closure f(id, n); the contract is that f will be called once per thread with that thread's id and the total count, and each call handles its own slice. The inner call function is the type-erasure trick: workers can't know the closure's concrete type, so we store a plain function pointer that knows how to cast the context pointer back to &F and invoke it. This is how the mailbox holds "any closure" in two words.
The publish sequence is the fast path the whole chapter exists for: write the mailbox, zero done, increment generation with Release ordering. That increment is the dispatch; there is nothing else. Then the caller immediately runs its own share, f(0, self.n_threads), doing real work while the workers do theirs. Finally it spin-waits until done reaches n_threads - 1. The Acquire load pairs with each worker's Release increment, which also guarantees the caller sees every byte the workers wrote to the output before dispatch returns.
std::hint::spin_loop() tells the CPU "this is a busy-wait" so it can relax the core briefly. If the wait somehow runs long (2^16 spins), thread::yield_now() offers the OS the core; in steady-state decode that branch essentially never triggers, but it keeps an idle or oversubscribed pool from pegging CPUs pointlessly.
impl Drop for ThreadPool {
fn drop(&mut self) {
self.shared.stop.store(true, Ordering::Relaxed);
self.shared.generation.fetch_add(1, Ordering::Release);
for w in self.workers.drain(..) {
let _ = w.join();
}
}
}
fn worker_loop(shared: Arc<Shared>, id: usize, n_threads: usize) {
let mut last = 0usize;
loop {
let mut spins = 0u32;
loop {
let g = shared.generation.load(Ordering::Acquire);
if g != last {
last = g;
break;
}
std::hint::spin_loop();
spins += 1;
if spins > 1 << 16 {
thread::yield_now();
}
}
if shared.stop.load(Ordering::Relaxed) {
break;
}
let (func, ctx) = unsafe { (*shared.job.get()).unwrap() };
unsafe { func(ctx, id, n_threads) };
shared.done.fetch_add(1, Ordering::Release);
}
}worker_loop is the mirror image. Each worker remembers the last generation it saw and spins until the counter moves. When it does: check stop (shutdown reuses the same wakeup mechanism, which is why Drop bumps the counter after setting the flag), read the mailbox, run the job with this worker's id, bump done with Release. Then back to the top, spinning at the next generation. A worker's whole life is: watch one cache line, work, increment, repeat.
The spin itself is cheap in the way that matters: generation lives in one cache line that every worker holds in its L1 cache in shared state. Spinning on it costs no memory traffic at all until the caller's increment invalidates the line, at which point every worker re-reads it and sees the new job. That is why publishing to N workers is effectively one store plus one cache-coherence broadcast.
Two helpers round out the file:
pub fn split(total: usize, id: usize, n: usize) -> (usize, usize) {
let base = total / n;
let rem = total % n;
let start = id * base + id.min(rem);
let len = base + if id < rem { 1 } else { 0 };
(start, start + len)
}
pub fn default_threads() -> usize {
if let Ok(v) = std::env::var("INFERNO_THREADS") {
if let Ok(n) = v.trim().parse::<usize>() {
if n >= 1 {
return n;
}
}
}
if let Ok(out) = std::process::Command::new("sysctl")
.args(["-n", "hw.perflevel0.physicalcpu"])
.output()
{
if let Ok(s) = String::from_utf8(out.stdout) {
if let Ok(n) = s.trim().parse::<usize>() {
if n >= 1 {
return n;
}
}
}
}
thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(4)
}split(total, id, n) deals total items into n nearly equal contiguous ranges, giving the first total % n threads one extra item. Thread id gets [start, end); the ranges tile 0..total exactly and never overlap, which is what makes the disjoint-writes argument above hold.
default_threads picks the pool size: the INFERNO_THREADS environment variable if set (that's how the sweep below is run), otherwise sysctl hw.perflevel0.physicalcpu, which on Apple Silicon reports the number of performance cores; 6 on this M2 Pro. Why P-cores only, and not available_parallelism's 10? The benchmark section makes the case with numbers, but the short version: a spinning barrier moves at the pace of its slowest thread, and this machine has two speeds of core.
The pool backend
src/backend/pool_cpu.rs is the Backend that puts the pool to work. Same decorator shape as every backend since II.3:
use super::Backend;
use super::pool::{ThreadPool, default_threads, split};
use super::simd_cpu::{SimdCpu, matvec_q8_0};
use crate::tensor::q8_0::{Q8_0_BLOCK_SIZE, Q8_0Block};
use crate::tensor::{Tensor, TensorData};
use rayon::prelude::*;
const MIN_ROWS_FOR_PARALLEL: usize = 4;
struct SendPtr<T>(*mut T);
unsafe impl<T> Send for SendPtr<T> {}
unsafe impl<T> Sync for SendPtr<T> {}
pub struct PoolCpu<B: Backend> {
inner: SimdCpu<B>,
pool: ThreadPool,
}
impl<B: Backend> PoolCpu<B> {
pub fn new(inner: SimdCpu<B>) -> Self {
Self {
inner,
pool: ThreadPool::new(default_threads()),
}
}PoolCpu wraps a SimdCpu and owns one ThreadPool, created once with the backend and alive for the whole process; that's the "persistent" in the chapter title. SendPtr is a two-line wrapper that lets a raw output pointer cross into the dispatch closure; raw pointers aren't Send/Sync by default, and again the unsafe impl is us signing for the disjointness argument rather than proving it to the compiler.
The reason this exists, the pooled matvec:
fn matvec_pooled(&self, blocks: &[Q8_0Block], x: &[f32], p: usize, cols: usize) -> Tensor {
let bpr = cols / Q8_0_BLOCK_SIZE;
let mut out = vec![0.0f32; p];
let out_ptr = SendPtr(out.as_mut_ptr());
self.pool.dispatch(|id, n| {
let (r0, r1) = split(p, id, n);
if r1 > r0 {
let rows = r1 - r0;
let slice = &blocks[r0 * bpr..r1 * bpr];
let part = matvec_q8_0(slice, x, rows, cols).expect("pool Q8 matvec failed");
let dst = &out_ptr;
unsafe {
std::ptr::copy_nonoverlapping(part.as_ptr(), dst.0.add(r0), rows);
}
}
});
Tensor::new(out, vec![1, p])
}Thread id takes output rows r0..r1 via split. Its slice of the Q8 weight blocks is blocks[r0 * bpr..r1 * bpr] (bpr blocks per row), and it runs the unchanged matvec_q8_0 from II.6 on just those rows: the same sdot integer kernel, the same activation quantization, everything. The partial result is copied into out at offset r0; every thread's destination range is disjoint by construction of split.
One property matters enough to call out. Each output element is one full-length dot product computed start to finish by exactly one thread, with the same kernel walking the same blocks in the same order as the single-threaded version. Splitting by rows never splits a dot product, so no floating-point additions get reordered. The pooled matvec's output is byte-identical to simd's, at any thread count, which is the same discipline IV.1 established: optimizations within the CPU backend must not change the numbers. (The benchmark runs below generate token-for-token identical text to simd, as they must.)
Prefill keeps its rayon path, lifted verbatim from Parallel:
fn matmul_fp32_q8_0_rayon(
&self,
a_data: &[f32],
blocks: &[Q8_0Block],
m: usize,
n: usize,
p: usize,
) -> Tensor {
let mut data = vec![0.0f32; m * p];
data.par_chunks_mut(p).enumerate().for_each(|(i, out_row)| {
let x = &a_data[i * n..(i + 1) * n];
let row = matvec_q8_0(blocks, x, p, n).expect("pool_cpu Q8 matvec failed");
out_row.copy_from_slice(&row);
});
Tensor::new(data, vec![m, p])
}
fn matmul_fp32_fp32_rayon(
&self,
a_data: &[f32],
b_data: &[f32],
m: usize,
n: usize,
p: usize,
) -> Tensor {
let mut data = vec![0.0f32; m * p];
data.par_chunks_mut(p).enumerate().for_each(|(i, out_row)| {
let op = out_row.as_mut_ptr();
unsafe {
for j in 0..n {
let aij = a_data[i * n + j];
let b_row = b_data.as_ptr().add(j * p);
SimdCpu::<B>::saxpy_row_f32(op, b_row, aij, p);
}
}
});
Tensor::new(data, vec![m, p])
}
}Why not run prefill through the pool too? Because rayon is better at big jobs: work-stealing balances uneven rows, and the wakeup cost that killed it for decode is irrelevant across hundreds of milliseconds of matmul. The pool's fixed even split is the right shape for 197 identical tiny jobs; rayon's stealing is the right shape for a few huge ones. Use each where it wins.
The matmul method routes between the three:
impl<B: Backend + Sync> Backend for PoolCpu<B> {
fn name(&self) -> String {
"pool".to_string()
}
fn matmul(&self, a: &Tensor, b: &Tensor) -> Tensor {
assert_eq!(a.shape().len(), 2);
assert_eq!(b.shape().len(), 2);
let m = a.shape()[0];
let n = a.shape()[1];
match (a.as_data(), b.as_data()) {
(TensorData::Fp32(a_data), TensorData::Q8_0(blocks)) => {
let p = b.shape()[0];
assert_eq!(n, b.shape()[1], "Q8_0 matmul inner dim mismatch");
if m == 1 {
self.matvec_pooled(blocks, a_data, p, n)
} else if m < MIN_ROWS_FOR_PARALLEL {
self.inner.matmul(a, b)
} else {
self.matmul_fp32_q8_0_rayon(a_data, blocks, m, n, p)
}
}
(TensorData::Fp32(a_data), TensorData::Fp32(b_data)) => {
let p = b.shape()[1];
assert_eq!(n, b.shape()[0], "tensor shape mismatch");
if m < MIN_ROWS_FOR_PARALLEL {
self.inner.matmul(a, b)
} else {
self.matmul_fp32_fp32_rayon(a_data, b_data, m, n, p)
}
}
_ => panic!("matmul: LHS must be FP32"),
}
}Read the Q8 arm as the map of this chapter against II.4. m == 1 is a decode matvec: it goes to the pool, the case rayon couldn't touch. m >= 4 is prefill: rayon, unchanged. In between (2- and 3-row matmuls, which the batched-decode path from III.7 can produce) falls through to single-core SIMD, same as II.4 decided. The FP32 arm has no pooled case because with the Q8 checkpoint every decode-path matmul is Q8; FP32 matmuls only show up in shapes the rayon path already handles.
Everything else delegates to the wrapped SimdCpu, including IV.1's fused_attention:
fn sum_squares_axis(&self, x: &Tensor, axis: usize) -> Tensor {
self.inner.sum_squares_axis(x, axis)
}
fn fused_attention(
&self,
q: &Tensor,
k: &Tensor,
v: &Tensor,
n_heads: usize,
n_kv_heads: usize,
head_dim: usize,
) -> Tensor {
self.inner
.fused_attention(q, k, v, n_heads, n_kv_heads, head_dim)
}
fn add(&self, a: &Tensor, b: &Tensor) -> Tensor {
self.inner.add(a, b)
}
fn hadamard(&self, a: &Tensor, b: &Tensor) -> Tensor {
self.inner.hadamard(a, b)
}
fn scale(&self, x: &Tensor, s: f32) -> Tensor {
self.inner.scale(x, s)
}
fn silu(&self, x: &Tensor) -> Tensor {
self.inner.silu(x)
}
fn add_scalar(&self, a: &Tensor, s: f32) -> Tensor {
self.inner.add_scalar(a, s)
}
fn rsqrt_elem(&self, x: &Tensor) -> Tensor {
self.inner.rsqrt_elem(x)
}
fn broadcast_row_scalars(&self, t: &Tensor, d: usize) -> Tensor {
self.inner.broadcast_row_scalars(t, d)
}
fn transpose_2d(&self, a: &Tensor) -> Tensor {
self.inner.transpose_2d(a)
}
fn softmax_rows(&self, x: &Tensor) -> Tensor {
self.inner.softmax_rows(x)
}
fn gather_rows(&self, table: &Tensor, row_indices: &[usize]) -> Tensor {
self.inner.gather_rows(table, row_indices)
}
fn reshape_data(&self, x: &Tensor, shape: Vec<usize>) -> Tensor {
self.inner.reshape_data(x, shape)
}
fn fill_strict_upper_tri(&self, x: &Tensor, value: f32) -> Tensor {
self.inner.fill_strict_upper_tri(x, value)
}
fn copy_2d_from_cols(&self, src: &Tensor, w: usize, col_offset: usize) -> Tensor {
self.inner.copy_2d_from_cols(src, w, col_offset)
}
fn copy_2d_into_cols(&self, dst: &mut [f32], dst_cols: usize, src: &Tensor, col_offset: usize) {
self.inner.copy_2d_into_cols(dst, dst_cols, src, col_offset)
}
fn repeat_row_as_matrix(&self, weight: &Tensor, rows: usize) -> Tensor {
self.inner.repeat_row_as_matrix(weight, rows)
}
fn apply_rope(&self, x: &Tensor, head_dim: usize, rope_theta: f32) -> Tensor {
self.inner.apply_rope(x, head_dim, rope_theta)
}
fn apply_rope_single_row(
&self,
x: &Tensor,
position: usize,
head_dim: usize,
rope_theta: f32,
) -> Tensor {
self.inner
.apply_rope_single_row(x, position, head_dim, rope_theta)
}
fn concat_dim0(&self, a: &Tensor, b: &Tensor) -> Tensor {
self.inner.concat_dim0(a, b)
}
fn copy_row_2d(&self, x: &Tensor, row: usize) -> Tensor {
self.inner.copy_row_2d(x, row)
}
fn copy_contiguous_into(&self, x: &Tensor, start: usize, dst: &mut [f32]) {
self.inner.copy_contiguous_into(x, start, dst)
}
fn argmax_with_prob(&self, x: &Tensor) -> (usize, f32) {
self.inner.argmax_with_prob(x)
}
}Wiring it in
src/backend/factory.rs gains the "pool" arm, fifth in the list:
use std::sync::Arc;
use super::Backend;
use super::{CpuBackend, Metal, Parallel, PoolCpu, SimdCpu, TracingBackend};
pub fn create_backend(name: &str, enable_tracing: bool) -> Result<Arc<dyn Backend>, String> {
let name = name.trim();
match name {
"scalar" => Ok(wrap_scalar(enable_tracing)),
"simd" => Ok(wrap_simd(enable_tracing)),
"parallel" => Ok(wrap_parallel(enable_tracing)),
"pool" => Ok(wrap_pool(enable_tracing)),
"metal" => Ok(wrap_metal(enable_tracing)),
other => Err(format!(
"unknown backend {other:?} (supported: scalar, simd, parallel, pool, metal)"
)),
}
}fn wrap_pool(enable_tracing: bool) -> Arc<dyn Backend> {
let pool = PoolCpu::new(SimdCpu::new(CpuBackend));
if enable_tracing {
Arc::new(TracingBackend::new(pool))
} else {
Arc::new(pool)
}
}The composition reads the same way as II.4's: PoolCpu::new(SimdCpu::new(CpuBackend)), the SIMD backend with its scalar base, wrapped by the pool. And the module exports:
mod backend_trait;
pub(crate) mod cpu;
mod factory;
pub(crate) mod metal;
pub(crate) mod parallel_cpu;
pub(crate) mod pool;
pub(crate) mod pool_cpu;
pub(crate) mod simd_cpu;
pub(crate) mod tracing;
pub use backend_trait::Backend;
pub use factory::create_backend;
pub(crate) use cpu::CpuBackend;
pub(crate) use metal::Metal;
pub(crate) use parallel_cpu::Parallel;
pub(crate) use pool_cpu::PoolCpu;
pub(crate) use simd_cpu::SimdCpu;
pub(crate) use tracing::TracingBackend;Running it: the thread sweep
The interesting experiment isn't one number, it's the shape of the curve as threads increase. INFERNO_THREADS overrides the default pool size, so:
cargo run --release --bin model-generate -- --kv --backend simd path/to/qwen3-0.6b-q8_0.gguf "Once upon a time, in a small village by the sea, there lived a baker" 64
for t in 2 4 6 8; do
INFERNO_THREADS=$t cargo run --release --bin model-generate -- --kv --backend pool path/to/qwen3-0.6b-q8_0.gguf "Once upon a time, in a small village by the sea, there lived a baker" 64
doneThe single-core baseline, simd with IV.1 and IV.2 in place:
backend: simd
kv cache: basic
metrics:
time_to_first_token_ms: 268.041
decode_tokens_per_second: 45.896
per_forward_ms: min 20.497 max 268.041 mean 25.636 (n=64)The pool at 2, 4, 6, and 8 threads:
backend: pool
kv cache: basic
-- INFERNO_THREADS=2 --
time_to_first_token_ms: 141.477
decode_tokens_per_second: 77.211
per_forward_ms: min 12.014 max 141.477 mean 14.960 (n=64)
-- INFERNO_THREADS=4 --
time_to_first_token_ms: 130.204
decode_tokens_per_second: 109.505
per_forward_ms: min 8.042 max 130.204 mean 11.024 (n=64)
-- INFERNO_THREADS=6 --
time_to_first_token_ms: 138.551
decode_tokens_per_second: 106.230
per_forward_ms: min 8.060 max 138.551 mean 11.431 (n=64)
-- INFERNO_THREADS=8 --
time_to_first_token_ms: 139.970
decode_tokens_per_second: 72.032
per_forward_ms: min 10.606 max 139.970 mean 15.853 (n=64)Decode moved. After three Act 2 chapters where threads couldn't buy decode a single token per second, the persistent pool takes it from 45.9 to 77.2 tokens/sec with one extra thread and 109.5 with four. The thing rayon's overhead made impossible was real the whole time; it just needed a dispatch cheap enough to survive being called 197 times per token.
Figure: the sweep in one picture. Two threads buy real speed, four buy a little more, six buy nothing, and eight give speed back: the barrier waits for whichever core is slowest, and past the six P-cores that means an E-core.
Now the curve's two bends, because they teach as much as the peak.
Four to six threads: flat. 4 threads give 2.39x, not 4x. Decode is memory-bandwidth-bound; that was II.3's central measurement and it hasn't stopped being true. One core can't pull bytes from RAM fast enough to keep even itself fed, and four cores pulling together get more aggregate bandwidth, but they share one memory system and the returns shrink with each core added. By six threads the memory bus is the bottleneck, not the arithmetic, and the sixth thread adds nothing (106.2, statistically the same as 109.5).
Past six: the cliff. 8 threads are slower than 2. This M2 Pro has 6 performance cores and 4 efficiency cores; ask for 8 threads and the OS must put at least two of them on E-cores, which run the same code substantially slower. Here the pool's own design bites back: dispatch is a barrier, and a barrier is II.3's lockstep idea at thread scale. All 32 SIMD lanes finish together; all 8 pool threads finish together; either way the group moves at the pace of its slowest member. split deals every thread an equal share, so every one of the 197 barriers per token waits for an E-core to finish a P-core-sized slice. That is why default_threads asks sysctl for hw.perflevel0.physicalcpu, the P-core count, and stops there.
Prefill also improved, a side effect we didn't order: TTFT halves versus simd, 268 ms to 130 ms. No mystery, the pool backend carries the rayon prefill path that plain simd never had; on this short prompt it matches what --backend parallel would do.
Long context, same 511-token pangram prompt as always, at the default 4 threads against IV.2's single-core result:
backend: pool
kv cache: basic
metrics:
time_to_first_token_ms: 2351.592
decode_tokens_per_second: 34.663
per_forward_ms: min 26.678 max 2351.592 mean 65.142 (n=64)24.3 to 34.7 tokens/sec at 511 tokens of context (1.43x, less than the short prompt's 2.4x because the per-token attention work, which the pool doesn't split, is a bigger slice of each step here), and TTFT drops from 10.9 s to 2.4 s courtesy of the rayon prefill path.
One honest calibration before declaring victory. llama.cpp decodes this same model on this same machine at roughly 71 tokens/sec on a single thread, half again our four-thread number, from better kernels and tighter glue than ours. Cores are not free speed; they multiply whatever per-core efficiency you already have, and ours has headroom. The pool's lesson stands anyway, because it's llama.cpp's own lesson: they built this exact machinery (persistent workers, atomic wakeups, caller participation) because no thread library's general-purpose dispatch survives 200 calls per token.
Where this leaves us
--backend pool is a fifth CPU backend: three spinning workers plus the caller, jobs published with one atomic increment, the II.6 integer matvec split by output rows with byte-identical results, rayon retained where it's genuinely better. Decode: 45.9 to 109.5 tokens/sec on four P-cores. And the sweep mapped the machine itself: bandwidth flattens the curve at four threads, E-cores cliff it at eight.
That closes out the CPU. The one processor in this laptop still sitting idle during decode is the GPU, and II.5 left it that way deliberately: every trip to the GPU cost a dispatch and a synchronous wait, so single-row matvecs stayed home. The same disease we just cured on the CPU (per-job overhead comparable to the job) is the GPU's disease too, two orders of magnitude worse. The next chapter applies the same cure: stop paying per-op overhead, encode the entire forward pass into one command buffer, and ask the GPU for a token, not a matmul.