1-Bit LLMs in Pure Rust — A field report from OxiBonsai v0.1.3
A hands-on field report. Quadri built three variants of the engine on an Intel Core Ultra 9 285K with an RTX 4080 SUPER and measured decode and prefill throughput across the CPU AVX-2 and CUDA backends, reporting a 7.6× decode and 74× prefill speedup on the GPU path. He then read the source: scoped and feature-gated unsafe, borrowed weights read in place from the mmap, an allocation-free hot path, per-crate error types, and a test suite including property and distribution tests. His assessment is that the code reads as senior-level human Rust rather than machine-generated filler.
The report also documents three defects — the OpenAI-compatible server serialising all requests behind a single mutex, that server discarding the temperature field, and a CUDA/AVX-2 divergence at temperature=0 — and raises an open question about whether an AVX-VNNI kernel tier would outperform the current AVX-2 float path.
Alongside OxiBonsai he surveys the wider organisation — SciRS2, OxiMedia, OxiCUDA, OxiBLAS, OxiFFT, OxiONNX, OxiLLaMa — with a table of stars, releases, first-commit dates and line counts, and notes that release count rather than commit count is the meaningful activity signal for a repository that squashes each release.