1-Bit LLMs in Pure Rust

A field report from OxiBonsai v0.1.3

Reading time: ~17 minutes

Those who know me know I have a soft spot for Rust. I think the experimentation side of AI will stay firmly in the Python world for years yet, but the delivery side — the production binaries that actually ship inference to users — is going to land in Rust. The ecosystem is building. I've been quietly tracking it.

The thing I keep finding myself wanting is smaller than what the AI press talks about. Most of the time I don't need a frontier model. I don't even need an out-of-process SLM with a sidecar server, a network round-trip, and a separate lifecycle to manage. I want a library. I want to write cargo add some-llm, call a function, get a string back, and move on. In-process, statically linked, no HTTP dance, no API keys, no billing, no GPU required. The vast majority of LLM use cases I care about — classification, summarisation, structured extraction, lightweight agent steps — don't need anything more than that.

That's the lens I've been using as I scan the Rust ML ecosystem. Which brings me to OxiBonsai.

I downloaded an 8-billion-parameter language model that compresses each weight to a single bit, ran it on my workstation, and watched it answer questions. The whole engine — the CPU SIMD kernels, the CUDA kernels, the OpenAI-compatible server — is one Cargo workspace of pure Rust. No llama.cpp. No BLAS. No C, no C++, no Fortran. Not even an FFI shim.

The cool-japan ecosystem

I'm genuinely new to COOLJAPAN, the team behind OxiBonsai. This team is rebuilding most of the scientific and AI computing stack in Rust, deliberately 😉. Each project carries the same banner: zero C/C++ dependencies, zero FFI, sovereign Rust.

I decided to take a walk over some of the repos they have. scirs is SciPy in Rust, the flagship of the family. oximedia is FFmpeg and OpenCV in one unified pure-Rust framework. oxicuda is replacing the entire NVIDIA CUDA toolkit, type-safe — yes, really. oxiblas is BLAS and LAPACK from scratch. oxifft is a 99% port of FFTW3. oxionnx is a pure-Rust ONNX inference engine with 147 operators that runs anywhere, including in the browser via WebAssembly. oxillama is explicitly framed as "the sovereign alternative to llama.cpp."

A snapshot of where each project sits, as of writing:

We can see most of these projects are less than six months old — scirs is the old man of the group at 13 months; oxibonsai and oxicuda are around a month old at the time of writing. And the visible commit count on master is tiny across the board, because the team squashes every release into a single "Availability of 0.x.y" commit. Real development happens in feature branches. Count releases, not commits, when reading activity.

A quick detour: what "sub-2-bit LLM" actually means

If you haven't been following the quantisation literature, here's the gist. A normal LLM weight is a 16-bit floating-point number — bf16 or fp16, two bytes each. An 8B-parameter model in bf16 is 16 GB. That's the lower bound on what fits in GPU memory before you can run inference.

In 2024 a Microsoft team led by Ma et al. published The Era of 1-bit LLMs, with a model they called BitNet b1.58. The trick: every weight in the network is restricted to one of three values — -1, 0, or +1. That's a ternary weight. The "1.58 bits" comes straight out of Claude Shannon's 1948 information theory: three equally-likely states carry log₂(3) ≈ 1.585 bits of entropy. Shannon defined the bit as the unit of information for binary choices; the same maths cheerfully tells you that a ternary choice is worth a bit and a half. Eighty years later that arithmetic is what lets a frontier-scale neural network fit in a gigabyte. Critically, the paper showed performance comparable to 16-bit Llama 2 at the same parameter count, when the model is trained natively in this regime rather than quantised after the fact.

The reason this matters for an inference engine isn't just the memory shrink — though the shrink is dramatic. Eight billion bf16 weights take 16 GB. Pack the same eight billion at 1.58 bits each (using two bits of storage per ternary code, since you can't address fractional bits in hardware) and you're at 2 GB. Pack them as true 1-bit ±1 values and you're closer to 1 GB. The Bonsai-8B model on disk is 1.15 GB — a ~14× shrink against bf16, which is what gets the model into the L3 cache of a laptop CPU instead of straining a 24 GB GPU. But the bigger win is what happens at compute time. When every weight is -1, 0, or +1, the matrix multiplication at the heart of every transformer layer stops needing real multiplies. Multiplying by +1 is identity, by -1 is negation, by 0 is skipping. The whole matmul collapses into adds and sign flips — exactly the kind of work an 8-bit integer dot-product instruction was designed to do.

True 1-bit (binary, -1 and +1 only) takes the same idea further at a small accuracy cost. OxiBonsai supports both: the 1-bit line uses Q1_0_g128 (one bit per weight, packed in groups of 128 with a shared FP16 scale per group), and the ternary line uses TQ2_0_g128 (the BitNet b1.58 encoding, ~1.585 bits per weight). The Bonsai-8B model I tested is the 1-bit variant: 8 billion parameters, 1.15 GB on disk. Same parameter count as a frontier model from 2023, in less space than a copy of Doom Eternal's intro cinematic.

This is the workload OxiBonsai is built for, and it's why the engine's CPU SIMD choices matter so much. The integer dot-product instructions on modern CPUs (AVX-VNNI, NEON UDOT) are exactly what you want for a network whose weights are tiny integers. More on that below.

The audacity is the point. The cool-japan team aren't wrapping the C ecosystem to make this work. They're replacing it.

What it actually does on my desk

I have an Intel Core Ultra 9 285K (Arrow Lake, 24 threads) and an RTX 4080 SUPER. I built three variants of the binary — pure CPU AVX-2, AVX-2+AVX-512, and AVX-2+native-CUDA — and ran the same workload across them.

Decode throughput, sustained over 512 tokens at temperature=0:

Prefill throughput, 812-token prompt:

That's a 7.6× decode speedup and a 74× prefill speedup on the GPU. The prefill number is the headline — single-command-buffer fused matmul does its job. For an 8B model running entirely off Pure Rust kernels with no BLAS underneath, those numbers are real and worth respecting.

The model itself is a Qwen3 thinking model, so every short answer arrives wrapped in 100+ tokens of <think>...</think> chain-of-thought before the actual reply. That's a model property, not an engine property. With max_tokens >= 400, you get correct answers. The math is right when you watch it work through it.

What's right about the codebase

Before I get into what's wrong, I want to spend a moment on what's right, because there's a lot of it and the rest of this post will read unfairly without it.

This is a well-written Rust codebase. Not "fine for an early-stage project" — actually well-written. The signals:

• The SIMD kernels are the showpiece. unsafe blocks are tightly scoped and gated by #[target_feature(enable = "avx512f", ...)] exactly as they should be. The bit-packed block types like BlockQ1_0G128 are #[repr(C)] with const _: () = assert!(size_of::<...>() == 18) size-checks. The zero-copy reinterpret of mmap'd weight bytes is sound. The unsafe-density is unavoidable when you're doing 1-bit AVX-512 intrinsics and Metal/CUDA FFI, but the discipline around it is correct.
• The lifetimes do real work. BonsaiModel<'a> borrows from GgufFile<'a> borrows from the mmap. Weights aren't copied into the engine; they're read in place out of the file. That's the Rust idiom that makes this engine load an 8B model in three seconds — easy to talk about, hard to actually pull off.
• The hot path doesn't allocate. Sampler reuses a probs_buf across every token. TransformerBlock carries explicit ScratchBuffers with pre-allocated Vec<f32> for every intermediate. Someone profiled this. No one starts with this.
• The error model is clean. thiserror with #[from] chains, separate well-named error types per crate. Errors compose; nothing is Box<dyn Error> in the places that matter.
• Test coverage is real. 3,500+ tests, Criterion benchmarks, real property tests with proptest, and chi-squared / KL-divergence tests on the sampler distribution. Not just assert_eq!(2+2, 4) filler.
• The crate-level rustdoc on each lib.rs is excellent — the kind you can land on cold and orient yourself in five minutes.
• The examples/ folder is curated — six runnable examples, each well-commented, walking through actual usage from basic_inference.rs to server_client.rs.

It reads like a team of experienced human Rustaceans, not an LLM wrote this. I see a lot of AI slop code in the wild, this is not it. The variable names are specific. The comments explain why, not what. The commit messages are terse but real. Whatever else you might think about the team's release cadence or their TODO bookkeeping, the source itself is the work of someone who knows Rust at a senior level and cares about getting it right. Tbh, I kind of like the TODO bookeeping and might steal it.

The bugs

I spent a few hours on a Saturday afternoon using the engine in anger. I found three issues. Some of this might have started as user error on my part, but I went deep enough on each one to convince myself the problem is in the code, not in how I was holding it.

1. The OpenAI-compatible server can't serve concurrent users.

I ran 1, 2, 4, and 8 parallel chat completions against the local server (CUDA backend, identical 64-token request per client) and watched the clock.

Wall time scales linearly with N. Aggregate throughput is flat. Two clients hitting the server simultaneously each get the latency of a serialised queue, not the parallelism the OpenAI API shape implies.

Tracing the source confirms it: the server wraps the entire inference engine in a single tokio mutex, and every chat endpoint takes that lock for the full duration of generation. Two clients = two queued sessions. Eight clients = an eight-deep queue.

The shape of the choice is the classic I don't know the minimum I need to lock, but I know it's safe to lock all of it. The author needed &mut access to the engine because generation mutates the KV cache, and rather than tease apart which fields are actually mutable per-request, they wrapped the whole thing. I get it — I've shipped that exact shortcut. The trouble is, almost everything inside the engine is gigabytes of read-only model weights that never need a lock; only the KV cache and the sampler's RNG are genuinely per-conversation state.

The fix is to split the engine in two. A ModelHandle carries the immutable bulk — embedding table, transformer blocks, kernel dispatcher — and gets shared across requests as Arc<ModelHandle>. A Session carries the per-conversation KvCache and Sampler and borrows the model handle. Each request constructs its own Session; concurrent decodes read the same weights in parallel. Bound the number of live sessions with a semaphore sized to your RAM budget so a flood of clients can't OOM the box. That's a weekend's work and unblocks real CPU concurrency. The GPU path is harder because CUDA stream sequencing is its own beast, but the existing continuous_batch.rs module is the right answer there if someone wires it in.

I've written about why locks like this are quietly expensive in The Lock You Didn't Know You Were Taking — same shape of problem here.

2. The OpenAI server discards your sampling params, and the kernels disagree about what "greedy" should produce.

I started by sending five fixed prompts with temperature: 0 in the JSON body to the OpenAI-compatible server, expecting greedy decoding. The outputs across backends matched on three prompts and diverged sharply on two, which I initially read as a CUDA correctness bug. Then I re-checked the server. The chat handler only forwards body.max_tokens to the engine — body.temperature is parsed off the JSON and silently discarded. The engine generates with whatever sampling defaults were baked in at server boot (which is not greedy in this build). That's a spec violation and a real bug in itself.

So I bypassed the server and went direct via the Run subcommand, which exposes --temperature 0 --seed 42 and pins the kernel-level sampler properly. Re-tested the same five prompts on both binaries with RUST_LOG=error to strip log noise. Four of five prompts produced byte-identical output across AVX-2 and CUDA — the README's correctness gate works for those four. One prompt diverged: AVX-2 produced a sensible chain-of-thought working through the question; CUDA produced gibberish ("away from the city. The city was a place of opportunity..."). Re-running the failing prompt three times in isolation on CUDA gave byte-identical gibberish each time, so the kernel path is deterministic; the divergence from CPU is real.

Worse, when I ran the same prompt as part of a batch versus on its own, both backends produced different output for that one prompt — same --seed 42, same flags, same binary. Some host-level state (likely CUDA driver caches or NVRTC kernel JIT) is leaking across process invocations and influencing decode. That's not what seed=42 is supposed to mean. There is no automated test in the suite enforcing the README's "byte-identical at temperature=0 seed=42" claim across kernel tiers, which is presumably why this corner went unnoticed.

Net of all this: the README claim is mostly true (4 of 5 prompts agreed), the chat server can't even let you test it (temperature discarded), and where the kernels do diverge, the CUDA output is the qualitatively-worse one. None of that is fatal, but it does mean the "byte-identical" gate the README leans on needs both a test and one kernel bug fix before it's a real guarantee.

3. The SIMD story is more complicated than the docs let on.

They built AVX-512 support, but most users will never get to use it. I built the engine with --features simd-avx512 expecting the 512-bit kernels to take over. Auto-detect quietly picked AVX-2 instead. cpuid and /proc/cpuinfo confirmed: this CPU doesn't have AVX-512. Intel removed AVX-512 from consumer parts after Alder Lake to keep ISA parity with the E-cores — every Raptor Lake, Meteor Lake, Arrow Lake, and Lunar Lake CPU shipped in the last four years has none of it. AMD Ryzen 7000 and 9000 do, and Xeon Scalable does, but that's a sliver of the deployable installed base. The docs list AVX-512 as a CPU tier with no caveat, so the typical Intel laptop or desktop user enables the feature flag, sees a clean compile, and silently runs on AVX-2 anyway. The team built the feature; they just aimed it at a rung most users can't reach. (The Parallel Lanes Nobody Uses covers how the register zoo evolved.)

And there might be a tier that would have reached more of them. Between AVX-2 and AVX-512 sits an instruction set the engine doesn't currently use: AVX-VNNI. INT8 dot-product, 256-bit, present on every Intel CPU since 12th-gen and every AMD CPU since Zen 4 — every x86 chip you can buy new today. From the outside, looking at the kernel for an afternoon, it's tempting to say that's the rung you want — a 1-bit or ternary network is a machine for doing absurd numbers of small integer dot-products, which is exactly what vpdpbusd was designed to accelerate. The current AVX-2 path converts the integer weights to f32 and uses an FMA, which feels wasteful when one VNNI instruction would do the same work in integer space.

But "tempting from the outside" is doing a lot of work in that sentence. The team that built clean #[target_feature]-gated SIMD across three platforms did not skip VNNI by accident. They might have run the numbers and found the float path is faster on this workload because AVX-VNNI's INT8 dot-product saturates differently than the math demands, or because the weight-to-int8 conversion overhead eats the benefit, or because the f32 accumulator paths through the rest of the pipeline (RMSNorm, RoPE, softmax) make a mid-stream integer detour cost more than it saves. There are good reasons hidden in profiles I haven't seen. ARM has a parallel gap with NEON's UDOT/SDOT dot-product instructions — they've shipped on every iPhone since 2017, and the kernel doesn't touch those either, presumably for similar reasons.

What I can say is that the tier ladder is one of the few places where the visible coverage looks like it leaves throughput on the table. Whether it actually does is a question only a real benchmark — VNNI tier vs current AVX-2 tier on the same hardware — can answer. That benchmark might be the most interesting thing someone could contribute to this project.

None of this is a bug. Either the team evaluated VNNI/UDOT and decided the f32 path was faster on real workloads — in which case the gap I'm pointing at isn't really a gap and the docs could just say so — or they prioritised the AVX-512 tier first and these are still on the backlog. I don't know which. I do know that the tier ladder is the single most leverageable surface area in this engine for unlocking deployable CPU performance. Whichever way the answer falls, it's worth chasing.

Where this leaves me

A new and ambitious project at v0.1.3 isn't going to do everything I want it to do. That's not a criticism — that's how new things start. The point of writing this up isn't to enumerate flaws; it's to mark where the project sits and what it's reaching for. The Rust is real. The SIMD discipline is real. The sub-2-bit math is sound. The throughput numbers, when the GPU path engages cleanly, are competitive with anything in the open-source low-bit inference space. The bugs I found are tractable. The biggest one — the server's single-mutex serialisation — is a refactor, not a redesign. The CUDA correctness corner is a kernel chase, not a fundamental flaw. The SIMD-tier question is genuinely open and might be answered by someone with a Zen 4 box and an afternoon.

So the answer to the question I started with — can I cargo add a Rust LLM library, call a function, and move on? — is almost, but not yet, and worth watching closely. The library shape is right. The kernel work is the kind you can't shortcut. The polish around the edges (server concurrency, test coverage of correctness gates, ladder coverage) is the kind that lands across the next few releases if the team keeps shipping at this pace.

I'll be keeping a close eye on OxiBonsai and on the broader COOLJAPAN family. If even half of what they're trying to ship lands at production quality over the next year, the Rust ML ecosystem will look meaningfully different on the other side. That's a project worth watching, and a team worth commending.

Plenty of people are paying attention. You should be too.

I'm writing a book about what makes developers irreplaceable in the age of AI. Join the early access list →

Naz Quadri spent an afternoon proving his CPU doesn't have AVX-512 with three independent tools, just to be sure he wasn't guessing. He blogs at nazquadri.com. Rabbit holes all the way down 🐇🕳️.

Further Reading

• OxiBonsai on GitHub — the project itself. Read the README, then the Cargo.toml, then the SIMD kernel files.
• The cool-japan organisation — the broader sovereign-Rust ecosystem. Start with scirs and oximedia.
• The Parallel Lanes Nobody Uses — what AVX-2, AVX-512, and AVX-VNNI actually do, and why most code never uses any of them.
• The Lock You Didn't Know You Were Taking — why a single mutex around a hot path costs more than it looks.
• AVX-VNNI on Wikipedia — context on the 256-bit INT8 dot-product instruction set, why it exists, and which CPUs ship it.