Version note · updated November 3, 2026. The model and device results discussed here refer to the historical implementation at a030ba7. The source reviewed for this update, 971a3e1, adds arithmetic guards and CPU reproduction checks. Those checks do not revalidate historical whole-model or CUDA results for the changed source.
A fixed seed does not fix every source of variation in an LLM. The same request can enter a different batch, use a different reduction kernel or run on another device. Small numerical differences can change a token choice and then the rest of the continuation.
BitExact LLM controls this source of execution variability. It implements Llama-3.2-1B inference with integer arithmetic, including the nonlinear functions and sampling rules. The target is matching logits and sampled tokens under explicitly tested execution changes.
Conceptual illustration of bit identity. The displayed bit string is a symbol, not an experimental output.
Fix the arithmetic, not just the random seed
Floating-point addition depends on summation order. Fixing matrix tiles and reduction schedules can remove batch-dependent variation within a particular implementation. Cross-hardware agreement asks more: different implementations must perform equivalent arithmetic.
Integer sums are order-independent when intermediate values stay within their allowed range. BitExact uses fixed-point representations and explicit integer rounding rules. Matrix products split values into small integer limbs, compute partial products and combine them into wider accumulators. Nonlinear operations use integer procedures or precomputed tables, so inference does not call device-specific transcendental functions.
This moves the hard part into the contract: the scales, rounding, tables and intermediate bounds must all be specified. Integer arithmetic alone is not sufficient if an intermediate overflows or a hidden floating-point operation changes a decision.
Three execution paths
The historical implementation separates three paths:
- Exact mode retains the original bf16 weights in linear layers and evaluates their products with quantized fixed-point activations. The input embedding is rounded once to the residual stream’s fixed-point grid. “Exact” describes the specified arithmetic, not equality with a real-valued or fp32 model.
- Int8 mode uses quantized weights and a cheaper integer path. The vLLM plugin uses this mode.
- Batch-invariant floating point fixes Triton execution choices. Its reproducibility scope is one GPU architecture and Triton version, rather than the integer paths’ tested cross-device scope.
Keeping these separate prevents an accuracy result from one path being paired with a speed result from another.
What the checks actually compare
The historical exact-mode identity checks hash the entire int64 logit row and compare sampled tokens at each generated position. They cover changes in batching, KV-cache use and selected kernel implementations on H800, H100, x86 CPU and Apple M4 Pro CPU. The same constants file identifies the model under test.
The vLLM tests ask a different question: does the int8 path remain stable when a scheduler changes execution? The recorded cases include mixed requests, chunked prefill, shared prefixes and preemption. The tests described at that revision use the offline engine API with enforce_eager=True; the plugin can capture its own decode graphs. They do not establish compatibility with every vLLM configuration or the serving API.
A distinction in the reported evidence is especially important: the large teacher-forced accuracy evaluation and the cross-device identity tests are different experiments. A count of positions in one is not evidence of identity in the other.
Reproducibility has a cost
This is not a general throughput win. Against stock vLLM with CUDA graphs, the integer paths have substantial overhead, particularly in prefill and larger batches. A comparison against eager Hugging Face inference can give a much more favorable result because that baseline has different launch overheads. It answers a different performance question.
The unavoidable work includes extra limb products, wider intermediate representations, nonlinear integer operations and deterministic sampling. The implementation must preserve these semantics while reducing their cost. Faster kernels are useful only if they retain the equality being tested.
At the historical revision, some outputs were checked only after residual addition, and final logits were not checked. The updated README records checks before exact-mode residual addition and for final logits, plus linear-epilogue shift guards. These changes improve detection within the stated arithmetic contracts; they do not prove every operation overflow-free. The new CPU checks do not establish whole-model or CUDA identity for the changed source.
Applications in evaluation and compression
For behavior research, reproducible inference removes one source of variation when comparing interventions. For neural lossless compression, matching probability computations matters because the decoder must reconstruct the encoder’s decisions. These applications share a need for execution that can be compared precisely.
The useful deliverable is therefore a bounded, inspectable contract: which model and constants, which arithmetic, which execution changes, and which outputs were compared. The repository contains the implementation and reproduction commands; the pinned README records the results and limitations described here.
Comments