Int8 SIMD Inference: Making Tiny Models Fast on Phone CPUs
How TinyLM uses 8-bit integer math and SIMD vectorization to run real inference quickly on the modest CPUs inside ordinary phones.
The Performance Question
A model that runs in a browser is only useful if it is fast. Phone CPUs are not powerful by datacenter standards, and there is no GPU to lean on. So how does TinyLM stay responsive? The answer is a combination of int8 arithmetic and SIMD vectorization in the Rust-to-WASM engine.
Why Int8 Instead of Float
Floating-point math is precise but relatively expensive, and floating-point weights are large. By quantizing activations and weights to 8-bit integers, TinyLM shrinks both the memory traffic and the per-operation cost. Integer math is fast on every CPU, and 8 bits is enough precision for a well-trained tiny model to behave well.
What SIMD Adds
SIMD — Single Instruction, Multiple Data — lets the CPU apply one operation to many values at once. Modern phone CPUs have SIMD units, and WebAssembly exposes them through WASM SIMD. The TinyLM engine packs int8 values into vectors and processes them in batches, multiplying throughput several times over compared to scalar code.
Putting Them Together
Int8 and SIMD reinforce each other. Smaller data types mean more values fit in each SIMD vector, so each instruction does more useful work. The engine is written in Rust to keep tight control over data layout and memory, then compiled to WASM so it runs in the browser sandbox. The result is CPU inference that feels instant for short tasks.
Ternary Makes It Even Cheaper
For the ternary models like meeny, many operations reduce to additions and subtractions because weights are only -1, 0, or +1. Combined with int8 SIMD on the activation side, this strips away a large chunk of the arithmetic a normal model would require. Less math means faster responses and longer battery life.
Feel It in Practice
Numbers are abstract; responsiveness is concrete. At https://ai.sprapp.com you can type and watch a model respond with no perceptible delay, all on your phone's CPU. That smoothness is the int8 SIMD engine doing its job behind the scenes.
The Honest Limits
This optimization makes tiny models fast, not large models possible. Int8 SIMD does not let a billion-parameter model run in a browser — it lets a few-million-parameter model run well. And quantization has a quality cost that must be managed in training. The engine is tuned for the tiny regime, and it does not pretend to scale beyond it.
Why It Matters
Most of the world's devices are ordinary phones with no AI accelerator. An inference engine that is fast on those devices, using only the CPU and standard browser features, reaches everyone. Int8 SIMD is the quiet engineering that turns "a model that technically runs in a browser" into "a model that feels good to use." It is the foundation the whole TinyLM experience sits on.