Every few months a demonstration circulates of a capable AI model running entirely on a laptop or phone: no cloud, no account, no data leaving the device. The demos are real. The explanation that usually accompanies them, that clever compression made the model small enough, is only half the story. The other half is the physical layout of the machine.
The bottleneck is not compute
When a language model generates text, it produces one token at a time, and for every token it has to read essentially all of its parameters. A model with several billion parameters, stored at a couple of bytes each, is several gigabytes of data that must stream past the processor for every word.
The processor is rarely the limit. Modern phone and laptop chips can multiply numbers far faster than they can fetch them. What decides the speed is memory bandwidth: how many gigabytes per second the memory system can deliver. Double the bandwidth and, for this kind of workload, you roughly double the words per second.
That is why the devices that run local models well share a specific design choice. They place memory very close to the processor, often in the same package, with a wide bus between them. It is also why devices with plenty of storage but modest memory bandwidth struggle regardless of how fast their chips are on paper.
Quantisation is a bandwidth trick
The compression techniques that make local models feasible, collectively called quantisation, work by storing each parameter in fewer bits. A model stored at four bits per parameter needs a quarter of the memory traffic of the same model at sixteen bits. The quality cost is real but often small, and the speed gain is roughly proportional to the reduction in bytes.
In other words, quantisation is a way of buying bandwidth you do not have. It is a software technique in service of a hardware constraint.
The dedicated accelerator question
Phone and laptop makers now ship dedicated neural processing units alongside the main processor and the graphics chip. Their advantage is efficiency: they can do the specific arithmetic that models need using far less energy, which matters when the battery is the budget. Their limitation is that they share the same memory system, so they hit the same bandwidth wall. A more efficient engine does not help if the fuel line is the constraint.
Where these accelerators shine is in the many small, continuous tasks that were previously too expensive to run all the time: transcription, image understanding, on-device search over your own files. The headline chat-with-a-model demos are the least representative use of them.
Why it matters beyond speed
Local inference changes who sees your data. A model running on your device can read your messages, photos and documents without any of it being uploaded. That is a meaningful privacy shift, and it also changes the economics: a feature that costs the vendor nothing per use is a feature they can ship to everyone by default.
It also changes what fails. A cloud model fails when the network or the company does. A local model fails when your device is too old, which pushes the upgrade cycle and raises a fairness question about who gets the good version of the software.
What to watch
Ignore parameter counts in marketing. Look for memory bandwidth figures, how much memory is actually available to applications and whether the vendor publishes which tasks run locally and which fall back to the cloud. The interesting progress in local AI over the next few years will be visible in chip layouts, not model names.
