Local inference used to fail for a mundane reason: the weights did not fit. A model stored at 16 bits per parameter needs roughly two gigabytes for every billion parameters, which put anything interesting out of reach on a typical laptop.
Quantisation changed the arithmetic
Storing weights at four or five bits, with careful handling of outlier values, cuts footprint by roughly three quarters while keeping output quality close to the original for many tasks. That single change moved mid-sized models inside consumer memory budgets.
Bandwidth sets the speed
Token generation reads the whole active weight set for each token, so output speed tracks memory bandwidth almost linearly. A system with a fast neural accelerator but modest bandwidth will feel slower than one with less compute and wider memory.
When evaluating a machine for local models, compare memory capacity first, memory bandwidth second, and accelerator throughput last.