A local model runs out of memory: should you change quantization or shorten the input?
First distinguish memory exhaustion while loading weights from failure after the input grows longer. For the former, check weight size and quantization support; for the latter, also inspect the KV cache, batch size and generation length.
Start troubleshooting here
First distinguish memory exhaustion while loading weights from failure after the input grows longer. For the former, check weight size and quantization support; for the latter, also inspect the KV cache, batch size and generation length. Shortening input does not shrink weights that cannot fit, and quantizing weights alone does not guarantee that a long context will run.
Locate the failure during loading or generation
This applies when local inference reports an allocation failure or a model starts but exits on longer questions. Record the device and actual weight format, and distinguish GPU memory from system RAM. Save the runtime's peak-memory measurements and distinguish allocated memory from reserved caches rather than assuming that a displayed free-memory figure guarantees capacity.
Check whether weights load with a minimal input
Start with a minimal input and a small output allowance to establish whether the weights load. If loading still fails, check parameter scale, quantization bit width and whether the backend recognizes the format. If a short input works, increase input length gradually and observe cache growth with tokens, changing only one variable at a time.
Reduce weight and cache usage separately
For loading failures, choose a smaller model or quantized version supported by the runtime and check for additional full-precision copies. For generation failures, reduce concurrent requests and batch size before trimming irrelevant history and output allowance. Cache offloading also requires sufficient system memory and tolerable transfer overhead; it is not a cost-free speed improvement.
Retest target lengths and task quality
Cover loading, short questions and the intended input length, saving peak memory for each. Check whether quantization preserves critical task results, especially numbers and strict formats. One occasional success does not establish sustained stability: include consecutive requests and a restart after cleanup.
Memory measurements differ across cache implementations and hardware backends, so model-file size alone cannot establish total usage. This page gives no capacity promise for a particular GPU and does not recommend terminating other people's processes. Coordinate resources on shared machines; retain errors before changing the model or runtime if offloading still fails.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.