What is video memory? Weights are only part of runtime usage
Explain GPU memory versus system memory and how loading, context and concurrency affect capacity checks.
Understand the concept
Video memory is GPU device memory for computation data. Weights, context caches and temporary work all consume it. It differs from disk size and cannot always be simply added to system memory. Check models, input lengths and concurrency; one successful load does not establish sustained long-task capacity.
Allocated and allocator-reserved memory differ
PyTorch distinguishes tensor allocations from allocator reservations, so monitoring numbers need context. Reserved cache is not all active weights. Transformers documents attention-cache memory during generation. Longer inputs or concurrent requests change peaks even with unchanged parameters. Identify processes and metrics rather than predicting capacity from an idle screenshot.
Inspect loading and generation peaks under one configuration
For one model, record post-load memory and then target context and concurrency peaks. Match backend and precision when comparing. Preserve lengths and failure points for out-of-memory errors. No device or memory monitor was tested here. Shared or unified memory needs device-specific interpretation rather than assuming discrete-GPU behavior.
Freeing memory does not validate task quality
Quantization, shorter contexts or lower concurrency can reduce usage but affect different things. Lost input evidence or changed models require task revalidation. Keep recovery settings and recheck peaks after upgrades. Include capacity headroom and recovery. Determine memory from real workloads rather than promising all scenarios from a fixed parameter multiplier.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.