Serverless versus persistent inference: measure cold starts and idle resources
Compare demand-based scaling with retained instances, checking loading, concurrency queues and recovery for inference services.
Choose an approach
Consider demand scaling for intermittent traffic with supported resources and retained instances for sustained load or controlled first-request waits. Serverless can keep minimum instances and persistent services can scale. Names alone do not determine cost or latency. Measure cold and warm operation and confirm target model loading.
Autoscaling does not mean reloading for every request
Cloud Run can scale to zero and retain minimum instances; Kubernetes Deployments maintain desired replicas. Check billing and hardware per service; serverless does not eliminate servers. Weight loading affects new-instance latency and persistent instances can consume idle resources. Both need health and capacity checks. Process survival does not establish model readiness; record application startup and model readiness separately.
Test idle recovery and sustained requests with the same model
Fix model, input length and concurrency and compare first requests after idle with sustained traffic. Separate loading and queueing instead of comparing cold requests against warmed ones. Simulate instance loss and check recovery, failures and duplicates. Mark unsupported models inapplicable rather than substituting smaller ones. No cloud resources or load tests were created and no default cost or speed advantage is claimed.
Maintain scaling and recovery for actual traffic
Revisit minimum and maximum capacity as traffic changes; persistent services also need rollouts and resource cleanup. Include idle time, storage and retries, not only saturated throughput. Keep recoverable images and settings and replay cold starts after upgrades. Mix on-demand low-frequency tasks with retained frequent workloads when outputs and states are checked consistently. Choose actual latency and maintenance needs rather than fashionable deployment labels.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.