The first model request is slow: how do you separate loading from generation time?
For a slow first response, time process startup and model loading separately, then record the first token and generation completion rather than calling the whole wait inference time. Compare cold and already-loaded requests with identical input to locate the dominant cost.
Start troubleshooting here
For a slow first response, time process startup and model loading separately, then record the first token and generation completion rather than calling the whole wait inference time. Compare cold and already-loaded requests with identical input to locate the dominant cost. Also distinguish service processing from DNS, connection and queue delays.
Fix the input and establish whether the model stays loaded
Use this process when the first question is slow but the next identical question is faster, or a model becomes slow again after inactivity. Record whether the runtime unloads idle models and whether requests pass through a proxy or queue. Fix evidence and the output budget before comparison when input lengths differ substantially.
Measure loading separately from the first useful content
Where local logs are available, separate the end of weight loading from the start of generation. On the client, record connection establishment and the arrival of the first useful content, excluding empty events and heartbeats. Rendering completion is not model completion; without server logs, explicitly identify stages that can only be measured together.
Compare cold and already-loaded requests
Run one fixed short task on an idle service as a cold-start sample, then repeat with identical parameters while the model stays loaded. Record input and actual output lengths and compare first-token latency with later generation speed. Change model retention only when supported and resources permit; test cache changes separately from loading-policy changes.
Report waiting stages and actual output length
Distinguish initial-loading and already-loaded states and state whether the model remained in memory. Success means locating the main waiting stage while the target task returns completely within an acceptable time. If a shorter output produced a faster result, report that length change rather than presenting it directly as faster model computation.
Cold starts depend on storage, memory and scheduling; there is no fixed speed for all devices. This page benchmarks no specific GPU and does not recommend clearing shared-machine caches or terminating other work for testing. If model state is unobservable, report end-to-end latency without asserting that the second request was a cache hit.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.