What is latency? Time first output separately from completion
Explain end-to-end latency, first output and queueing under matched inputs and quality criteria.
Understand the concept
Latency measures waiting from a defined start to a usable result. First-token time differs from completion and tools, networks or queues contribute. Define boundaries and match inputs and quality. Early text does not establish faster completion, and loading is not all generation time.
Full requests include time outside the model
OpenAI discusses generation length and request organization; vLLM distinguishes first-token, queue and end-to-end metrics. Fast internal computation may still face buffering or parsing. Streaming shows partial output earlier while structures need completion. Check whether timing starts at submission, receipt or inference and ends at first content or usable output before comparing numbers.
Record cold and steady requests for identical inputs
Use fixed public material and fields with matched model and concurrency. Separate first loading from steady requests and include retries. Faster shorter output must still meet requirements. Include tool completion, not only model text. No timing test or speed ranking occurred here. Preserve inputs and settings for repeatable conditions.
Set latency goals together with quality and task type
Interactive chat values prompt feedback while offline work may prioritize volume by a deadline. Inspect slow-request distributions because averages hide long waits. Recheck quality after cache, concurrency or model changes instead of silently lowering requirements. Include recovery effort. Evaluate usable task completion rather than extending fast first text to the whole workflow.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.