Small or large model: establish a classification baseline
Compare small and large models on matched classification tasks, considering labels, error groups and actual usage rather than parameter counts alone.
Choose an approach
Establish a model that meets requirements, then test whether a smaller one preserves quality. Smaller models may simplify deployment or reduce latency, but are not automatically cheaper; larger models cannot repair ambiguous labels. Check every class and out-of-scope inputs consistently.
Size does not replace classification evidence
Size is associated with parameter count, while training and task design also affect performance. OpenAI recommends meeting accuracy needs before optimizing cost and latency. Define labels and confusing counterexamples before ranking models by size.
Different predictions may reflect overlapping class definitions rather than model weakness. Have maintainers label the same inputs independently and resolve disagreements before comparison. A larger model does not replace missing evidence.
Hold inputs and label rules constant
Sample normal traffic and rare edge cases, reporting errors per class. Use the same instructions and permitted labels, documenting interface differences. Overall accuracy can hide failures in a small but important category.
Include short descriptions, irregular spelling and irrelevant inputs. Check forced classifications and machine-readable output. This is an evaluation design, without measured winners or a universal passing threshold.
Count review and adaptation work
Record input and output usage plus correction time. Frequent retries can outweigh a small model’s per-call savings. For self-hosting, measure throughput and concurrency; pricing cannot be inferred from parameter count.
Route uncertain inputs to another model or a person if the smaller model is reliable only for some classes. Recheck retained cases after label or version changes, including routing mistakes. Keep reproducible evidence rather than relying on a demo.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.