How to choose an AI model: compare the same task and inspect its results
Define your input and acceptance criteria before narrowing the candidates. This guide provides a practical comparison record and separates documented capabilities from task results, without presenting public rankings or demonstrations as HotAI tests.
Fix the task before comparing model versions
Begin with the required deliverable. Give candidate versions the same input, comparable tool permissions, and output requirements, then inspect acceptance. A family name identifies provenance but does not replace a version or product interface. This page explains a comparison method; it does not report a cross-model benchmark, unmeasured prices, or accuracy scores.
Exclude candidates that do not support the input
For screenshots, verify image input on the exact model instead of checking only whether the app accepts attachments. A product may extract text before passing it to a text model, losing color or spatial information. For text classification, a simpler candidate can provide an initial baseline.
GPT, Claude, and Gemini have separate model and product pages. Versions, tools, and limits can differ between a chat app and an API. Record the actual callable model and confirm its status in the official catalog. Do not pool different services under a family name.
Include difficult cases in the task contract
Choose representative tasks and keep expected answers or supporting evidence. Identifier extraction should include leading zeros; policy summaries should include exceptions after the general rule. Easy demonstrations do not reveal the inputs that create rework.
Write acceptance criteria before seeing the output. Record formatting and factual failures separately, and define the expected response when sources lack an answer. Keep some cases out of prompt tuning to check whether the new configuration only fits familiar examples.
- Keep the input and evidence, assign a stable case ID, and specify fields that must remain unchanged.
- Define acceptance and allowed differences before running the same inputs with each candidate.
- Keep failed outputs and the locations needing correction, then replay them after changes.
Control settings and disclose unavoidable differences
Use the same task brief and source material. If sampling or reasoning settings are incompatible, record their actual values instead of pretending they match. State whether search or code execution is available because tools can change the answer.
State where timing starts and ends. The first text and a complete usable file are different milestones; initial loading may also differ from later calls. Preserve failed acceptance results instead of treating a fast incorrect response as a usable deliverable.
| Record | Enter the actual observation | When not measured |
|---|---|---|
| Model and interface | Exact version and product or API | Availability unconfirmed |
| Task result | Accepted case IDs and raw failures | Not run |
| Completion wait | Timing boundaries and measured value | N/A |
| Cost | Actual billing scope and conditions | Not calculated; no inferred ranking |
Choose the next change from the failing stage
If retrieval never found the evidence, a larger model may not solve the problem. Locate the failure in retrieval or generation before choosing a fix. For exact calculations, consider a formula or program rather than relying unnecessarily on text generation.
Keep a simpler candidate as the baseline when it completes the task reliably, and test harder candidates on the failures. Model routing also needs evaluation of the routing decision; do not show only the best isolated result from each model. The suggested process still needs evidence from your own task.
Retain versions and rollback evidence after deployment
Replay retained failures after a model or default configuration changes. If an old task breaks, save the request and result before comparing conditions. Changing prompts and every dependency at once makes the regression difficult to locate.
Scheduled editorial review can discover deprecations, but an accessible page does not establish suitability for your account. Check review dates and updates, and use the research page for execution evidence. Tests that were not run must remain labeled as such; a network observation is not a test date.
Method reference
These sources explain methods and technical conditions. This is a task guide; samples and results from HotAI tests are documented separately in the linked tested guides.