Start with an acceptable result
OpenAI's model selection guidance addresses the balance between performance, speed and cost. For a business, that comparison becomes concrete only when there is an expected output. ‘Answer well’ needs to become observable criteria.
For request classification, those criteria might include the correct category, use of available information and a handoff when context is missing. For a summary, preserving the facts and avoiding invented conclusions matter.
Compare candidates with the same examples
Use a set of situations representative of the task, with synthetic or authorized data. Apply the same evaluation criteria to each candidate and hold back some examples for a final check. The aim is to avoid choosing solely on the case that worked in the presentation.
- Quality
- Record significant errors, omissions and required corrections. An error's importance depends on the task, not just how often it occurs.
- Time
- Observe the time until there is a usable output, including information retrieval, retries and review.
- Full cost
- Include model usage, supporting services and human effort. A cheaper call can create more work afterwards.
Model size does not decide the answer on its own
An extraction task with a stable structure may have different requirements from an analysis involving conflicting information. Different models can be assessed for different steps, but this adds routing and maintenance decisions. The complexity needs to bring a verifiable advantage.
OpenAI's evaluation documentation recommends turning quality criteria into tests. For A1, those tests should accompany significant changes to the model, instructions or data source. A previous result does not guarantee the same behavior after a change.
A useful first step
Choose a small task and write examples of acceptable and unacceptable answers. Then compare alternatives, including doing the task without AI. If the team still disagrees about what makes an answer good, clarifying the process takes priority.
A process assessment helps define that standard and establish whether there is an automation opportunity. Model selection then follows the task, with costs and limits made visible.
