A bad answer is the start of an investigation
One disappointing response can identify a useful test case. It cannot establish how a model performs across your work. OpenAI's evaluation guidance describes structured testing for nondeterministic systems and warns about judging biases, including a preference for longer answers. That makes a fixed rubric more useful than asking whether an answer feels intelligent.
The procedure below is a small screening exercise proposed by VSike. It is not a published benchmark, an independent Astra performance result, or a substitute for a larger evaluation where mistakes are costly.
Choose ten tasks you actually need
Our suggested starter set has two tasks in each of five groups: extracting facts, reasoning with constraints, following instructions, editing a real artifact, and completing a task that requires a tool. Replace groups that do not match your use. Include ordinary requests as well as cases that have previously failed.
Before testing, save the source materials and define an acceptable result for each task. For extraction, list the required fields and reference values. For a calculation, save a verified answer. For an article update, require accurate claims, working references, and the requested format. For a tool task, inspect the resulting file or application state.
Keep the tasks fixed for the first comparison. Adding only examples that favor your preferred model makes the result less representative. Maintain a separate collection of new failures for the next evaluation round.
Define the comparison before running it
A model comparison uses the same prompt, inputs, tools, and acceptance rules with different models. A regression check compares an earlier and a later run of the same setup. If an earlier model version or its original inputs are no longer available, label the comparison as limited; an old screenshot alone cannot recreate the conditions.
- Record the date, timezone, product, client version, and visible model selection or API model identifier.
- Record reasoning settings, enabled tools, and any usage-limit or access notices.
- Save the exact prompt, attachment versions, relevant conversation history, and output.
- Record live dependencies such as search results; a changing webpage can change the answer independently of the model.
The Astra model page includes the currently available API identifiers. Use a documented versioned snapshot if one is available for your model and account. Do not invent a dated identifier; if only an alias is available, note that limitation in the comparison.
Repeat each task and grade without the label
For a lightweight first pass, try three runs per task per condition. Start independent conversations with the same supplied context and avoid correcting one condition while leaving the other untouched. Alternate the order when comparing two configurations so that service timing is less likely to favor one consistently.
Hide the model name while grading where practical. Use separate pass/fail checks for factual correctness, required content, and successful execution. A tone preference can have its own score. For code, use a relevant behavior check; for a spreadsheet, verify the referenced cells and formulas. A model's claim that the task is finished is not the completion check.
If you use another model as a judge, review a sample yourself and inspect disagreements. Keep both original outputs. Otherwise, an evaluator can quietly reward confident wording or length instead of the properties your task requires.
Report counts, uncertainty, and the next action
A useful report might say, "Condition A passed 24 of 30 attempts; condition B passed 19 of 30. Four additional failures involved reading the same attachment." Those are hypothetical numbers illustrating the reporting format, not measured GPT-6 results. Such a finding suggests checking the attachment path before announcing a general reasoning decline.
Three repeats across ten tasks produce correlated, small-sample evidence. Report outcomes per task as well as overall counts, keep failure examples, and avoid declaring statistical certainty from that screening set. Expand the relevant task group when the result would affect a deployment or purchasing decision.
If correctness stays stable while review time improves, use the efficiency method to measure the benefit. If failures increase, apply the diagnostic checklist, document the smallest reproducible case, and send that evidence to the appropriate support or engineering team.