We tested five AI models two ways: — scripted scenarios on the test sheet, and every real conversation in ElevenLabs (—, each read in full). On the sheet the models finish within — points of each other. In real use the gap is wider: GPT 5.6 Luna makes the fewest mistakes but is the slowest to reply, GLM-5.2 is the most dependable of the fast models, and Qwen 3.5, the current model, makes the most mistakes. — scenarios fail on every model and need app fixes.
Score gives full credit for a pass and half credit for a partial result, out of the scenarios each model was tested on. Slow replies are scenarios where the tester wrote down a slow response time (usually 5 to 9 seconds).
Weakest areas at the top. Green is 85% or more, amber 60 to 85%, red below 60%.
Switching models will not fix these. Each needs a product or app change.
| ID | Scenario | Area |
|---|
| ID | Scenario | GPT 5.6 Luna | GPT 6 Luna | DeepSeek 4.1 Flash | GLM-5.2 | Qwen 3.5 |
|---|
We read every conversation the TiAiMe In-App assistant had in ElevenLabs while each model was live: — in total, of which — had a real request (the rest were greetings, silence or empty sessions). Each was scored from its full transcript on the same rubric: did the assistant pick the right tool with the right details, did it say something was done when it wasn't, and did it sound like a person or a machine.
Green marks the best model in each column and red the weakest. Scores and rates use only conversations with a real request. Tasks done leaves out calls where the user left before the assistant could finish. Reply start is the median time for the model to begin answering.
| Model | Test sheet score | Conversations scored | Tool use (of 5) | Sounds natural (of 5) | Tasks fully done | False "done" per 10 calls | Wrong tool or details per 10 calls | Calls with technical speech | Reply starts after |
|---|
Share of each model's conversations (with a real request) where the problem or strength showed up at least once. Darker means more often.
Recommendation: move off Qwen 3.5. Run GPT 5.6 Luna and GLM-5.2 head to head on the same agent version and test script. Pick GPT 5.6 Luna if its reply delay is acceptable in that test; otherwise pick GLM-5.2 and fix the confirmation-card and event-edit issues in the prompt.
Caveats: each model ran in a different week, on a different version of the agent's prompt, mostly with the same QA testers but different scripts. The GPT 5.6 Luna weeks had more and simpler sessions (about 6 tool calls per conversation against 10 to 12 for the others), which is why the harder conversations were also compared separately. Treat this as a strong signal, not a controlled test.
| Model | What the user asked | Outcome | Tool use | Natural | False done |
|---|
Source: the TiAiMe voice-based daily user test sheet, where a tester wrote a free-text note for each model on each scenario. Claude read every note and graded it against the expected result. A slow reply alone still counts as a pass. Partial covers answers that were right but carried wrong extra details, needed an extra confirmation step, or saved something the tester could not see in the app. Blank notes count as not tested and are left out of the score. These grades are a judgment call on the notes, so treat small score differences as a tie.
The conversation review covers all 442 conversations of the TiAiMe In-App agent in ElevenLabs from 18 September to 27 October 2026, each read in full. Each model was matched to the agent version it ran on, and the model was confirmed in every conversation's usage record. A handful of conversations also show a few turns from Gemini 2.5 Flash, an automatic fallback; they are counted under the main model. Tool errors caused by the user talking over an on-screen card were not counted against the model. Reply start uses ElevenLabs' first-token or first-sentence timing, which is close to but not exactly the same measure for every model.