What is evaluation — CCA-F Exam Prep

PencilPrepPencilPrep
L1.30|What is evaluation
1/12
Countdown
A countdown timeline. Day -5: dashboard showing 'All Tests Passing' in green, team celebrating. Day 1: customer chat showing 'Your order has shipped!' with a red annotation: 'It hasn't.' Day 2: chatbot offering '90% discount!' with red annotation: 'Doesn't exist.' Day 3: a screen showing 'BOT OFFLINE.' Dramatic progression.

5 days before launch. The AI customer service bot passes all internal tests.

Day 1 live: it tells a customer their order shipped when it hasn't. Day 2: it offers a 90% discount that doesn't exist. Day 3: pulled offline. The team scrambles.

The bot passed every test. Every scripted scenario. Every QA check. The tests were right. But the tests only covered the happy path.

Production is not the happy path. The evaluation was wrong.