What the interviewer is checking
AI is a means, not the goal
You can say when a rule, a search box or a better form would beat a model, and why this problem genuinely needs one.
You plan for being wrong
Models make mistakes. Strong answers design what the user sees when the output is wrong, uncertain or missing.
You can evaluate it
You'd define what a good output is, build a set of examples to test against, and decide what quality is good enough to launch.
You know what it costs
Latency and cost per request shape the product. A feature that takes ten seconds or costs more than it earns is a design problem.
How to run the answer
Start with the user problem
Same as any product question: who, what job, why now. Then ask whether a model is the best tool for it.
Describe inputs and outputs
What goes into the model (the user's data, documents, context) and what comes out. This is where privacy and data rights come up; raise them yourself.
Design for the failure cases
Wrong answers, confident nonsense, harmful content, empty results. Say how the product shows uncertainty and lets people correct it.
Define quality and the launch bar
An evaluation set built from real cases, the measures you'd grade on, and human review where the stakes are high. Launch to a small group first.
Close the loop
How user corrections and feedback become better examples, and which metric tells you the feature is earning its cost.
Common misses
Model trivia instead of product judgment
Naming model architectures doesn't answer the question. What the user experiences does.
Ignoring the wrong answer
A design that only works when the model is right will fail in the first week.
Accuracy as the only metric
Users care whether the task got done. Pair a quality measure with an outcome metric and a cost line.
Worked example
Design an AI feature that drafts replies for a customer support team.
- Picks agents handling repetitive billing tickets, where drafts save the most time.
- Drafts from the help center and the customer's history, never sends on its own, and cites the article it used.
- Shows low-confidence drafts as suggestions only, and routes refund and legal topics to a human.
- Builds an evaluation set from past resolved tickets and launches to one team first.
- Tracks handle time and reopen rate together, plus cost per drafted reply.