How AI Interviews Score Candidates: What to Measure, What to Never Score

What an AI interview actually is

An AI interview is a structured interview where the questions, the follow-ups and the scoring rubric are held constant by software instead of by an interviewer's memory. Some are async video. Some are live voice. Some are text. The format matters far less than whether the scoring is defined before the interview starts.

If a tool cannot show you the rubric before the candidate answers, it is not running a structured interview. It is running a vibe check with a transcript.

The three things worth measuring

Job-relevant knowledge. Can the person reason about the actual problems in the role. This is measured by follow-up depth, not by keyword matching.

Work sample behaviour. Given a realistic scenario, what do they do first, what do they ask for, what do they refuse to guess at.

Consistency under probing. The same claim, examined twice from different angles. This is the single strongest signal AI adds, because a machine never gets bored of asking "how did you know that was the cause?"

The three things that should never be scored

• Facial expression, eye contact or "enthusiasm" from video. The evidence base is weak and the disparate impact risk is high.
• Accent, speech rate or fluency, unless the role's core requirement is spoken language and that is stated in the posting.
• Answer length. It correlates with confidence, not competence.

If a vendor's model uses any of these as features, ask them to remove it or walk away.

What good AI interview questions look like

Bad: "Tell me about a time you handled conflict."

Better: "Describe a decision you made in the last year that turned out wrong. What signal did you miss, and what would you check earlier next time?"

The improvement is not the wording. It is that the second question has a rubric: does the candidate name a specific decision, identify a concrete missed signal, and propose a checkable change. Three binary criteria, scored the same way for everyone.

Build every question this way: a prompt, three to five observable criteria, and an anchor example of a weak, adequate and strong answer.

Candidate rights you should honour anyway

Whatever your jurisdiction requires, do these four things:

1. 01Tell candidates before the interview that AI is involved and what it evaluates.
2. 02Offer a human alternative on request, without penalty.
3. 03Give a reason for rejection that references the rubric, not a score.
4. 04Delete recordings on request and on a fixed schedule by default.

These are cheap. Retrofitting them after a complaint is not.

Running a pilot

Take twenty candidates you already interviewed the old way. Run them through the AI interview blind. Compare rankings. If the orderings disagree wildly, find out which one was right by looking at who you hired and how they performed. Do not deploy a system whose disagreements with your humans you cannot explain.

More detail: AI interviews explained and AI hiring compliance in 2026. Candidates preparing for one should read the free candidate interview guide.

Octively runs structured AI interviews as part of an end-to-end hiring service and hands you the transcript and rubric with every shortlist. Join the waitlist.