Imagine a hiring manager randomly rating a candidate as a “four out of five” because they “had a great conversation.” Then, when asked what made that conversation great, exactly, the manager explains something to the effect of the candidate being articulate, seeming sharp or seeming like a perfect “culture fit,” whatever the heck that’s supposed to mean.
When it comes to interviewing, the above is only a best practice if the sole requirement for that respective req is being likable on a 20 minute Zoom call. Otherwise, it’s just a waste of everyone’s time, really.
The thing is, recruiting has spent years making interviews easier to schedule, document and summarize. We can preserve every word of a conversation, identify the speakers and generate a pithy paragraph explaining that the candidate demonstrated strong communication skills. We’ve become considerably better at preserving our first impressions, our “gut feeling,” for the sake of posterity (and complicance).
Unfortunately, we tend to be a lot less diligent about checking whether or not those first impressions actually deserved a second thought.
A recording can tell you which questions an interviewer asked; what it can’t do, at least by itself, is establish whether those questions helped predict that candidate’s future performance or productivity.
An AI summary can capture a candidate’s relative performance for a job without actually determining whether the skills that interview was designed to measure had anything to do with the job in the first place
And, obviously, a structured interview scorecard can make a first impression or gut feeling look superficially scientific, particularly if someone remembers to include a decimal and some perfunctory grading criteria along with it. Data without clear causation is, in fact, just a bunch of meaningless numbers.
The State of Interview Intelligence: Moving From Data Capture to Direct Causation
These are the primary findings from The State of Interview Intelligence, the brand new industry benchmark report I recently published on behalf of Kyle & Co.
To arrive at our findings, we surveyed hundreds of employers and job seekers, using mostly mirrored questions, to examine how organizations describe their interview processes and how candidates experience them.
Reading the responses, you’d be forgiven for wondering whether the employers and candidates had attended the same meeting, since the results seemed to suggest that the respective sides of the proverbial table, in fact, live in entirely different universes.
Employers rated their interviews’ ability to predict job performance at an average 8.3 out of 10. Meanwhile, only 51% of candidates described their recent process as structured and consistent; another 41% of respondents reported an experience that can best be described as some sort of arbitrary mixture of standardization and improvisation.
The problem seems clear enough: confidence – and perception- of an interview process can (and does) coexist with candidate experiences that are drastically different, to understate the obvious.
Still, the practical question is hard to dodge. After all the money spent capturing interview information, what are we actually learning from it?
Useful interview intelligence should help us identify questions that produce relevant evidence, interviewers who apply criteria inconsistently, and extra rounds that add time without resolving uncertainty. Eventually, it should help us examine whether our assessments bear any relationship to what happens after someone starts.
That work requires an owner, though, as well as a standardized review process and a willingness to discover that the executive everyone considers an excellent judge of character might simply have a bunch of biases that are both unconscious and undetected – the strongest of which seems to be a simple affinity bias (we like interviewing people who, fundamentally, remind us of ourselves, which seems pretty obvious to anyone with any recruiting experience).
Software and solutions exist – in abundance – to easily provide evidence that the interviews happened, and from a documentation standpoint, exactly what was said, and by whom.
The problem is content without context is, at least in the case of generating actionable insight or predictive intelligence from a job interview, pretty pointless.
But, in an age where AI interviews are becoming more or less ubiquitous, that doesn’t seem to be stopping anyone from trying. If you truly want to take interview intelligence from transcribing what was said in the past to data that can actually impact future recruiting outcomes while improving quality of hire, here are some stories worth checking out, first.
Because asking “tell me about yourself” tells you absolutely nothing – about the candidate, or your organization’s interviewing process.
Required Recruiting Reading: Trending in TA Today
- Observations on Conversations: Interviewing’s Biggest Blind Spots
One of the clearest findings in our The State of Interview Intelligence research concerns feedback: Eighty-one percent of employers said finalists always or usually receive substantive, personalized feedback.
Among candidates describing a recent final-round rejection, only 43% reported receiving feedback at all, irrespective of whether or not that feedback was useful or even valid.
The questions and populations differ, obviously, so this isn’t an exact measure of messages lost in translation, but it’s a strong sign that before employers declare their interview process a success, it’s probably worth consulting somebody who has actually experienced that process.
The same research, for what it’s worth, found that 23% of candidates received no explanation of the evaluation criteria, much less any feedback on how well they performed against it.
Apparently, the rules are confidential, at least until you’ve failed to meet them.
Why you should care: Recruiters need access to the evidence behind hiring decisions if they’re expected to communicate those decisions meaningfully. Give every interviewed candidate a named owner and an update deadline. Require interviewers to connect their assessments to job requirements.
A recruiter shouldn’t have to translate “didn’t quite click” into a credible explanation of why somebody lost out on a paycheck.
- The Résumé and the Recruiter Share a Ghostwriter.
In a Financial Times column, republished by The Irish Times, author Pilita Clark describes a recent study comparing old fashioned human assessments with those of an AI interviewing agent, after an IT opening attracted 178 applications. The agent, it found, heavily favored polished applications whose claims the proverbial humans in the loop found, well, implausible (or, in most cases, completely impossible).
Clark also discusses a working paper suggesting models can favor résumés produced by the same model. That research was awaiting peer review; the firm’s experience alone doesn’t explain why its rankings differed.
For interviewing, the danger’s familiar: somebody sounds convincing, so we stop investigating; fluency can describe relevant work without ever establishing whether or not that candidate actually did it.
Recruiting’s always had this problem; now, it’s becoming entrenched as training data, and scaling just as quickly.
Why you should care: Ask follow-up questions that clarify the candidate’s contribution, decisions and results. Then assess those answers against defined criteria. Interview intelligence should help reviewers locate and evaluate that evidence.
Otherwise, we risk producing beautifully written summaries of beautifully rehearsed answers, with the actual job becoming an increasingly minor character.
- McKinsey’s New Interviewer Doesn’t Charge By Billable Hours
In a similar study, The Wall Street Journal reported that consulting firm McKinsey is currently piloting a ten-minute AI-based experiential assessment for selected undergraduate applicants across North America.
In this automated prescreening environment, candidates must interact with an AI avatar, representing a client, while partners observe how they respond. Important to note, the candidates aren’t actually scored or assessed by AI; it’s simply designed to automate a critical element of the selection process.
That design gives AI a ton of utility that many agentic solutions lack: making candidates create a situation in which they can demonstrate skills that are directly relevant to the work.
After all, management consulting involves responding to clients and their changing needs or priorities, although presumably the McKinsey simulation is still too short to include a steering committee or a templated executive findings deck full of specious charts and superficial recommendations.
Why you should care: My takeaway is that employers should decide what evidence they need before deciding which technology to buy.
A simulation can give interviewers something concrete to assess, but they’ll still need consistent criteria and a way to check whether their judgments hold up. The pilot isn’t proof of better hiring; it’s a practical example of using AI to change the evidence an interview produces, while leaving people responsible for interpreting it.
Which is sort of what the whole AI thing is really all about in recruiting and hiring.
- Joveo’s AI Interviewer Has a Follow-Up Question.
Joveo’s recently launched AI Interviewer provides structured, on-demand interviews across the web, phone and WhatsApp, available around the clock. Candidates can interview immediately after applying or at a scheduled time, which could be useful for anyone whose availability doesn’t conveniently match a recruiter’s calendar.
According to Joveo, the system asks role-specific questions, probes responses, seeks clarification and answers candidate questions. Configurable scoring rubrics guide evaluation, while summaries, transcripts, scores and hiring recommendations flow back into the ATS. Analytics cover completion, engagement and candidate performance.
A candidate can rehearse the opening answer. Asking them to explain a specific decision, their personal contribution or an unexpected result gives reviewers more to work with than another recital of the résumé.
Why you should care: Those capabilities create an opportunity to collect more consistent evidence without making every first conversation depend on recruiter availability. Employers still need to define the rubric, review recommendations and check results.
Consistent scoring is only applicable when the underlying criteria driving those scores is consistently applied, too. Automating a bad question won’t improve it – which just might be one of AI interviewing’s greatest weaknesses.
Good thing Joveo now offers the capability to transform that into one of your greatest strengths – and biggest competitive advantages, too
- Taking On the Work Between Interviews.
Joveo’s recent announcement of their new agentic capabilities showcased a redesigned platform capable of seamlessly connecting talent attraction with sourcing, engagement, screening, scheduling, interviewing and analytics.
The company describes interview agents conducting structured first-round interviews and returning feedback to the ATS, with recruiters able to approve, override or adjust agent actions.
These newly launched agentic capabilities extend interview intelligence far beyond the actual interview, too; Joveo’s agents actually share context across the hiring cycle, connecting often disparate workflows and aligning recruiting results with individual recruiter execution against pre-determined hiring goals
The intended benefit, obviously, lies in creating less manual work for recruiters, eliminating the need to move candidates and information between disconnected systems and processses. Which is exactly the kind of solution recruiters need to stay competitive in today’s tight talent market, since an interview can go off without a hitch without actually fixing the bottleneck that leads to reqs sitting around aging while TA chases down hiring managers for some sort of feedback or direction on what to do next.
Candidate experience is aligned with one single employer and bigger brand reputation, regardless of how many vendors’ point solutions have been integrated into the process of disappointing or disillusioning them.
Why you should care: If you’re a TA leader, start by evaluating interview capabilities, ensuring those are correctly aligned with the connected workflow; it’s imperative to note whether or not the assessment reaches the right reviewer, whether the next step happens promptly and, ultimately, whether someone owns the candidate experience and associated communication and feedback.
Shared context should help people act on evidence; if it simply turns an automated recommendation into a decision nobody remembers making, then even the most advanced artificial intelligence becomes really dumb.
It’s important for people leaders to judge AI interviewers by whether candidates progress, whether reviewers get usable evidence and whether or not empirical, standardized evidence exists for optimizing hiring decisions or interview outcomes.
After all, no agent in the world can get a req filled or an offer letter signed by themselves. That’s why recruiters still exist – and likely always will, too.
In Conclusion:
The Transcript Won’t Take Responsibility. That’s up to You.
The future of interview intelligence, ultimately, depends almost entirely on what employers do after the conversation concludes and the transcription finishes auto-generating.
When considering how best to leverage interview intelligence capabilities within your organization, it’s best for employers to start with the smallest unit first; think one job family, if you’re hiring high volume, or a single business unit or location.
Agree on the capabilities the interview should assess, the evidence that would demonstrate them and what successful performance would look like after hiring; require interviewers to score those results independently before debriefing or sharing notes, and capture the reasons why those scores change, and how.
Then, review the process and policies driving your hiring decisions before looking at a potential new platform or pilot program. Dig deep for any unanswered questions, inconsistent standards, unnecessary rounds and candidates who never received an outcome.
As sufficient post-hire information becomes available, then examine whether interview assessments correspond with performance, with the obvious caveat that management, training and working conditions obviously influence results just as much as any tooling or tech.
Even in a perfect world, hiring won’t ever become totally predictable, or even entirely rational. But we can still check whether the people confidently making hiring decisions are actually competent to do so in the first place.
That’s the biggest opportunity our research uncovered: interviews contain a ton of information about candidates, but they also reveal how clearly the company understands job requirements, whether its managers agree, how consistently it treats people and whether or not all of these factors can be quantitatively connected with future performance or positively impact quality of hire.
That’s pretty much the holy grail in TA today – and the real data that real recruiters really need to really move the needle when it comes to hiring success and interviewing efficacy. For real.
After all, the only thing going with gut feeling will get you is a little indigestion and a whole lot of regret. And with the holidays fast approaching, that’s the last thing any of us need right now.
Happy hiring,
Matt Charney on behalf of Team Joveo
