Identifying True Talent and Fending off Frauds in the AI Age.
I’ve been working for over a decade in the information technology space. In that time, I’ve lead multiple teams. While my background is technology and not HR, that means I’m often asked to weigh in on the hiring process. As a trusted and technically adept team member, I’m asked to weigh in and evaluate candidates and evaluate whether their skills are a match for the role.
What I’ve found in my time doing this is something many recruiting professionals have echoed when I talk to them: It’s difficult to gauge a candidate’s true skillset in the limited time companies can devote to vetting them. With the advent of AI Agents specifically designed to replicate industry expertise, that is only getting more difficult! I want to take a little time to talk about the problem that this poses companies today.
Why is this so hard? And what can we do to Fix it?
I am going to approach this blog post by listing some common problems I see and identify things I have tried to implement in order to alleviate them.
The HR Skill Gap
Often times, the people who are evaluating resumes and booking interviews are not skilled in technical subjects. Nor should they be! Recruiting is a vital skill in and of itself, and skilled recruiters don’t necessarily need to know a Join from a Window Function in order to be good at recruiting for technical roles. Some are, but I don’t think it’s necessary at all for recruiters to be experts in cutting-edge technology to be effective.
The issue stands, however, that if you’re hiring for a position for which the person you’re recruiting has a deeper knowledge than you, it becomes difficult to sift through bluster, humbug, and buzzword salad. Prospects can sound confident and savvy while having no practical experience whatsoever. To make matters worse, sometimes excellent candidates don’t give the best verbal interviews.
Solution: Collaborate early and often.
In my current role as a senior consultant, when I know recruiting is going to be happening on my team, I often offer to look at the job requirements with the hiring manager and help refine them. When I do, I often find them relieved to have someone to bounce ideas off of. Someone who may be a great recruiter may be infinitely versed in the cutting-edge of technology and still be insulated from what goes on day-to-day in the data roles at their organization.
As someone who has been on both sides of the table, I help refine the requirements. Every inaccurate or superfluous demand filters out otherwise qualified candidates, so I often start conversations about what needs to be in the spec vs. what is nice to have, and which nice-to-haves will actually make an impact in the position.
Once a candidate has actually made it to the interviewing stage, it is important to remember to solicit feedback from team members who know what the technical workload is like on a day-to-day basis. I often see companies use team interviews as a culture fit check, or a preview for current employees to see their future coworker. While this can be a great morale builder, it is a missed opportunity. For small- to mid- sized companies who may not have a robust technical vetting practice, the team interview can be a great forum for digging in to the types of questions that hiring managers might not feel comfortable handling.
I find it useful to give feedback on the resumes themselves to recruiters when I have the time, even before I have the interview. I will list things that attract me about the candidates, as well as potential weak spots I see in their resume or places where it isn’t clear whether they fit the requirements for the position.
Solution: Building Technical Evaluation Libraries
For organizations who have more time to devote to the problem, or for the ones for which failing at vetting candidates is particularly painful, a good step might be to build out resources for recruiters that will help them weed out candidates who do not meet the minimum standards needed for success in a given role.
This includes tests, sample questions, and other tools that recruiters can use to establish a rough baseline for a candidate’s skill level in a given area. Again, it is important to make sure that the areas tested are actually critically relevant for the role so that otherwise qualified candidates are not weeded out.
Problem: Illusory Expertise
In the age of AI, it is becoming more common for candidates to supplement their skills with language models or agents. This can be a great value add for companies, but in the hiring stage, it really muddies the waters. As an employer, you want to be sure that the folks you hire actually have the skills you need, and aren’t just using AI to fake it.
That can be helped by a technical question database, but feeling out whether or not an effective interviewer is leaning on AI to puff up their skills is more complicated than asking difficult questions. AI can be good at digging up obscure documentation, but it can still be tripped up by some pretty simple strategies.
Solution: Active Listening
This is of course a great practice in any interviewing setting, but active listening can really help reveal whether someone truly understands what they are saying. AI is good at spouting buzzwords. Even before AI, a type of interviewee was someone who would present well and sound good, but upon reflection it was difficult to tell if anything meaningful had been said. Active listening is a crucial skill in interviewing today because it allows us to follow up on claims and ask for clarification.
For example, if an interviewee is talking about the impact that a project that they worked on had, an active listener will pick up on details that they can ask about, further drilling down on their answer in either detail or shifting the conversation into another aspect of their project. Perhaps the interviewer might ask what language they wrote something in, and then ask what the most difficult part of getting it to work was. Or they could ask whether their solution was widely adopted and to whom they presented their work. One colleague asked a candidate to describe the chair they sat in to do the work. Often, asking for more detail will reveal whether or not the story that the interviewee is presenting makes sense, or takes place in a cohesive world.
Again, fabricated stories are by no means a new phenomenon, they are just made more common with more candidates using AI assistance in their interviews. Finding and pulling at plot holes in stories remains a good way to tell whether what a candidate is presenting is based in actual experience or not.
Solution: Narrative-based Interviewing
One perennial problem with interviewing is that it’s not much like the job. In interviews, we ask discrete questions, and get direct answers. In our day to day jobs, however, we’re given incomplete information and given a wide scope of possible actions to take. AI is great at the former: it can produce excellent responses when it is given a lot of parameters about the expected answer. It’s not as great at synthesizing a plan when it’s given less to go on. An experienced analyst, for example will be able to quickly identify missing pieces of information and describe next steps naturally and simply. AI tends to mimic technical writing since that is easiest to ingest, which often leads to monologues even when presented with scant information.
I’ve found success in presenting narrative scenarios, trying to distill a day’s worth of analysis work into a few questions. Before the interviews begin, I will create a challenge that is typical to the type of roadblocks encountered on the team. I’ll think of the key stakeholders and subject matter experts and what information they might be able to provide and create a reference page for myself.
At the time of the interviews, I will explain that I’d like to walk through a pretend scenario that they might encounter at work, and that I would like to role-play what their first few steps to approaching the problem would be. Then I give them the bare minimum as a starting point and ask them what they would do. At that point, I wait and dole out appropriate information based on the actions they take. I find that this is a great way to test skills that normally get passed over in an interview like critical thinking and self-determination. Often, people with great technical skills are used to being in an environment where they are placed directly in front of the problem that needs solved and given all the information to solve it up front. In small teams, it’s important to hire people with the ability to gather the information themselves and work with others who know more to synthesize answers to the challenges they’re facing.
Conclusion
Tech hiring can be hard, and AI tools are making it easier for green candidates to seem well-seasoned, and while having AI skills to augment their capabilities is great once candidates are on board, it’s vital that companies and organizations have a realistic evaluation of their actual skills and experience at the time of hiring. I’ve shared some of the tricks and tips that I use in my role, but if you’d rather collaborate with someone with experience reach out and I’ll be excited to figure out a way to work together.
Leave a Reply