Building an Effective First Round Technical Interview

cdm-ium
15 min readOct 9, 2020
https://xkcd.com/1293/

It is no secret that interviewing for software engineering jobs is not pleasant. Software engineering is a young and complex field. The best practices around hiring candidates are still evolving and rarely talked about. Even worse many companies have poor interviewer onboarding processes and people are thrown into interviews as interviewers without being given taught how to interview. In my next few posts I want to give a glimpse into the framework I use for building an interview kit. I will start by talking about the first round technical screen.

Getting started

In order to figure out what to do in an interview, you need to know what you are interviewing for. Someone you hire is going to work in the team you have and the more interviews can mirror the day-to-day work experience, the better.

In addition to simulating real working scenarios, it is also important to remember that your interview pipeline should be a funnel not a random walk. A well defined interview process will move from the general to the specific. In the beginning you want to check for a variety of criteria at a high level, and then progressively get into more detail in later stages of the interview. This prevents a couple common failure modes for interviews when candidates are disqualified either too early or too late.

In the former case, this is the interviewer who fails a candidate for not knowing obscure language features during their coding screen. Putting aside whether this is a valid screening criterium at all — I tend to say no. This is undesirable because this candidate could be an all-around great contributor, and who may have other skills which will make them an extremely valuable member of your team.

In the latter case, this is when you disqualify a candidate after sending them through multiple rounds of interviews across several days (or even weeks) for a simple behavioral question that could have been asked in the initial screen. Both these cases can be avoided by setting the expectation that the initial screen will be more broad and later interviews will focus on specific, clearly defined criteria.

To make this post a little more concrete, I am going to use a fictitious role at a fictitious company as an example.

Introducing Relatable

Let’s say you are working at a small start up, called Relatable. Relatable’s mission is to declutter dating by matching people based off of proven psychological methods. Your team is small with two person backend, frontend and mobile teams. In addition to the standard workflow of most dating apps consisting of viewing potential matches and expressing interest in them, Relatable differentiates itself with another key experience: 1-on-1 sessions between users on the platform and dating coaches who are experienced in counseling people on relationships. In addition to helping people meet each other, the platform also provides a safe place to help people talk through issues people experience in their relationships.

The role you are trying to fill is on the frontend team. The frontend team is primarily responsible for the dating coach experience. Providing the dating coaches with a UI to securely discuss confidential relationship matters with them. Relatable is a small, lean team so “frontend” is a bit of a misnomer. People on the frontend team are really full stack and will often support the backend team by building parts of what they need on their own.

Framing the Phone Screen

The phone screen will check that the candidate meets a certain minimum bar of technical skill, but it also tests that the way the person works is compatible with how your team works. I emphasize that you want to focus more on making sure the candidate meets the bar for your organization rather than trying to measure how far they exceed the bar because the former is much easier to make an objective assessment on than the later. For more on why this is a good idea, see my post about the dangers of hiring the candidate that looks the most qualified on paper.

This approach will leave you with time to cast a broader net and gather valuable information for the rest of the interview pipeline. The point of the phone screen isn’t to answer any questions definitively, but to get an idea about which questions need to be dug into for an onsite and which questions probably need less attention. A common pitfall many phone screens fall into is being too myopic. They are laser focused on testing a couple of things, but ignore other topics entirely. This is not a good idea because even a superficial discussion of those excluded topics can sometimes completely exclude a candidate. By deferring assessing those criteria to later in the interview, you may find yourself disqualifying candidates late in the interview process on the basis of information which you could have gotten after 30 seconds of asking the right questions.

No process is perfect and you can’t cover everything in a short 45min-1hr phone screen, but going into the phone screen with this attitude goes a long way.

Assuming you have a concrete job description and understanding of company culture in hand, here are some more helpful questions you may want to use on the phone screen. This list is not meant to be exhaustive, but rather serves as an example of the type of questions a good interview will help you to screen for. In addition to explicitly asking these questions, a good interview design will let you test how they actually behave and allow you to measure that behavior against these questions.

  • How does the candidate respond to new technologies? Do they prefer things they’ve done in the past simply because they are familiar or do they express curiosity over new things and apply their past experiences to help them learn more quickly?
  • If your team does a lot of pair programming, is the candidate comfortable with this? If your team prefers more of an asynchronous code review style, do they show good attention to detail with written cues in the code and leave helpful comments and descriptive variable names to make the code easier to review?
  • How does the candidate respond to APIs — both literal and metaphoric — in your example problem? Do they treat them as fixed and unchangeable or question assumptions to arrive at a holistic solution?
  • How does the candidate respond to feedback and technical criticism? Different organizations have different styles, and different aspirations for what they would like their style to be.
  • Where does the candidate see their career going? This question often takes the cliched form of “Where do you see yourself in 5 or 10 years?” I try to stay away from far off timelines and focus more on the present. For example asking “What do you want out of a new role and why? What are you looking for now and how has your experience led you to wanting to try this?”

Example: Applying this to Relatable

Let’s see how these questions factor in to things an interviewer will look for during a tech screen at Relatable:

How does the candidate respond to new technologies? Front end technologies are always changing. As a small start up, Relatable has to walk a careful balance between using the latest standards to make development as smooth as possible and relying on established, mature platforms to ensure speed of development. As an example, the team initially wanted to use Flutter for simultaneously building iOS and Android apps, but after a short spike decided that while Flutter may be the future, React Native is still much more mature and allows for higher velocity by having more established development patterns. Candidates need to be bought in to the current stack, and excited to work in it, even if they have a different vision for where the codebase could go.

How does the candidate primarily like to code? Relatable is a distributed team, working across time zones all the way from San Francisco to Tel Aviv. As such, the windows where everyone is available online is fairly limited. This means that written and asynchronous communication skills are key. Candidates need to be senior enough to unblock themselves for minor blockers, while knowing how to document possibly suboptimal solutions for other teammates when they come online.

How does the candidate respond to APIs? As mentioned earlier, even though our job is listed as a frontend posting, candidates need to be comfortable jumping into other parts of the stack. If the backend engineer is in a timezone 12 hours away, is this candidate going to be able to throw up a mock PR on the backend for the things they need to flesh out or will they waste precious days waiting to schedule a meeting that works for both parties?

How does the candidate respond to feedback and technical criticism? Relatable follows a Disagree and Commit decision making process. Significant design decisions, regardless of their primarily domain of implementation, need to be documented and shared with the entire team. The team has 72 hours to review the design and voice any concerns. It is the responsibility of the document owner to respond to the concerns of reviewers, but the document owner usually has the final say on decisions. By virtue of having final say, the document owner also has final responsibility on implementation and delivery of features related to the decision. Engineers at Relatable need to be comfortable letting teammates make decisions they feel to be suboptimal, and they must support them. A good plan execute today is better than the perfect plan executed next week.

Where does the candidate see their career going? Relatable is a small company, founded with more of a Pegasus funding model. This is important for engineers to know because it means that the company is going to be fairly budget conscious and try to keep the team as small as possible for as long as possible. Team members will also wear many hats and often step outside their area of expertise. Some people want to work on larger teams and/or have more well-defined roles. If you encounter someone who expresses goals like this, it isn’t an immediate stop to moving forward in the interview pipeline, but you need to be cautious and start expectation setting early so a potential candidate knows what they are getting into.

Basic Template for a Phone Interview

My basic template for a 45 minute phone screen is as follows:

  1. 5 minutes of introductions. Interviewers should be able to give a strong introduction to themselves and the company. Adding a personal touch and explaining how an interviewer feels that they fit in to the company makes candidates feel more comfortable. Candidates that can give a clear introduction help a hiring panel match candidate expectations to the actual role available.
  2. 5 minutes of framing the interactive coding interview. I am careful to call the code session interactive rather than pairing. Different organizations have different work flows for writing software. Making this clear up front and setting expectations at the beginning helps the candidate to understand what the interview is actually testing for and helps the candidate to understand what actual interactions with colleagues will be like. If you organization does not do a lot of pair programming, you may not want to pair with them, but to make the most of your time together the interview should be interactive. As an example, for an interactive, non-pairing session, an interviewer may answer a candidate’s questions about syntax or in-built functions, but not volunteer information.
  3. 25 minutes of interactive coding. Try to run this session like the person you are interviewing is your coworker. If you see them making a syntax mistake, just tell them. Do not waste time on trick questions or gotchas. There are many great tools out there to facilitate this. I like both CoderPad and CodePen. CoderPad is the more polished platform for phone interviews in general, whereas CodePen is especially nice for working on frontend code.
  4. 10 minutes of time for candidate-led discussion. If the candidate doesn’t have many questions, this can be a good time to circle back to the types of questions mentioned above to get a better sense of how a candidate would fit in your organization.

This can be extended to a full hour session depending on what works for your team. I recommend giving more time to the non-coding parts of the interview as these are the harder to evaluate and more common reasons for someone not being a good fit. I try not to nitpick on the coding session. As long as I am reasonably certain that a candidate knows what is going on with the problem, and can work through the first milestone, I save deeper technical probing to later interviews.

Example: A Relatable Interview

  1. My interviewer introduction: “As a young professional, and a man, I spent my 20s focusing more on my career than relationships because that was what society and my parents told me to do. It wasn’t until my late 20s that I realized that there would be no end to pursuing my career and I needed to make time for a more balanced life that put the same level of attention to quality that I put into my work into how I treated the people around me, especially romantically. What brought me to Relatable was how well their mission aligned with my own personal goals. It got me excited to help people to do the same self-work. I am the original engineer and I built the first iterations of the whole platform. I am also naturally very independent and so I like the small team which gives me freedom to own the things I work on completely and talk directly with the founders over the direction of the business.”
  2. Framing the problem: “We are a small, remote team so we often don’t have time to work synchronously on problems together. Instead most collaboration is carried out through code reviews and design documents, and to a lesser extent, slack discussions. For this interview that means a few important things: You need to ask for help. If you get stuck on anything (even a simple syntax question) just ask. Variable and function naming, code readability, and highlighting idiosyncrasies to other teammates matters a lot. The next person who looks at your code won’t have you next to them to ask for answers in real time. We are moving fast right now and while quality is important, we also have deadlines to meet. In this interview you will write working code with objective success criteria. The first priority is meeting these objective user-oriented criteria. The second priority is clearly documenting any complex areas of the code. Self-documenting code is ideal, but also add comments and link to other documents as needed.
  3. The coding problem: We will look at a full sample problem in the next section!
  4. Discussion: Give the candidate free reign here. If the candidate needs help getting started you can circle back to their introduction or any questions you may have based on how the coding portion went. Relatable will be extra concerned about how independent candidates are and how well they are able to take initiative during the interview, asking questions, but proactively proposing solutions and hypotheses.

An Example Problem: A Form Library

Relatable gathers a lot of feedback from all users on the platform — both dating coaches and users. As a result we built a simple form library which allows us to declaratively define a survey through configuration. Before writing our own library, we did a bit of research and found a few libraries that did things similar to what we wanted to do. We seriously considered using SurveyJS, and in fact used it for our MVP, but later decided that the survey experience was part of our core intellectual property and decided to invest in building our own library which was easily configurable from the back end and built to integrate closely with React hooks. In this exercise we will build a simplified version of a form library that supports three different question types:

  • A free text question
  • A multiple choice question
  • A yes/no question

Refer to this code pen for details: https://codepen.io/cdm-ium/pen/ExyYvoO?editors=0011

To go along with the CodePen, here are the sorts of questions I would ask in such an interview:

  1. The example has sample question types for a free text question and a radiogroup/true/false question. Add another question to the form for a multi select question, e.g. “Which of these things do you like? (Select all that apply.)
  2. What is limiting in this approach? What happens when you want to drive this form content from the backend/change it dynamically? Once the candidate answers that question the follow up is to actually implement a solution which addresses this limitation.
  3. What other considerations would be involved in a full production application? (E.g. internationalization, data fetching, component libraries/styling, etc).

For a tech screen, my bar for technical skills is that they can make it through the first question. This is the question I will calibrate for 10–15 minutes time with someone on my team. The second and third questions are a buffer in case I meet an amazingly quick candidate, or follow ups I can return to in later interviews if we decide we want to move the candidate forward.

About the Coding Problem

When writing a coding problem, it is important to resist the temptation to make something too complex. Before inflicting a coding problem on a candidate, make sure you try it with some people on your team. It is usually better to error on the side of less complexity rather than more. Seeing people in your own organization take your coding problem is often a humbling experience for how complex even a “simple” problem really is. Problems should be simplified versions of actual problems solved on the job and should have clear success criteria, ideally fully fledged tests, and at the very least clear, objective success criteria — e.g. implementing a specific user story.

As an example, rather than asking someone to “build an app” which is too open ended, you might ask them to go into an existing app and add a feature while keeping overall style consistent with the rest of the app.

In terms of timing, assuming you are leaving about 25–30 min for the exercise, someone on your team should be able to work through the exercise in 10–15 min. This allows for overhead in having discussion during the screen, which is what you want.

When to Pass on a Candidate and on Ending Interviews Early

I am in the camp that if you have started an interview with a candidate you should not end it early. Even if the interview isn’t going well, that is a consequence of vetting done at previous stages of the interview, not the fault of the candidate, and so it is worth trying to give the candidate a good experience. One of the core assumptions I take going into every interview is that both sides of every interview have something meaningful to share with the other. Even if a candidate is not a good fit for a given role, they still have valuable experiences and insights waiting to be shared. Even if a problem reveals a serious gap in skill, a well-crafted interview can teach a candidate a new way of thinking about things and leave the candidate with a positive impression of the company.

The first tech screen is the gate you use to decide if a candidate is a fit for further rounds of interviewing. Here are a few of the key questions and considerations I make before deciding whether to move a candidate forward:

  • Don’t be afraid to say no. If there is a concrete reason why you don’t think a candidate is a good fit — struggling with the coding, wanting a completely different role than you can offer — do not pass them along because you feel bad rejecting them or second guess your own opinion. If you are having trouble sorting out your own feelings, talk it through with a team leader.
  • Don’t be afraid to say yes. Not everyone is charismatic at every moment of every day. If a candidate does well, but doesn’t amaze you, you have to trust that passing the criteria set out for your screen is good enough. Meeting someone for the first time is always awkward, and first impressions can be misleading. Disregard the thought line of “they did everything fine, but they didn’t feel good.” Trust your criteria and go off of that.
  • They can’t finish the coding in time. Assuming you have properly tested the amount of time it takes to do your exercise and you have a baseline for difficulty, you shouldn’t bend the rules for one candidate just because “it feels right.”

I will also pass on a candidate if they demonstrate any red flags, for example:

  • They *only* like to work on Technology X which you don’t use, and they actively criticize the technologies you do use.
  • They are unable to respond constructively to feedback or unable to provide feedback constructively.
  • There are inconsistencies between things listed on their resume and what they are able to demonstrate in the interview, etc.

Wrapping Up

Interviewing is just as hard for the interviewer as it is for the candidate. Within engineering there is often a culture that underestimates the effort required to craft a quality interview. My goal in writing this post is to offer some structure for coming up with a first-round technical screen and highlight some of the pain points.

To recap:

  • Remember your interview pipeline should be a funnel, not a collection of random tests.
  • Put thought with your team into what sorts of behavioral questions are valuable to discuss with candidates.
  • For coding problems, error on the side of asking simpler questions with pre-prepared follow up questions if the candidate writes code more quickly than you expect.
  • Test your problems on people on your team before trying them with candidates.

Stay tuned for the next post in this series about managing on-sites!

--

--

cdm-ium

Musings on companies, movements and organizations and how individuals fit themselves into these things.