When I reflect on this scenario, it’s clear where I went wrong – I didn’t construct an interview that evaluated for technical problem framing, and this skillset was a requirement for them to succeed in their role. Once I realized this, I started experimenting with ways to bake this into my interview process. Fortunately, the thing I found most effective required only a a minor adjustment.
Here’s what I do differently.
For at least one technical interview, I embed the technical task in a real-world business scenario where it’s required to fully understand the added context in order to adequately solve the problem.
In addition to evaluating technical ability, this adjusted interview also evaluates the candidate’s ability to infer from requirements what the actual intention of the project is, and then ensure that this intention is achieved when designing the technical solution.
Next I’ll walk through an example interview that does not evaluate technical problem framing, and discuss what a strong solution looks like. Then I’ll show that same interview but with the technical problem framing adjustment, and show how it modifies what’s considered a strong solution.
You can find the original dataset I use for this interview here. You can also find the interview prompt configured as a Kaggle notebook here.
Here’s the interview prompt without any technical problem framing assessment.
###############################################
# Interview WITHOUT technical problem framing #
# as part of the assessment. #
################################################ We have provided you with a dataset
# consisting of patient health information
# related to cardiac arrest (heart-attacks).
# Each record represents a patient that
# visited the Emergency Room (ER) because they
# were experiencing chest pains. Each column
# corresponds to a measurement that was taken
# at the time they arrived at the ER, including
# the type of chest pain they were
# experiencing. The dataset also contains a
# binary column that indicates whether or not
# the patient ended up having a heart-attack
# within 48 hours of their ER visit.
import pandas as pd
df = pd.read_csv(f"{filepath}/heart.csv")
display(df.head(5)
# Your task is to construct a model that
# predicts whether a candidate will have a
# heart-attack based on the provided inputs.
def predict_heart_attack(row):
"""
Accepts one row of heart-attack dataset.
Returns 0 or 1 as the prediction.
"""
# TODO
pass
I used to conduct this interview in a live setting, where it’s convenient to have a small and clean dataset to work with. The dataset is small (303 rows and 13 inputs) and relatively clean, so candidates with any amount of ML experience can build a classifier without much difficulty.
Assessment
Weaker candidates are easy to identify since they typically struggle to build even a basic model within the allotted time, let alone a good one. The more nuanced task as an interviewer is to identify the “good” candidates from the “great” candidates. In addition to demonstrating the ability to build a working classifier in a short time frame, stronger candidates typically differentiate themselves by (1) taking an iterative approach – they get something working quickly and then improve on it, and (2) by making deliberate decisions. For instance, when I ask them why they chose a specific performance measure to assess their model’s performance, they have a specific answer. Weaker or less experienced candidates will give answers, but without any real justification for it.
Here’s that same interview question but with an embedded business scenario, so it includes technical problem framing as part of what’s being assessed.
###############################################
# Interview WITH technical problem framing as #
# part of the assessment. #
################################################ An Emergency Room (ER) is receiving an
# overwhelming number of patients experiencing
# chest pains, which are a symptom of heart
# attacks. Patients who are showing other
# symptoms of heart attacks should be
# prioritized (fast-tracked) upon entering the
# ER waiting room in order to mitigate the
# effects of the heart-attack, or avoid it
# altogether.
#
# On average, the ER is equipped to fast-track
# 20% of patients who are experiencing chest
# pain, allowing them to skip the patient
# queue. Currently, the ER's policy is to
# fast-track any patients who are
# experiencing Type 2 chest pain (atypical
# angina). This corresponds to a value of
# `df['cp'] == 1` in the dataset. The ER staff
# thinks that their existing policy is
# suboptimal, and is requesting that you
# perform an analysis on this patient data in
# order to develop a policy that better
# prioritizes high-risk patients.
#
# We have provided you with a dataset
# consisting of patient health information
# related to heart-attacks. Each record
# represents a patient that visited the ER
# because they were experiencing chest pains.
# Each column corresponds to a measurement that
# was taken at the time they arrived at the ER,
# including the type of chest pain they were
# experiencing. The dataset also contains a
# binary column that indicates whether or not
# the patient ended up having a heart-attack
# within 48 hours of their ER visit.
import pandas as pd
df = pd.read_csv(f"{filepath}/heart.csv")
display(df.head(5)
# Your task is to use the dataset to construct
# a fast-track policy that is better than the
# ER's current policy.
def fast_track(row):
"""
Accepts one row of heart-attack dataset.
Returns 0 or 1 as the decision to
fast-track.
"""
# TODO
pass
Notice that the technical aspects of the problem remain unchanged – the exact same dataset is used and the solution signature is the same. But there is now additional information that changes the profile of an ideal solution.
New problem statement
The added business context introduces two new pieces of information that the candidate needs to understand prior to starting on their solution. The first is that there is a constraint that only 20% of patients can be fast-tracked. This corresponds to 60.6 patients, or 61 if we round up:
.20 * len(df) # Outputs 60.6Therefore, the maximum number of patients we can “save” by fast-tracking is 61, since the ER cannot fast-track more than that.
The second new piece of information that the ER context provide is that the ER has a baseline strategy that needs to be outperformed in order for the new policy to be considered. This baseline strategy results in correctly predicting 41 heart attacks:
# The ER baseline policiy is to fast-track
# any patient with Type 2 chest pain,
# which corresponds to
# `df['cp'] == 1`. So the ER baseline
# strategy is to return 1 when
# df['cp'] == 1, and 0 otherwise.
(
df.groupby(['cp'])[['had_heart_attack']]
.agg(['mean', 'count'])
)
.82 * 50 # outputs 41Combining the added constraint (61 total fast-tracks) with the goal of beating the baseline (41 correct predictions), we can formulate the new objective as: find a classifier that has a recall@k (with k=61) greater than 41.
Weak candidates
Candidates who are weak at technical problem framing will gloss over these two pieces of information and jump immediately into solution mode. This typically results in one of two suboptimal solutions: one with high precision, but with recall equal to or lower than 41, or one with high recall but the precision is so bad that the first 61 patients fast-tracked won’t yield more than 41 heart attacks. As the interviewer, I’ll give hints to steer the candidate if I see them heading in the wrong direction. Some candidates will pick up on my hints and adjust course correctly, while others will still struggle to identify the right problem to solve.
Strong candidates
Candidates who are strong at technical problem framing will approach the problem differently. Instead of jumping into solution mode right from the start, they take time up front to read through the prompt thoroughly, often multiple times, in order to make sure they understand the context.
Next, they do something that is strongly correlated with success, and something I pay close attention to:
The best candidates write out their approach before starting on it, and then ask me (the interviewer) if it sounds reasonable.
When I observe this, it’s music to my ears. Why? Because this is exactly what I want them to do if they were to join my team. I want someone who can articulate their plan in advance before diving into it, and also has the awareness to run it by me before they start on it. Although this takes longer up front, it reduces the need to double back and change approaches in the middle of the interview, so it ensures that the remaining time they have is well spent.
Candidates that are able to articulate the correct problem are typically able to solve the challenge as well. This shouldn’t be surprising since it’s not very difficult to beat the baseline. For instance, even the following simple rule-based solution beats the baseline:
def fast_track(row):
"""
A very simple solution that still beats the
baseline.
"""
# "cp" is the column for chest pain.
if row['cp'] == 2 and row['sex'] == 0:
return 1
elif row['cp'] == 1 and row['sex'] == 0:
return 1
else:
return 0# Check the performance
df['pred'] = df.apply(
lambda row: fast_track(row),
axis=1)
top_k_preds = df.sort_values('pred').tail(61)
recall_at_k = len(
top_k_preds
.query('had_heart_attack == 1')
.query(pred == 1))
print(f"Recall@61 = {recall_at_k}")
# Outputs Recall@61 = 50
But bonus points are definitely given if the candidate can state the problem clearly and solve it to its maximum extent (perfect recall at k=61).
The major benefit of interviewing for technical problem framing is the people who pass, and therefore get hired, are able to operate with more independence. Because they’re able to internalize the objectives they are tasked with improving, they can reduce the amount of overhead work needed from their manager, as well as from Product Owners. This is critical towards scaling the impact of a technical team, especially at smaller organizations where there is little PM support and where managers are expected to be IC’s and so they have limited bandwidth to oversee lots of projects.
We’ve been able to keep the Data team at Fi, for instance, very small and nimble, and much of this is because we’ve only hired individuals with a strong technical problem framing ability. We’re currently a team of just four individuals (soon to be five), yet we serve all Data-related needs for a business of 100+ and own all of the ETL processes, Data Warehouse design and maintenance, Tableau reporting, deep dive and root cause analyses, machine learning and predictive modeling, and more recently R&D for new feature development. And the domains we cover are literally every aspect of the business — Finance, Marketing, Customer Experience, Engineering, Hardware, Firmware, Operations, and Product. The way that we’re able to take on so much work, and cover so many domains, is because everyone on the team is skilled at taking a loosely defined problem and mapping it to a technical problem statement.
Stay tuned for a future post where I talk through how to improve your own ability to frame technical problems, as well as how to improve your team’s ability to do this if you’re a manager.