The X-Factor: Identifying Residency Candidates with the Best Attitudes for your Program's Culture
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The X-Factor: Identifying Residency Candidates with the Best
Attitudes for your Program’s Culture
APPD Annual Spring Meeting, Chicago, IL, April 4, 2014
What is the Problem with the Traditional Interview?
• Most applicants have canned answers to typical interview questions and it does not allow for
differentiation among all the students you are interviewing. Most every candidate has
memorized, in their own words, the “right” answer to all the questions.
• Questions are not specific to your training culture. The candidate may have a great personality
and a strong academic record but may not be a good fit with the competencies required for
success as a resident in your specific program.
• The same questions are not asked of every candidate.
• The interview process is subjective and is dependent on the interviewer’s perceptions of the
candidate as a whole and not on objective performance criteria. What is important to one
interviewer may be irrelevant to another.
• Standard interview questions don’t assess attitude.
Based on the findings from the book “Hiring for Attitude” New Hires do not Succeed because
they lack (which is very similar to anecdotal evidence of medical residents):
• Coachability (26%) – the ability to accept and implement feedback
• Emotional Intelligence (23%) – the ability to understand and manage one’s own emotions and
accurately assess others’ emotions
• Motivation (17%) – sufficient drive to achieve one’s full potential and excel in the job
• Temperament (15%) – attitude and personality suited to the particular job and work
environment
• Technical competence (11%) – functional or technical skills required to do the job
Attitude is a bigger issue than skills (or knowledge)
Behavioral Interview Questions
“Behavioral interviewing techniques attempt to relate a candidate’s answers to specific past experiences
and focus on projecting potential performance from past actions. By relating a candidate’s answers to
specific past experiences, you’ll develop much more reliable indicators of how the individual will most
alter their behaviors. They do, however, assume that a person’s future behavior will closely reflect past
actions.” - Paul Falcone
Falcone, P. (2008). 96 great interview questions to ask before you hire. AMACOM Div American Mgmt
Assn.
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.“What is wrong with behavioral interview questions? Yes, asking about past behavior can work, but
most behavioral questions contain a “tip off” that tells candidates how to give you the “right” answer.
The main issue with behavioral questions is that they rob you of your chance to find out if someone is a
‘problem bringer’ or a ‘problem solver.’ Let’s say you ask a candidate a pretty typical behavioral
question: “Tell me about a time when you had to adapt to a difficult situation.” This question may
sound fine, but the word “adapt” ruins it. That single word signals that you only want to hear about a
time the candidate “adapted” (instead of the hundreds of times they failed to adapt). In the case of true
high-performer candidates, these folks have plenty of examples to share that describe a time when they
successfully ‘adapted’ to a difficult situation. If you asked them about a time they “faced” a difficult
situation, they’re naturally going tell you not only about the difficult situation, but also about how they
adapted to it. For high performers, it’s practically impossible to even imagine ‘facing’ a difficult situation
without also successfully ‘adapting’ to it. But for problem bringers (low performers), the word ‘adapted’
renders this interview question ineffective. Problem bringers have faced countless difficult situations.
But it’s unlikely they’ve successfully adapted to any of them. In fact, the times they successfully
‘adapted’ probably constitute such a tiny fraction of the times they ‘faced’ difficult situations that it
wouldn’t even occur to them to search their mental database and find an instance where it happened.
And that’s something you want to know about. But when you introduce a leading interview question,
you’re not giving them the chance to disclose that information. Just remember that attitude is the key
driver of new hire success.” – Mark Murphy
http://hiring.monster.com/hr/hr-best-practices/recruiting-hiring-advice/interviewing-
candidates/behavioral-questions.aspx
Competency-Based Behavioral Interviewing (CBBI) and Why it is Better than Traditional
Interviewing Techniques:
Competency-based behavioral interviewing is a structured interview process that combines
competencies with the premise that the best predictor of future performance behavior is past behavior.
These X-factors are behaviors (skills and/or abilities) or sets of behaviors that describe the expected
performance in a particular work context. When they are appropriately developed, competencies are
the standards of success and the behaviors that are needed to support the strategic plan, vision,
mission, and goals of the training program and institution.
It is better because:
• CBBI is designed through a process to determine the competencies unique to your working
environment.
• Interview questions are planned and directly tied to the competencies for success as a resident
in your program.
• Interviewers are trained on the CBBI process.
• Rating scales are provided to minimize the subjectivity of the interviewing process.
• Interview questions focus on actual current and past behavior rather than “might do” behavior.
• CBBI makes it easier to compare candidates because they are all measured against the same
criteria.
• CBBI focuses exclusively on competencies that are job related.
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.The Process for Interviewing for Attitude:
1. Read the book Murphy, M. (2011). Hiring for attitude: A revolutionary approach to
recruiting and selecting people with both tremendous skills and superb attitude. McGraw
Hill Professional.
2. Identify your “brown shorts” – The unique attitudinal characteristics that make your program
different from all others. They are a list of the key attitudes that define your best residents, but
they also describe the characteristics of the residents who aren’t making it.
a. Survey faculty and current residents
i. What are the characteristics/traits of the residents that excel in our program?
ii. What are the characteristics/traits of the residents that perform poorly in our
program? Be as descriptive as possible. Think about specific residents and
describe what they do.
b. Focus groups with key stakeholders
i. Be sure you know exactly what people are talking about. Try to elicit the actual
behaviors.
ii. Could you tell me about a specific instance?
iii. Could you tell me how you knew they were _______?
iv. 3-3-3 exercise: Write down (in a behaviorally specific way) the attitudinal
characteristics of your 3 best and 3 worst residents over the past 3 years.
c. Modified Delphi process with program leadership, especially helpful if you are trying to
change the culture of your training program or you want to obtain consensus (If you are
going to change the culture you need to agree on what characteristics you are looking
for).
d. Whatever method you use, keep asking for more specifics until you get to the place
where two strangers could actually observe and grade someone based on the
characteristic/attitude that has been described.
3. Create your Brown Short questions: a four-step process
a. Pick one of your Brown Shorts characteristics.
b. Identify a differential situation to elicit Brown Short characteristics. These are the
moments when the differences between high and low performers are most starkly in
contrast.
c. Begin the question by asking: “Could you tell me about a time you…” and then insert
the situation you just identified. Always make sure that nothing in your question will
lead the candidate to give the “right” answer.
d. Leave the question hanging. (This last step differs from traditional behavioral
interviewing strategies).
i. Leading question: Could you tell me about a time when you had to balance
competing priorities and did so successfully?
ii. Better question: Could you tell me about a time when you had to balance
competing priorities.
iii. Behavioral questions are only helpful when they prompt a response that reveals
the truth about both weaknesses and strengths.
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.4. Create Brown Short Answer Guidelines
a. Create a list of good answers (positive signals) and a list of bad answers (warning signs)
i. Survey
ii. Focus groups
b. Create a scoring rubric
5. Train your interviewers both in the process and in evaluation of candidates to enhance
interrater reliability.
Example of the characteristic of Teamwork
Teamwork Evaluation Form
Self-absorbed, passively follows Working collaboratively,
others, limited participation in team cooperative, flexible, trustworthy,
activities or discussions, little initiative share the workload, allow others to
to interact with team members, little show what they know, willing to
acknowledgement of the contribution help, admits when they need help
of others, self-centered approach to and accepts that help, accepts
feedback from team members,
work, tries to “outshine” other team
listens and seeks understanding of
members, throws other team
other team members opinions and
members “under the bus”, doesn’t issues, supportive, tries to make
help pick up the slack, focuses solely other people on the team look good,
on getting their work done, disagrees willing to do extra work to help the
with others, wants everything his or team, recognizes when others are
her own way, does not ask for swamped, handles differences of
assistance even when overloaded opinion constructively
1 2 3 4 5 6 7 8 9 10
Comments:
Example Questions to ask the candidate:
• Could you tell me about a time when your team was overwhelmed with work?
• Could you tell me about a time when one of your team members was overwhelmed with
work?
• Could you tell me about the most challenging encounter with a team member?
• Could you tell me about an interaction with a team member that you wished you had
handled differently?
• Could you tell me about a time you received constructive feedback from a team member?
• Could you tell me about a time when a team member was under stress?
• Could you tell me about a time when you and a team member disagreed?
• Could you tell me about a time when a team member appeared to be struggling in front of a
supervising resident, fellow, patient or attending?
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.Interviewing Basics
• Know the candidate you are interviewing
o Welcome them and call candidate by name- Be warm and inviting, even if you don’t feel
like it! Using the applicant’s name when you first meet helps put the person at ease.
Simply showing you care enough to use the person’s name helps communicate that this
is a good environment in which to train.
o Utilize an opening icebreaker. Jumping in too quickly into a structured interview may
leave the applicant feeling cold.
Most interviewers look for something they have in common with the applicant
based on the individual’s application, such as school and geographic
connections, special interests and hobbies, foreign languages, research interests
and the like.
Or use this time to verbally recap the highlights of the student’s application
Look for common experiences or shared values
o Call attention to something in their application
Ask about past performance or something they have written about- they have
spent a lot of time preparing the application and want you to read it. This is the
most frequently cited complaint by the applicants about the interview process.
• If you only have a limited time – skim their CV and personal
statement
• Review the MSPE for strengths, weaknesses, rotation comments,
professionalism issues, or difficulties experienced during medical
school. Be careful with bottom line statements. Make sure these
are commiserate with comments in the MSPE.
• Research suggests that letters of recommendation have no
predictive value on performance during the first year (unless
there is something negative). Letters of recommendation are
usually always glowing. Review for comments and to identify
letter writers whom you know.
• Allow the candidate to loosen up and feel somewhat in control
o Make it feel more like a conversation and less like an interview. This will provide you
with a much greater amount of useful information
o Make your questions seem optional rather than like orders…use ”could you tell me
about” rather than “tell me about”
o Spend the majority of the interview asking the applicant questions that would
allow you to gain a greater understanding of how well he or she would perform
at our program.
• Allow time during the interview for the applicant to “sell” themselves
o Refrain from spending a significant portion of the interview soliciting questions
about the program. This is the second most cited complaint about the interview
process that the faculty interviewer spent the majority or “all” of the interview
time asking the candidate what questions he or she had about the program. It
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.is fine to answer questions spontaneously generated by the applicant, but
don’t focus on eliciting questions.
• Use the” Interview Sandwich”-
o rapport building,
o Harder questions, (non-leading, behavioral questions) *you will spend most of your
time here*
o End with a question that allows them to sell themselves
Tell me about your proudest accomplishment in medical school
What makes you stand out among your peers?
What else should I know about you that we haven’t already discussed?
• Close the interview- maintain rapport and leave the person with the clear impression that yours
is the best place to do residency
Do
• Use questions that are open to perception/interpretation
• Probe to clarify understanding (It will help me if you can describe in more detail how you
handled; think of a specific time you…and then tell me about what you did)
• Leave the question hanging (resist the urge to grammatically resolve your statement)
• Allow for silence
• Seek contrary evidence- If your interview questions are painting a one sided picture, either all
good or all bad, ask about a specific incidence in which things didn’t turn out that way
• Ask questions about past performance. The single biggest predictor of future performance is
past performance.
Don’t
• Don’t feel the need to spend time eliciting “questions about the program”. They will ask if they
really want to know.
• Reminder- Don’t ask
o 3 most common questions (strengths, weaknesses, tell me about yourself)
o Hypothetical questions (you want to know what they actually have done)
o Undifferentiating questions (which of the 7 dwarves would you be?),
o Leading questions
o Illegal questions (sex, age, marital status, family planning, religion, sexual orientation,
medical history, medical disability)
Probe for Understanding
You will sometimes need to ask probing questions when you need more specific information or more
focused information. Instances where probing questions can be useful are when the candidate:
• Is nervous
• Doesn’t understand the kind of information you want
• Only partially answers your question
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.• Answers with a hypothetical situation or what he or she has observed others do in the situation
(will need to redirect the candidate that you would like to hear about a past experience that he
or she has had)
o That was a very difficult situation with a parent that the resident encountered, but Sam
could you tell me about a specific situation in which you yourself had a difficult parent
interaction.
Key phrases you can use to probe for more specific information include:
• Tell me exactly how you dealt with…
• It will help me if you can describe in more detail how you handled…
• Could you tell me more…
• What steps or actions did you take?
• Could you expand on that some more?
• What happened after that?
• How did the other person respond?
• How did you deal with that?
• Tell me why you chose that course of action.
• How do you think other people felt?
• What was going through your mind when you…?
Allowing for Silence
Occasionally, a resident applicant will simply have trouble thinking of a specific instance of the kind of
behavior you are interested in and will sit for a moment or two in silence. Remember silence can work
for you, rather than against you. Sometimes our questions pay off only when we wait for the answers.
• Let the candidate know that silence is okay and that you don’t mind waiting while the person
tries to come up with the best answer
o That’s all right Lauren. There is no need to hurry. Take whatever time you need to think
of an example you want to tell me about.
• Allow the applicant to answer later in the interview if they are unable to come up with an
answer immediately
o We have plenty of time during the interview, Joey. Just let me know when you have an
example you would like to share.
Asking for Contrary Evidence
If ever you realize you are forming a one-sided impression of the candidate, stop and challenge yourself
to ask for contrary evidence. This is a great technique to use whether your impression is all good or all
bad. For example, if you have been consistently impressed with how the applicant has handled all the
difficult situations you have discussed, ask the student to describe a situation in which things just didn’t
work out as planned:
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.• Mark, could you describe a situation in which you had to interact with a nurse or other health
professional and it didn’t work out the way you wanted?
What to do if the Candidate Can’t Provide Specific Examples
Sometimes, because of nervousness, a candidate just will not be able to recall the kind of specific
behavior-based examples you are asking for. In this situation, you can
• Coach the student by giving an example of the depth of information you are looking for
• Refer to the student’s application and ask for detailed information about a specific item listed in
there
• Let the person talk in generalities for a brief time, then try asking for in-depth specifics once
again.
Tips for Interpreting the Interview
The grammar that people use predicts whether they are a good or bad fit.
Table of Examples of Words High and Low Performer’s Use When Answering Questions Based on the
Different Parts of Speech:
High Performers usually use Low Performers usually use
these words in their answers these words in their answers
Pronouns I, me, we (talk about themselves You, your, he , she they, it, itself
and what they did) (talk a lot more in second or third
person language)
Verb tense Past tense (when you ask a high Present tense, future tense (a low
performer about a past performer will describe what they
experience, they will actually tell are doing or what they will do –
you about that past experience) they can’t tell about past
experiences because they don’t
have them)
Voice Active voice Passive voice (if being used to
appear smarter than they really
are)
Emotions Positive emotions (happy, Negative emotions (angry, afraid,
thrilled, excited) pessimistic)
Qualifiers Adverbs (thoroughly, quickly)
negation (no, neither), waffling
(could be maybe, perhaps),
absolutes ( always, never)
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.Problem solvers vs. problem bringers
If your questions are constructed well, the answers should help you differentiate between two types of
candidates:
Problem Bringers
• Describe a problem but don’t reference how it was ultimately solved
• Tend to dump a problem or blame rather than taking responsibility
Problem Solvers
• Go on to describe how the person overcame the failure to eventually solve the
problem
• Can’t bring themselves to think of a situation as a total failure- they keep trying or at
least try to salvage some useful lesson
Putting Everything Together
• Make sure you base any judgment on all of the information presented by the applicant.
• Look for repetition of behavior or comments
• Do not jump to conclusions. Le the information gathered in the interview point you in the
direction.
• If something does not feel right, you are probably correct in your conclusion. Try not to
rationalize observed behaviors or statements.
• Minimize initial impressions you get from the written application. Look for confirming or
denying data in the interview.
Interviewing Tips from the Applicant Perspective
Do’s:
- Ask questions that show applicants that you have spent time reviewing and familiarizing
yourself with his/her application
- Refer to his/her personal statement as a significant amount of time went in to preparing this
part of the application
- Give the applicant sufficient opportunity to talk about him/herself
- Give the applicant information about yourself as well
- Think about and then express how an applicant can add to your particular program based on
his/her career interests and personal experiences
Don’ts:
- Do not open a conversation with “What questions do you have for me?”
- Do not spend more than half of the interview having the applicant ask questions
- Do not spend a significant amount of time trying to sell the program without being asked
specific questions
- Do not highlight the major weakness of your program voluntarily as this can become repetitive if
other interviewers in the department express the same weakness
- Do not tell the applicant that you have not had time to look over his/her application
Coordinators Top Ten Expectations from Faculty Interviewers
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.1. Honor time commitment to interview as a coordinator will depend on you to fill that slot.
2. Be willing to see multiple applicants on a given day as this saves coordinators time when
creating the daily interview schedule.
3. Be Punctual. If you are late the entire interview schedule will domino down.
4. Understand the complexity of the interview schedule. Coordinators spend hours customizing
an applicant’s schedule. Managing a schedule for multiple applicants can be more difficult than
scheduling the subway trains in any major city.
5. Be mindful of last minute changes. Your change could complicate the entire daily schedule.
6. Offer to find a replacement if unable to see candidate. This is extremely helpful to a
coordinator as we spend a great deal of time finding faculty to interview at a specific time.
7. Know the training program(s). Be up to date on all changes to the training program.
Coordinators send the program information so you are knowledgeable about the training
program.
8. Communicate to your admin the importance of your participation. Coordinators do not have
time to contact your admin for you.
9. Offer to escort candidate to next interview. This will give you a few more minutes to chat as
you walk, the applicant can ask last minute questions and you will save the coordinator time,
which is greatly appreciated. This is a win-win for everyone involved.
10. Focus on the applicant. Applicants know when you are having a “bad” day. During the
interview your main focus should be on the applicant not what is going on in another area.
Coordinators appreciate you very much. We are a great team!
Example Questions
1. Could you describe a difficult interaction you've had with a supervising resident?
2. Could you tell me about a time that a patient asked you a question that you were unsure of the
answer?
3. Could you tell me about the hardest stand (ethically or morally) you ever had to take as a
medical student.
4. Could you describe an interaction with a team member, nurse, or patient that you wished you
had handled differently?
5. Could you give me an example of a time when you saw something that you thought was
unprofessional?
6. Could you tell me about a situation where a supervisor's evaluation of your performance
differed from your own perception?
7. Could you give me an example of a time when you worked hard to achieve something, but
failed?
8. Could you describe a situation where you didn't pay as close attention to the details as you
should have?
9. We all have periods of downtime at work. Tell me about a downtime you had?
10. We all have to do things we really don't want to do on clinical clerkships. Could you give me an
example of a time that happened to you?
11. Could you give me an example of a situation when others knew more than you did?
12. Could you describe a time when you were not very satisfied or pleased with your performance?
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.13. Could you describe a work situation that brought out the worst in you?
14. Could you tell me about a time you received constructive feedback from an attending or
supervisory resident?
15. Could you tell me about a time you had to juggle a number of clinical duties and priorities.
16. There are times we each feel overwhelmed with a task or project. Could you tell me about a
time this happened to you?
Resources
Murphy, M. (2011). Hiring for attitude: A revolutionary approach to recruiting and selecting
people with both tremendous skills and superb attitude. McGraw Hill Professional.
Hoevemeyer, V. A. (2006). High-impact interview questions: 701 behavior-based questions to
find the right person for every job. AMACOM Div American Mgmt Assn.
Fitzwater, T. L. (2000). Behavior-Based Interviewing. Crisp Learning: Menlo Park CA.
Deems, R. S. (1994). Interviewing--more Than a Gut Feeling. American Media Pub..
Altmaier EM, Smith WL, O’Hallaran CM, Franken EA. The predictive utility of behavioral-based
interviewing compared with traditional interviewing in the selection of radiology residents.
Invest Radiol. 1992;27(5):385-389.
Easdown J, Castro PL, Shinkle EP, et al. The behavioral interview. A method to evaluate ACGME
competencies in resident selection: A pilot project. JEPM. 2005;7(1):1-10.
Greenburg DL, Durning SJ, Cohen DL, et al. Identifying medical students likely to exhibit poor
professionalism and knowledge during internship. JGIM. 2007;22(12):1711-1717.
Hamdy H, Prasad K, Anderson MB, Scherpbier A, et al. BEME systemic review. Predictive values
of measurements obtained in medical schools and future performance in medical practice. Med
Teach. 2006;28(2):103-116.
Websites
http://academicdepartments.musc.edu/hr/university/emp_corner/leaders/interviewtools/beh
avioralbased.pdf
Recruiting the right applicants by Katherine K. Lineberger, MD as part of the Learning to
Address Impairment and Fatigue to Enhance Patient Safety Curriculum (the article can be found
on pg 49 of this pdf document: http://medicine.uams.edu/files/2012/08/teachersguide2.pdf
Teri Turner, MD, MPH, MEd (tturner@bcm.edu), Mark Ward, MD, Michelle Barajaz, MD, Melodie Allison, BS, C-TAGME, Samuel
Rosenblatt, MD, Lauren Baber, MD, and Joey Spinner, MD (Baylor College of Medicine, Texas Children’s Hospital and Children’s
Hospital of San Antonio.You can also read
