# The Interview Loop for Judgment > A portfolio of screens tells you what tools somebody owns and who reviewed > their work. It tells you almost nothing about judgment, and it never really did. > Five stages, one of which is the four-generated-options exercise, plus a > scorecard that makes debriefs short. --- ## What you are hiring for Say it out loud before you write the job description, because it changes every stage below. You are hiring for someone who can look at work that already exists, in volume, and say which of it is right and why, in a form other people and systems can execute. Production skill is now table stakes at a much lower bar than it used to be. Judgment is the scarce input. That does not mean craft is irrelevant. It means craft is necessary and no longer sufficient, and your loop should be weighted accordingly. --- ## The five stages ### 1. Screen, thirty minutes One question does most of the work: **"Tell me about a design decision you made that you later found out was wrong."** Listen for whether they know it was wrong, how they found out, and what changed afterwards. Candidates who cannot produce one are either early in their career or not paying attention, and you can tell which within a minute. Red flag: the story where they were right all along and somebody else overruled them. One of those is fine. Every story being that story is a pattern. ### 2. Portfolio, forty-five minutes, but ask differently Still look at work. Ask three questions per piece: - What were the constraints, and how would I have caught you breaking one? - What did you kill to get here, and what did that cost? - What would you do differently if you could generate forty variants in an hour? The third question is the one that separates people. Some candidates light up. Some describe the same process with more steps. Both answers are informative. ### 3. The four-generated-options exercise, sixty minutes The centerpiece. Replaces the take-home. **Setup.** Before the session, generate four credible directions for a real problem in your product. All four should be plausible. At least one should be attractive and wrong in a way that takes effort to see. Do not label them. **In the session, forty minutes:** - Ten minutes: they read the brief and the four options, ask clarifying questions. - Twenty minutes: they pick one, and they write the constraints that make it the right answer. Actually write, on a shared document, not describe out loud. - Ten minutes: they say what they would need to know to be sure, and how they would find out this week. **Twenty minutes of discussion:** you push back on their choice. Not to see if they cave, to see how they handle new information. Changing their mind in response to a good argument is a strong signal. Changing it in response to seniority is not. **What you learn:** whether they can evaluate work they did not make, write a rule instead of a preference, name what would falsify their choice, and hold a position under pressure without being rigid. **What to avoid:** letting it become a redesign exercise. If they start sketching a fifth option, note it and redirect. Producing a fifth option is a reasonable instinct and it is not what this hour is testing. ### 4. The wrong-path session, forty-five minutes Give them a feature and one failure mode. Ask them to design the response: detection, disclosure, recovery, and the actual copy. Almost nobody has practiced this, so you are watching how they think with an unfamiliar problem rather than checking a rehearsed answer. Strong candidates ask what the failure costs and who notices before designing anything. ### 5. Collaboration, forty-five minutes, with a PM and an engineer Real working session on a live problem. You are watching for whether they can hold a design position without turning it into a boundary dispute, and whether they change the argument when they get information rather than repeating it louder. --- ## The scorecard Every interviewer fills this the same day. Not yet / getting there / solid, plus one sentence of evidence. No overall number. | Dimension | Evidence you are looking for | |---|---| | **Evaluates work they did not make** | Picks correctly among options, explains the criteria, not the preference | | **Writes rules, not preferences** | Constraints have failure conditions attached | | **Names the cost** | Can say what a bad output costs, in real units | | **Owns the wrong path** | Treats failure states as first-class, unprompted | | **Changes mind on evidence, not authority** | Distinguishable in the pushback segment | | **Craft floor** | Work is competent, taste is present | **The hiring bar:** solid on the first two, and at least getting there on the rest. Somebody solid on craft and not-yet on the first two is the profile the field is about to be full of, and it is the expensive hire to get wrong. --- ## Two things to stop asking **"Walk me through your process."** Everybody has rehearsed it, and the answer is the same double-diamond narrative regardless of what they actually do. **Take-home projects.** They test unpaid production time, which is the input that just became cheap, and they select for candidates with free evenings. The four-options exercise gets you better signal in one hour, live. --- ## Calibrating the loop Keep the four options fixed for a set of candidates so you build a comparison base. Rotate the problem when the current one stops discriminating, which you will notice when everybody picks the same option for the same reasons. Debrief in fifteen minutes using the scorecard only. If the debrief runs long, the scorecard is not specific enough, not the candidate. --- ## The one thing to do this week Build the four options for one real problem in your product. Generate them, make one attractive and wrong, and run the exercise on somebody already on your team. Their reaction will tell you whether the options are good enough to use on candidates, and it costs you an hour. --- _From "Hiring for Judgment", chapter 20 of The Design Operating Model._ _falkster.com/design/hiring-for-judgment_