Field Notes

Methods from practice: evals, discovery, measurement, prioritization.

• How do you write a design constraint instead of a design principle?A constraint is two clauses: the rule, and the observable moment it broke. Plus a named checker. Four tests decide whether a candidate is real.
• What breaks when a forward deployed engineering team scales?Palantir says it cannot satiate demand. Five things break between the tenth FDE and the hundredth, and the fix is fewer engineers per deployment, not more.
• Does a login wall cost a site its AI assistant referrals?Reddit fell 29% and will wall the page. Notion's top channel is an AI tool at 25.4%. The referrer is the assistant, and it sends people to the page it reads.
• How far should gross margin fall during an AI pricing migration?ServiceNow lost 27% on a gross margin of 70.5%. My per-seat to outcome migration bottomed at 58 to 62% in quarter four and was back at 71% by month 22.
• What should an AI agent's approval request contain?Arca says agents prompt people more than people prompt agents. Anthropic's agents idle 80% waiting on a human. Four fields on every ask, and a ledger column.
• How many of your scheduled agent runs have a reader?LangChain reports 70% of its agent runs are scheduled. The number next to it is the share with a named reader. In my fleet, 13 of 39 agents had none.
• Should an MCP write tool take a reason field?Yes, optional, on the history entry next to who, what, and when. Amplitude's agents filled theirs in three times in four; Jira's v1 critique called it unneeded.
• Should Anthropic buy Salesforce or keep partnering?Keep partnering. At about $275B, two thirds of Salesforce is seat revenue agents compress, and the parts worth owning already default to Claude.
• What did agent harnesses change for forward deployed engineers?The vendor sells the harness now. The FDE's deliverable moved up to connectors, skills, evals, and governance, and nothing in the tooling checks a prompt.
• Is the forward deployed engineer role a residency or a job that burns people out?Milos Mandic's State of FDE 2026 survey could not settle it from 24 people. The keys line at day 60 can: seven still hold the systems they shipped.
• How do you check an AI maturity self-assessment before writing the roadmap?Put Marco Di Santo's fourth question, who would notice within a day if it stopped, to the operator of each system, then verify it with two dated numbers.
• Who writes the eval in a forward deployed engineering deployment?The FDE writes it, before building, with the customer's experts labeling the rows. The eval is the statement of work, the handback's spine, and the exit test.
• Who should sign off when an AI agent gets a new tool?One named person per agent, on a reach line: tools, identity per tool, channels per tool, and the name that signs the next addition. Agent risk lives in reach.
• When should a company not hire forward deployed engineers?Five cases where an FDE is the wrong hire, from an opinionated product to hiring the title because everyone is, and the cheaper job to hire for instead.
• Where should the brake on AI spend go, the user or the loop?Three posts this week cap AI spend per user or key. Cost and value are independent, so brake the loop (a stop rule) and the surface (per outcome).
• What should an FDE team do when the model scores 60% on the key evals at scoping?OpenAI took a 60% eval to 90-plus with Research. If you are not a lab, 60% at scoping is a no with an eval number and a re-test date, after a harness probe.
• How should an AI product refuse a request?Two jobs per refusal: name the reason in a category the user understands, then name the nearest thing you can do. Four categories and the words to cut.
• When should an AI agent interrupt you?Off by default, a budget per session, and a count of acted-on against spoken. Dan McAteer's mod fired on 99 of 111 turns; my fleet audit found the same shape.
• What should an enterprise put in a forward deployed engineering contract?Six contract lines for buying from an FDE vendor: the customer layer, the codebase, falling hours, a named counterpart, an exit test, and the renewal price.
• Why is Salesforce buying Braze?Salesforce is paying $5.66B for Braze to give Marketing Cloud the real-time signal layer it never built. The integration is hard, and it's the point.
• Should an architecture review pick the model or the harness?The harness. Lesson 6 of Becoming FDE found two models three points apart; harness-only changes moved 13.7 the same week. Probe per config, ask the model once.
• Why did Character.AI go bankrupt with millions of users?Character.AI had 12M monthly users and lost $8 to $9M a month. It fails all three moat questions, and its data turned from asset into liability.
• How much does a forward deployed engineer make?Posted US base runs about $150K at seed to $243K at frontier labs, equity is 44 to 70 percent at the labs, and most postings never mention equity at all.
• How do you design for AI latency and long waits?Six duration bands, from instant to open-ended agent work, and the state each one requires. Pick the band by the slow case, never the median.
• Should AI agents use the same UI patterns as humans?Mostly yes. Atlassian's rule is nine times in ten. The tenth is any seat where the human carried unwritten judgment, and there the step needs an action class.
• What is the career path for a forward deployed engineer?A ladder inside the seat, defined by what the product absorbed, and three exits: into product, up to run the function, or out to found a company.
• Who should maintain an AI agent's context files?The agent can make the edits. A person has to read them. Keep the library in version control and stamp every rule a human actually said.
• How do you measure a forward deployed engineer?Four numbers, each a trend against your own last quarter: time to landing, reversal rate, generalization rate, and hours per deployment. No benchmark exists.
• How do you write a design system for AI agents?Agents read the design system every time. Give each line a failure condition and a check: a lint in the pipeline, a named reviewer, or cut it.
• How much of an AI's reasoning should you show the user?Five disclosure levels, a default per reader, and the one test that separates useful transparency from decoration that costs you attention.
• Who owns the customer after a forward deployed engineer leaves?Customer success owns the account from day 60, and a named person at the customer holds the keys. The handoff is one page with four lines and a recall rule.
• What is the one-problem-a-week leadership loop?Name the single biggest constraint, give it one owner, decide what Friday proves, repeat. The loop Andreessen attributes to Musk, and why it survives AI.
• Does a kill switch stop a prompt-injected AI agent?Only if something trips it. My expense agent approved about $2,400 while following every rule. Add a tripwire on the input side: instruction or data.
• What agent mistakes does purchase protection not cover?Statements an agent makes in your name. Stripe covers the purchase. A false auto-reply has no price and no reversal, so it needs a check, not insurance.
• What does undo mean when an agent acted on your behalf overnight?Undo splits into three problems for autonomous actions: the reversal, the notification, and the window. Four action classes, and the one that never runs alone.
• Who should own an OpenAI dot inside a company?A named person per dot, one org-wide stop rule a user cannot switch off, and a designer on the approval surface. From a fleet where 13 of 39 agents died.
• What stop rule should a self-healing AI agent have?Three fields, written before the loop runs: maximum repair attempts, a spend ceiling per incident, and the name it escalates to. Salesforce chose two and $3.
• Who owns the promise between a forward deployed engineer and sales?Sales owns the number, the FDE owns the scope. A one-page, three-column handshake (ships, builds, no) attached before signature. No quota for the FDE.
• What unit should product use to measure AI token cost?Per task, per engineer, and per company token bills each go blind somewhere. Product's unit is cost per outcome, on a five-number line per AI workflow.
• What can a forward deployed engineer change in the codebase?Three layers. The customer layer is the FDE's, same day. The harness changes by pull request, with a price. The core never forks. Plus four things owed back.
• What does a forward deployed engineer hand back to product?A one-page packet at three moments, and a written decision from a named owner within two weeks. It is what keeps an FDE team from becoming a services firm.
• How should a product show that it is uncertain?Pick the uncertainty pattern by what a wrong answer costs and whether the user can check it. When both are bad, silence beats a confidence badge.
• Where should I start reading falkster.com?A reading order for falkster.com by seat: CPO, PM, forward deployed engineer, designer, engineer, founder. What each category is for and what to open first.
• Why is there a shortage of forward deployed engineers?A recruiter study counts 17,000 US FDEs and 2,000 who deliver ROI. The gap is what the product lets an FDE do on site, not the talent pool.
• Why does an AI-first engineering org feel slow even though it ships more?The viral voxium post about Claude Code doing all the work at a big company describes a moved bottleneck. Here is where it moved and what to fund.
• How do you detect a silently failing AI agent?My monitor called 43 of 45 feeds healthy. Four had been dead for 72 to 612 days, all returning HTTP 200 and valid XML.
• What is Teresa Torres's three-step approach to AI evals?Error analysis, choose the eval type, run the loop. What the Product Talk guide says, what to steal first, and the step that turns an eval into a gate.
• Do forward deployed engineers scale, or is it just consulting?Forward deployment costs like services and has to earn like software. When the math works, and the signal to stop sending engineers.
• Does zero data retention mean the vendor keeps nothing from your AI agents?Salesforce promised at Dreamforce 26 that your data does not go in the models. True for the model layer. Here is what that promise does not cover.
• How is a forward deployed engineer different from a sales engineer?One line separates them: a sales engineer's job ends when the customer signs and an FDE's job starts there. Confusing the two is behind most stalled AI pilots.
• What is a forward deployed engineer?The role Palantir invented and OpenAI, Anthropic, and Sierra hire for. What it is, what it is not, and why it is a Product Builder in disguise.
• When can you actually enforce a rule on an AI agent?Spotify could gate file reads over 350 lines and not boilerplate writes. The three-question test that separates rules you can enforce from documentation.
• What is Marty Cagan's fresh definition of the product role?Cagan maps Benedict Evans's three product skills onto problem discovery, value risk, and viability risk. What it says, and what the mapping leaves out.
• Can you retrofit context for AI agents?No. Context is the only agent layer captured at write time. Control planes, routing, and governance can be added later. The reasoning behind a decision can't.
• Why did Miro sell for only 2.3x ARR?Miro is profitable with ~$600M ARR and sold at 2.3x. Compression explains the range, not the position. The mechanism is legibility.
• What did Salesforce's Q2 FY27 earnings say about agentic AI?Adjusted EPS matched consensus. GAAP jumped on an Anthropic gain. And Salesforce shipped its CRM inside Claude. The three numbers that matter.
• What are bottom-up evals, and why can't an LLM write them?Top-down evals restate the brief, so a model drafts them well. Bottom-up evals come from reading real outputs. Here is the conversion rule.
• Why do customers buy your product but not renew it?The reason a buyer approves you and the reason users keep coming back are different things. Most teams instrument only the first one.
• How do you design a screen that is generated at runtime?Specify the range, not the screen: best case, worst acceptable, and what the product refuses to do. Everything between the last two ships without review.
• How do you write a design rubric that scores work without you?Extract the rubric from thirty graded outputs, keep only dimensions with a named cost, score binary with severity, then calibrate on five outputs.
• What do you do in the thirty seconds after your AI is confidently wrong?Six steps in order: detect, admit in place, contain, correct visibly, name the prevention, adjust the confidence posture. Written before you need it.
• Does the IKEA vs Salesforce story prove AI automation fails?No. The viral story is quoted as proof AI fails, but both halves are stories of AI working, and the comparison misreads the tech, the 47%, and the sourcing.
• What are malleable loops in software?Dave Killeen's term for improvement moving through the software itself: the product self-checks, files scrubbed defect reports agent-to-agent, and ships.
• What is outcome-based pricing for AI agents and how does it work?Pay per resolved case, escalations free, tokens irrelevant. How Sierra's outcome pricing works mechanically, why it beats usage pricing, and where it breaks.
• Why are SaaS revenue multiples collapsing in the AI era?Airtable, Chegg, Stack Overflow, and Smartsheet repriced for two reasons. How to tell an AI repricing from a rate repricing, and which one you face.
• Why is Sierra AI growing so fast?Sierra hit $100M ARR in seven quarters and ~$200M by mid-2026. The mechanism is six operating choices, led by outcome pricing that ties revenue to landing.
• Why did Airtable sell for only 2.7x ARR?Airtable's ~$480M ARR grows over 20%, yet it sold at 2.7x revenue. AI didn't take the revenue, it took the option. Here is the mechanism and the fix.
• Are prompt logs the new switch interview?For what customers want and where your product fails, prompt logs beat the call: their own words, the exact moment, everyone. Interviews win on why.
• Can you trust your product intuition?Less than you think. Intuition is pattern-matching on a world that may no longer exist, and AI rotates the patterns faster than any gut can recalibrate.
• Does jobs-to-be-done still work in the AI era?JTBD assumes the underlying job is stable, and AI absorbs jobs faster than roadmaps react. It still names the job, it stopped saying how long it holds.
• What should you measure weekly versus quarterly for an AI product?AI features iterate ten times a week but outcomes take weeks to attribute. Measure direction weekly with leading indicators, outcomes on a slow cadence.
• How does a PM ship their first pull request with AI tools?The four-level PM PR ladder, PLANNING.md files in git, and the two non-negotiables that earn engineering trust before your first PR lands.
• How do you build a PM second brain from meeting recordings?PMs forget 90% of meeting context within a week. A four-step pipeline turns every recording into a searchable knowledge base that compounds over time.
• How do you do discovery when your customer is an AI agent?Agents don't answer interviews. Six methods replace the playbook: agent telemetry, failure-mode interviews with operators, capability audits, and more.
• How do you run a 15-minute sprint retro that improves things?Flip the retro ratio. AI generates a Sprint Health Report before the meeting from git, Jira, and Slack, so 15 minutes goes to decisions, not data gathering.
• How do you run your first product trio?A product trio is PM, designer, and tech lead deciding as a unit. Two hours weekly: 50 minutes discovery together, then decide with committed ownership.
• How do you run a weekly review that keeps you shipping?A three-phase weekly review in 30 minutes instead of 3 hours: gather the brief with AI, triage with five decision questions, communicate a standup opener.
• How do you train product judgment deliberately?Judgment trains like a sport: reps plus a scorecard. Use a decision log, written confidence scored quarterly, premortems, and one-way versus two-way doors.
• How do you write for machines, not just executives?Half your readers are agents now. Swap prose PRDs for eval-set-plus-brief, version prompts like code, and score every artifact with the cold-read test.
• How do you write OKRs that don't suck?Most OKRs are disguised task lists. Use the behavior change test: if the key result were hit, how would customer, user, or business behavior actually change?
• How does a decision log train product judgment?A decision log separates decision quality from outcome quality, records confidence as a number, and calibrates judgment against what happened.
• How do you avoid survivorship bias in product management?You only hear from the users who stayed. Talk to churned users, track non-events, run feature post-mortems, and put churn drivers in prioritization.
• How do you build a working prototype in 60 minutes?The exact five-phase workflow: frame the problem, describe to Claude Code, iterate UX, add real data, deploy and share. A testable app in one hour.
• How do you build a signal map in your first 30 days as a PM?Skip the coffee chats. In your first 30 days, map the seven places truth enters the building and audit whether each gets captured, synthesized, and routed.
• How do you measure the cost of being wrong for an AI feature?Speed is free now. Triage decisions by reversibility, not size. Ship cheap-to-undo calls instantly and slow down on the ones you cannot reverse.
• How do you run a customer interview that actually works?Show up with full signal context, run the 30-minute prototype interview format, and let AI synthesize the transcript in minutes. Depth over volume.
• How do you run continuous discovery?Agents ingest every call, ticket, and NPS response, extract signals, and surface a ranked opportunity brief every Monday. Then you prototype same day.
• How do you test a product assumption in a week?Map assumptions Monday, build a prototype that tests the riskiest ones, test with five customers, decide Friday. The prototype is the experiment.
• What is downside exposure and how do you score a feature on it?Downside exposure asks how fast a silent 20% quality drop would cost you customers. Score on three axes: revenue, cost of a wrong answer, reversibility.
• What are direction metrics?Leading indicators measured on the cadence of the work itself. Seven signals that predict AI outcomes 4-8 weeks ahead, run on a two-layer system.
• What can no product framework teach you?Frameworks are training wheels. They teach the moves, but not judgment under ambiguity, which only comes from making real decisions and being wrong at scale.
• What does it mean to ship with observability?No feature leaves staging without the traces, metrics, and evals that tell you whether it works, before your first customer hits it. A seven-item contract.
• What does 'the eval is the spec' mean?The eval set replaces the PRD for AI features. 30 to 200 real input/output pairs define what good looks like, scored daily. The eval is the contract.
• What is an opportunity solution tree?An OST connects a business outcome to customer problems, candidate solutions, and experiments. Teresa Torres created it. AI populates it in minutes now.
• What is mob prototyping?One day a week, your PM, designer, and engineer build a working prototype together in one room. 24 person-hours instead of 41. Better output than solo.
• What is the impact loop?A four-beat operating rhythm that replaces sprints: Sense, Build, Measure, Amplify. It optimizes for responsiveness, not predictability. Same-day to eight days.
• When does vibe coding get more expensive than agentic engineering?Vibe coding starts cheap, then costs 3 to 10x more per feature past the crossover point. The CapEx versus OpEx cost curve, from Google's SDLC whitepaper.
• Which mental models break when building gets cheap?Four decision frameworks quietly price building as scarce and invert when AI makes it cheap: Opportunity Cost, Bottleneck, Sunk Cost, and Local vs Global.
• Why can't users describe what they need?Users know their problems deeply but prescribe solutions limited to what they know exists. Feature requests are symptoms, not specs. Show, don't tell.
• Why should you design the tournament instead of picking winners?Cheap prototyping did not kill product judgment. It moved the decision from picking the winner to seeding the bracket: which eight of fifty get built.
• Why do cheap prototypes sometimes kill good ideas?When building is nearly free, the demo replaces the argument. A rough prototype of a strong idea reads as a weak idea, and it dies in review.
• How do you write an eval rubric for an AI feature?Grade thirty real outputs on gut feel first, extract the rubric from your own disagreements, tie every dimension to a cost, and score binary, not 1-5.
• Should you build your own data governance for AI agents?You will not out-govern Alation, Collibra, or Purview. Plug into the governance stack the enterprise already has, and design three things now.