Module 9AI systems and evaluationFor AI roles

Evaluation: proving it works

How to show that an AI system works for a specific customer: golden sets and sampling, model judges and their failure modes, human review, regression suites and drift, and evals as deliverables.

6 lessons
1 h 7 min
All in Pro
6 of 6 live
For AI roles

By the end of this module you can

Answer ‘how do you know it works?’ with a golden set and a decision
Use a model judge, and measure where it disagrees with people
Gate every change on a regression suite and watch for drift
Hand the customer evals they can run themselves

What postings and candidates describe

OpenAI publishedOpenAI’s San Francisco FDE posting says FDEs measure success partly by eval-driven feedback, and its Healthcare FDE posting lists defining evaluations and launch criteria as a responsibility. Source 1Forward Deployed Engineer (FDE) - SFPublisherOpenAI CareersSource typecompany job postingSource 2Forward Deployed Engineer (FDE), Healthcare - SFPublisherOpenAI (Ashby)Source typecompany job posting
Anthropic publishedAnthropic’s FDE posting lists production LLM experience, including evaluation frameworks, as a qualification. Source 3Forward Deployed Engineer (London)PublisherAnthropic (Greenhouse)Source typecompany job posting
Snorkel AI one candidate reportOne candidate reported, in April 2026, being told of a Snorkel AI FDE screen in which they would write evaluation code over a file of model outputs; the interview was later canceled. Source 4Got a Snorkel AI Forward Deployed Engineer interview next week, anyone been through this? Format is unusual (post by u/itskabeer)PublisherReddit r/csMajorsSource typecandidate report on RedditSource 5Got a Snorkel AI Forward Deployed Engineer interview next week, anyone been through this? Format is unusual (comment by u/itskabeer)PublisherReddit r/csMajorsSource typecandidate report on Reddit

No employer publishes a pass rate or a round weight.

The lessons

6 lessons, 1 h 7 min of reading. 6 of 6 live today.

9.1How do you know it works?12 minProLive

Answer ‘how do you know it works?’ with a named evaluation set, metrics tied to the customer’s decision and a plan for monitoring after launch.

What it covers

What the customer is really asking
Where evaluation shows up: postings, rounds and take-homes
The shape of a strong answer
Reading your misses: error analysis
Offline, online and human signals
Three follow-ups to rehearse: the data, the judge, the monitor
What a weak answer sounds like

Key takeaways

The postings we cite, from an AI lab and from enterprise software companies, ask for evaluation experience; the round and take-home reports we cite don’t say how an answer is scored, so the shape here is ours.
Read your misses before you report a score: one category each, counted by segment, and each category names a fix.
A complete answer leads with a one-line verdict, confirms the customer’s decision and its costly error, then gives the set, the bar agreed before the build, the baseline, the result by segment with counts, what you watch after launch, and the limits.
When the evidence doesn’t prove a bar, say so first, narrow the launch to what it does prove, and name the check that holds until the count is large enough.
Rehearse the three follow-ups, the data, the judge and the monitor, until your numbers survive them.

Practice

8 more planned.

Do this now

Answer aloud in two minutes, then take the three follow-ups without changing or inventing a number

9.2Golden sets and stratified sampling11 minProLive

Build a golden set with a customer who has no labels, stratify it by segment and risk, size it for its decision, and score a file of model outputs.

What it covers

What a golden set is for
Getting labels from experts
Stratifying by segment and risk
How many examples
Keeping it fresh
Leakage
Writing eval code over a file of model outputs

Key takeaways

A golden set exists to support a decision: a launch bar or a comparison of two versions. Size it for that decision.
Start labels from outcomes the customer already records, except for the costly class, where those records carry the old process’s mistakes. Write the labeling guideline before anyone labels, and spend experts’ time on the disagreements first.
Stratify so rare, costly cases are in the set, then weight the headline back to the production mix.
Compare two versions on the same cases, counting only the cases that changed, and size the comparison by those changed cases.
Split by account or document, never by row, and refresh the set as a new version when the product or its rules change.

Practice

Write a script that scores model outputs against a ground-truth file and groups errors by type. Pro

9 more planned.

Do this now

Say the denominator aloud, write the scorer, then give the client three bullets

9.3Model judges and their failure modes11 minProLive

Design an LLM judge with a rubric, measure its agreement with people, and test it for position, verbosity and self-preference bias.

What it covers

When a judge helps
Rubric prompts
Known biases
Measuring agreement with humans
Calibrating and monitoring the judge

Key takeaways

A judge is a model to be evaluated, not an oracle.
Test for position, verbosity and self-preference bias.
Report how often the judge catches known failures, not only how often it agrees.
Keep humans on a sample forever.

Practice

How do you make an assistant cite its sources, and how do you check the citations are real?

5 more planned.

Do this now

Answer the citations question, then write the entailment check’s rubric and how you would measure its catch rate

9.4Human review loops10 minProLive

Design a review queue that routes outputs by risk, measures the error rate on a random slice, checks reviewer agreement and feeds corrections back.

What it covers

Why humans stay in the loop
Sampling for review
Reviewer guidelines and agreement
Closing the loop
Cost of review
The whole design, in one minute
The same loop in enterprise data

Key takeaways

Route by risk to correct outputs, and keep a uniform random slice to measure the error rate.
Measure agreement between reviewers before using their labels.
Route corrections to the golden set and the backlog.
Budget review time explicitly.

Practice

7 more planned.

Do this now

Answer the wrong-merge follow-up: move the thresholds, add a random audit, and price the review hours

9.5Regression suites and drift detection12 minProLive

Set up a regression suite that gates every prompt, model and data change, and monitors that catch drift in production inputs, outcomes and rules.

What it covers

What a regression suite gates
Cases that used to work
The eval pipeline: datasets, runners, graders, gate, reports
Cost of running evals
Input drift and output drift
Alerting on quality

Key takeaways

Every fixed bug becomes a regression case.
Run a fast subset on every change and the full set before release.
Monitor inputs as well as outputs.
Alert on a quality metric, not only on errors.

Practice

9 more planned.

Do this now

Answer the harness question, then add an input-mix and an outcome monitor, each with who it pages

9.6Evals as customer deliverables and launch criteria11 minProLive

Agree launch criteria with the customer before you build, report results by segment so a VP can decide, and hand over an eval they can run.

What it covers

Agreeing criteria before you build
Acceptance thresholds
The evaluation report
Segment-level results
Who owns the eval after you leave

Key takeaways

OpenAI’s Healthcare FDE posting lists launch criteria against customer-specific acceptance thresholds, and a Scale AI posting for a senior director of forward deployed engineering asks for an evals-first approach to customer deployments.
Agree the criteria in writing, signed by the person who owns the risk, before you build.
Write the minimum sample per segment into the criteria, and what happens below it.
Lead the report with the decision, then show results by segment, with counts and real examples.
Hand over the eval with the system, including the set, the guideline, the runner and a named owner.

Practice

12 more planned.

Do this now

Before any architecture, write the claim intake launch criteria and name who signs them

Preview a Pro lesson

Lesson 9.112 minProLive

How do you know it works?

Answer ‘how do you know it works?’ with a named evaluation set, metrics tied to the customer’s decision and a plan for monitoring after launch.

What it covers:

What the customer is really asking
Where evaluation shows up: postings, rounds and take-homes
The shape of a strong answer
Reading your misses: error analysis
Offline, online and human signals
Three follow-ups to rehearse: the data, the judge, the monitor
What a weak answer sounds like

The pilot answered every question the freight forwarder’s head of compliance tried. Then she asks the question that decides whether her sales reps may stop emailing her desk: “How do you know it works?” Learn to answer it in two minutes for a system you built, and to take the three follow-ups without changing your story.

What the customer is really asking

Today every air shipment with a battery, a solvent or dry ice waits on an email to her desk, or on a quick-reference sheet that goes stale. She is asking whether reps can quote those shipments without her, and whether she could defend that to an auditor.

So start from her decision and its costly error: telling a rep a shipment can fly as ordinary cargo when the dangerous-goods rules forbid it or require a declaration, which puts an undeclared hazard on an aircraft. For air, the rules airlines apply are the IATA Dangerous Goods Regulations, a new edition every year. The opposite error, sending her desk a shipment the system could have answered, costs time; you watch that too.

In the interview, the interviewer plays her. Within your first three sentences, name her decision and the error she can’t afford, before any tool or score.

Practice

Cases

Claim intake on a model the insurer already boughtPro
Moving analytics to a new data platformPro
2 planned

Questions

Client scenarios

6 planned

Design prompts

3 planned

Coding drills

4 planned

Take-homes

3 planned

Locked behind Pro

Every lesson in Module 9 is in Pro, the first lesson included, with its preview and practice.

Get Pro

Onsite this week?

Do these first. Each is live and free.

The evidence behind this module

The plan for this module cites 16 claims: 10 from employers' own pages and postings, 5 candidate reports and 1 press report. How we source what we say.

What Pro adds to this module today

6 of 6 Pro lessons live, and the rest as they ship.
Model answers to 4 questions this module practices Pro

See Pro and pricing

Questions about Module 9

Is Module 9 free?

No. Every lesson in Module 9 is in Pro, the first lesson included.

How long is Module 9?

6 lessons, 67 minutes of reading in all, plus the practice each lesson ends with.

Is Module 9 written yet?

Yes. All 6 lessons are live.

Which kinds of FDE role is it for?

It is mainly for AI roles: AI labs and AI product companies. Candidates for enterprise software and data roles can still use it.

Sources

Forward Deployed Engineer (FDE) - SF OpenAI Careers, company job posting
Forward Deployed Engineer (FDE), Healthcare - SF OpenAI (Ashby), company job posting
Forward Deployed Engineer (London) Anthropic (Greenhouse), company job posting
Got a Snorkel AI Forward Deployed Engineer interview next week, anyone been through this? Format is unusual (post by u/itskabeer) Reddit r/csMajors, candidate report on Reddit
Got a Snorkel AI Forward Deployed Engineer interview next week, anyone been through this? Format is unusual (comment by u/itskabeer) Reddit r/csMajors, candidate report on Reddit