Module 8AI systems and evaluationFor AI roles

Production AI systems

Retrieval, reranking and hybrid search, prompt versioning, guardrails and injection defenses, hosting, cost and latency, and when to fine-tune, for systems that run inside customers’ environments.

7 lessons
1 h 22 min
All in Pro
7 of 7 live
For AI roles

By the end of this module you can

Build retrieval that finds the right passage, and say how you know
Version prompts with tests and a one-step rollback
Layer defenses against prompt injection, and say what they do not stop
Set a latency and cost budget for a model-backed system

What postings and candidates describe

Anthropic publishedAnthropic’s FDE postings say its FDEs build production applications with Claude inside customer systems and deliver MCP servers, sub-agents and agent skills. Source 1Forward Deployed Engineer (New York City, NY; San Francisco, CA; Seattle, WA)PublisherAnthropic (Greenhouse job board)Source typecompany job posting
Google one candidate reportOne candidate reported, in May 2026, a Google FDE loop with a role-related knowledge round on multi-agent systems, retrieval and generative AI implementation. Source 2Rejected by Google (L3 FDE) after "killing" the domain round and solving the coding prompt. Feeling blindsided. (post by u/Superb_Pen9988)PublisherReddit r/FAANGrecruitingSource typecandidate report on Reddit
Salesforce one candidate reportOne commenter in a Reddit thread about Salesforce’s Agentforce FDE role reported, in December 2025, that their last interview for that position asked about LLM concepts such as retrieval, vector databases and Prompt Builder. Source 3Agentforce Forward Deployed Engineer – Final Technical & Behavioral Interview Round? (comment by u/Cold-Information-497)PublisherReddit r/salesforceSource typecandidate report on RedditSource 4Agentforce Forward Deployed Engineer – Final Technical & Behavioral Interview Round? (comment by u/Cold-Information-497)PublisherReddit r/salesforceSource typecandidate report on Reddit

No employer publishes a pass rate or a round weight.

The lessons

7 lessons, 1 h 22 min of reading. 7 of 7 live today.

8.1What production AI work FDEs are hired for11 minProLive

You can list the AI deliverables your target postings name and say which ones you have shipped.

What it covers

What postings name
Agents, tool servers and skills
Retrieval systems
Operating what you ship
What candidates report being asked
LLM basics, and the follow-up behind each
Your deliverables map

Key takeaways

Postings name concrete AI deliverables: agents, tool servers, retrieval systems and evaluation frameworks, at AI labs and enterprise platforms alike.
Cursor’s postings also make the FDE own production quality after launch: tracing, evals, latency and cost trade-offs, and failure modes.
Single reports name RAG, multi-agent systems, MCPs, an AI system design question and building on the vendor’s API; none says how a round is scored.
Map each named deliverable to a story or a drill.

Practice

5 more planned.

Do this now

Say which of their APIs you’ve built with, and the limit you hit

8.2Retrieval: parsing, chunking and embeddings13 minProLive

You can choose a parsing and chunking strategy for a given corpus and an embedding model for it, and say how you would test both.

What it covers

Parsing first
Chunking by structure
Metadata that retrieval needs
Choosing an embedding model
Multilingual corpora
Testing retrieval alone
Saying it in the round

Key takeaways

Bad parsing cannot be fixed downstream.
Chunk along the document’s own structure.
Evaluate retrieval separately from generation.
Benchmarks guide the shortlist; your own queries decide.

Practice

11 more planned.

Do this now

Answer the chunking question, then test your choice on labeled questions

8.3Hybrid search and reranking11 minProLive

You can combine keyword and vector retrieval, add a reranker, and show on a small labeled set whether each step helped.

What it covers

When keyword search wins
Combining result lists
Rerankers
Measuring each stage
Latency and cost of reranking
The answer, in one breath

Key takeaways

Exact identifiers and rare terms favor keyword search.
Reciprocal rank fusion is a simple, strong way to merge lists.
Rerank a short list, not the corpus.
Keep each stage only if it moves your metric.

Practice

7 more planned.

Do this now

Answer the top-twenty question with a stage-by-stage ablation

8.4Prompt versioning and rollback10 minProLive

You can manage prompts as versioned artifacts with tests, per-customer overrides and a one-step rollback.

What it covers

Prompts are code and configuration
Versioning and environments
Per-customer variation
Tests that gate a change
Rollback
Logging the version

Key takeaways

Every output should be traceable to a prompt version.
Gate prompt changes on a regression suite.
Keep customer overrides small and explicit.
Rollback must be one step.

Practice

7 more planned.

Do this now

Design prompt versioning and a one-command rollback

8.5Guardrails and prompt-injection defenses12 minProLive

You can design layered defenses for an assistant that reads untrusted content and can take actions, and say what each layer does not stop.

What it covers

Direct and indirect injection
Least privilege for tools
Confirmation for consequential actions
Input and output checks
Monitoring and response
What no guardrail guarantees

Key takeaways

Treat every retrieved document and email as untrusted input.
Limit what the model can do, not only what it can say.
Require confirmation for actions that move money or data.
Say plainly what your defenses do not stop.
Scope generated SQL by an identity the query cannot change; session settings are defaults, not controls.

Practice

A support agent reads customer emails and can issue refunds. How do you defend it against prompt injection?

12 more planned.

Do this now

Answer the prompt-injection question for the refund agent

8.6Hosting, cost and latency budgets for model-backed systems14 minProLive

You can compare hosted and self-hosted model options for a customer, split a latency target across components, estimate cost per outcome, and cut both with caching, batching and routing without losing quality.

What it covers

Hosted API, private cloud or self-hosted
What the customer’s security team will ask
From a user target to a latency budget
Tail latency and fan-out
What drives cost per task and per outcome
Caching, batching and routing between models
Pricing the trade-off, then proving quality held

Key takeaways

Budget the tail, not the median; in a model-backed system, look at the model calls first.
Express cost per business outcome, not per token.
Employers describe on-premises and private-cloud deployments, storage in customer-controlled cloud, and routing between models, in Snowflake’s case to lower cost.
Caching and batching are the cheapest wins; routing needs an evaluation set.

Practice

Trim chat history to fit a token budget without cutting a message in half. Pro

20 more planned.

Do this now

Trim chat history to a budget, and explain the cache miss

8.7Fine-tune, retrieve or prompt: a decision matrix11 minProLive

You can choose between prompting, retrieval, fine-tuning and a trained classifier for a given task and defend the choice with cost and evidence.

What it covers

What each approach changes
Knowledge versus behavior
Data you need for each
Cost and maintenance
Classic models still win some tasks
When tuning is the answer
The matrix

Key takeaways

Retrieval adds knowledge; fine-tuning changes behavior.
Try prompting and retrieval before fine-tuning.
High-volume, narrow classification may be cheaper with a small trained model.
Decide with an evaluation, not a preference.

Practice

5 more planned.

Do this now

Answer the fine-tune, retrieve or prompt question with your own split

Preview a Pro lesson

Lesson 8.111 minProLive

What production AI work FDEs are hired for

You can list the AI deliverables your target postings name and say which ones you have shipped.

What it covers:

What postings name
Agents, tool servers and skills
Retrieval systems
Operating what you ship
What candidates report being asked
LLM basics, and the follow-up behind each
Your deliverables map

The interviewer asks: “Walk me through the AI systems you’ve put in production.” The weak answer is a tool list: “LangChain, Pinecone, some agents.” The strong answer names one deliverable, where it ran, the number that proved it, and what broke. This lesson builds it from a map of your target posting.

The rest of this module builds a model-backed system in order: retrieval, versioned prompts, guardrails, hosting and cost, then whether to fine-tune. Agents get their own module.

What postings name

Copy each AI line in your target posting, word for word, onto your posting page, and write beside it what it asks you to show.

As of September 2026, the postings of Anthropic, AWS, Databricks and Salesforce name overlapping AI work: agents or their tools in each, retrieval at AWS and Databricks, eval frameworks at AWS. Source 1Forward Deployed Engineer (New York City, NY; San Francisco, CA; Seattle, WA)PublisherAnthropic (Greenhouse job board)Source typecompany job postingSource 2Sr Forward Deployed Engineer, AWS Forward Deployed Engineering - Job ID: 10517491PublisherAmazon.jobsSource typecompany job postingSource 3AI Engineer – Forward Deployed Engineering (AI FDE)PublisherDatabricks (careers site / Greenhouse)Source typecompany job postingSource 4Forward Deployed Engineer (FDE) — Agentforce OrchestrationPublisherSalesforce (Workday careers site)Source typecompany job posting The table pairs each line with the evidence it asks for:

The posting line	What to show
Anthropic FDE: “MCP servers, sub-agents, and agent skills” Source 1Forward Deployed Engineer (New York City, NY; San Francisco, CA; Seattle, WA)PublisherAnthropic (Greenhouse job board)Source typecompany job posting	One you built, and what it could touch
AWS Sr FDE: “eval frameworks, agent tooling, RAG pipelines, and prompt engineering” Source 2Sr Forward Deployed Engineer, AWS Forward Deployed Engineering - Job ID: 10517491PublisherAmazon.jobsSource typecompany job posting	A pipeline, and the metric that judged it
Databricks AI FDE: “RAG, multi-agent systems, Text2SQL, fine-tuning” Source 3AI Engineer – Forward Deployed Engineering (AI FDE)PublisherDatabricks (careers site / Greenhouse)Source typecompany job posting	Which you chose, and why
Salesforce Agentforce Orchestration FDE: MCP and agent-to-agent standards for “context sharing, task hand-offs, and tool execution” Source 4Forward Deployed Engineer (FDE) — Agentforce OrchestrationPublisherSalesforce (Workday careers site)Source typecompany job posting	A hand-off you can draw

Each line asks for something you built and a fact about it. Shipped, it is an evidence line; not yet, a study line.

This module serves both kinds of FDE role. As of September 2026, Palantir’s Forward Deployed AI Engineer posting has the hire own Gen AI strategy and implementation with customers, and asks for past experience building solutions with LLMs. Source 5Forward Deployed AI Engineer (New York, NY) - Palantir on LeverPublisherPalantir Technologies (Lever)Source typecompany job posting Customer quotes on Microsoft’s Frontier Company page credit its FDEs with co-engineering LSEG’s Workspace AI Search, combining , and frontier LLMs. Source 6Microsoft Frontier CompanyPublisherMicrosoftSource typecompany website Integration work counts: a connector to a system of record, with auth, paging and rate limits, plus the schema the model sees and a check on every call, is a tool server.

Practice

Cases

Claim intake on a model the insurer already boughtPro

Questions

Client scenarios

5 planned

Design prompts

11 planned

Coding drills

13 planned

Take-homes

2 planned

Locked behind Pro

Every lesson in Module 8 is in Pro, the first lesson included, with its preview and practice.

Get Pro

Onsite this week?

Do these first. Each is live and free.

The evidence behind this module

The plan for this module cites 24 claims: 17 from employers' own pages and postings and 7 candidate reports. How we source what we say.

What Pro adds to this module today

7 of 7 Pro lessons live, and the rest as they ship.
Model answers to 10 questions this module practices Pro

See Pro and pricing

Questions about Module 8

Is Module 8 free?

No. Every lesson in Module 8 is in Pro, the first lesson included.

How long is Module 8?

7 lessons, 82 minutes of reading in all, plus the practice each lesson ends with.

Is Module 8 written yet?

Yes. All 7 lessons are live.

Which kinds of FDE role is it for?

It is mainly for AI roles: AI labs and AI product companies. Candidates for enterprise software and data roles can still use it.

Sources

Forward Deployed Engineer (New York City, NY; San Francisco, CA; Seattle, WA) Anthropic (Greenhouse job board), company job posting
Rejected by Google (L3 FDE) after "killing" the domain round and solving the coding prompt. Feeling blindsided. (post by u/Superb_Pen9988) Reddit r/FAANGrecruiting, candidate report on Reddit
Agentforce Forward Deployed Engineer – Final Technical & Behavioral Interview Round? (comment by u/Cold-Information-497) Reddit r/salesforce, candidate report on Reddit
Agentforce Forward Deployed Engineer – Final Technical & Behavioral Interview Round? (comment by u/Cold-Information-497) Reddit r/salesforce, candidate report on Reddit