The learning path
One idea at a time, in order
The series follows a fixed progression: foundations first, then the resource budget, then retrieval, agents, evals, and the production and customer skills an FDE needs. This page grows by one episode section every day.
What is an FDE, and what is AI, really?
The job, the model as a function, and the four primitives everything else builds on.
Go to Episode 1 ↓Tokens, context windows, cost & latency
The context window treated as a resource budget — money, time, and quality at once.
Go to Episode 2 ↓Retrieval & RAG in detail
Chunking, embeddings, vector search, reranking, and grounding answers in real sources — the step that decides what deserves a place in the window.
Go to Episode 3 ↓Agents & tools
Tool calling, agent loops, MCP, and human-in-the-loop — the step where a model stops only answering questions and starts doing things.
Go to Episode 4 ↓Evals & guardrails
Golden datasets, regression testing, hallucination risk, and drift — the tests that catch failures that do not look like failures.
Go to Episode 5 ↓Deployment & LLMOps
Cost, latency, caching, observability, permissions, and logging in production.
Security, privacy & data governance
Keeping enterprise AI safe, private, and auditable.
The customer side
Discovery, scoping vague requests, demos, handoff, and measuring outcomes.
Data engineering for AI
Pipelines, data quality, and messy enterprise data.
Fine-tuning vs prompting vs retrieval
When each approach wins — and when each is the wrong tool.
Portfolio & interview preparation
Building one end-to-end, evaluated agent project that proves the skills.
Evals and Guardrails: How You Know the AI Is Actually Working
AI systems fail quietly — no crash, no error, just a confident wrong answer. So this episode is about the tests that catch failures that do not look like failures: a golden dataset run on every change, hallucination measured claim by claim, drift caught by grading real traffic, and guardrails — deterministic code fences around the model.
The episode in two diagrams
The eval loop, step by step
Diagram · Test, score, fix, re-test
A small human-written test suite of real questions with expected answers — the unit tests of AI.
Run the dataset on every change — a new prompt, a new model, a new retrieval step.
Compare each answer against the expected one, and against the baseline score.
Work the worst failures first, and add every real-world failure to the dataset.
Run it all again and gate the release on holding the baseline — then the loop starts over.
Regression testing is what turns "it seemed better in the demo" into a number: if the score drops below the baseline, the change doesn't ship.
Quiet failures, and the fences against them
A normal program that breaks usually tells you — an exception, a failed test, a red log line. An AI system that breaks often just answers confidently and wrongly, and nothing in the output looks like an error. That is why hallucination is measured as groundedness, claim by claim: each claim in the answer is checked against a fact you can verify. And the suite always includes the "I don't know" refusal test — questions the system should decline, scored on whether it actually does.
Even a system that passes today can slide tomorrow. Drift comes in three kinds: the model changes underneath you, the data it reads changes, or user behaviour changes what gets asked. The catch is routine, not heroic: grade a sample of real traffic every week against the baseline and watch the score move. Around all of it sit the guardrails — deterministic code, not model judgement, that blocks prompt injection, blocks personal data, and requires a citation. Guardrails are code, so they get evals too.
Key takeaways
- AI systems usually fail quietly, so evals catch failures that do not look like failures.
- A golden dataset is a small human-written test suite of real questions with expected answers — the unit tests of AI.
- Run the golden dataset on every change and gate releases on the baseline score; add every real-world failure to the dataset.
- Measure hallucination as groundedness, claim by claim, and always include the "I don't know" refusal test.
- Drift comes in three kinds — model, data, behaviour — and is caught by grading sampled real traffic against the baseline.
- Guardrails are deterministic code fences around the model, and they get evals too.
Try it · Episode 5 homework
Pick one real task you actually do, like answering questions about a product you know well. Write ten test questions, and for each one write down the correct answer in your own words. Run them through any AI assistant, score them honestly, and count how many answers were fully grounded in facts you can verify. Keep the ones it got wrong — those ten cases are the start of your first golden dataset.
Agents and Tools: When the Model Starts Doing Things
A tool is a function described to the model. The model proposes a structured call, your code validates and runs it, and the result goes back in — think, act, observe, repeat — until the task is done or a limit you set is reached. This episode is about keeping that loop useful, and keeping it under control.
The episode in two diagrams
The loop, step by step
Diagram · Think, act, observe — until done or stopped
The task arrives, in plain language, with whatever context the system gives it.
The model decides the next move: answer now, or call a tool — and which one, with what inputs.
The model proposes a structured call. Your code validates it — known tool, well-formed inputs — and only then runs it.
The result comes back as an observation. The model reads it and thinks again — or finishes with an answer.
The loop is not open-ended: it runs under a step limit and a time limit, and when the same tool fails twice it stops and hands over to a human instead of retrying forever.
One plug is not a permission slip
MCP (Model Context Protocol) is a standard plug for exposing tools and data to a model, so one integration can be reused across clients. It standardises the connection — it does not handle trust or permissions. Authentication, who may call what, and logging are still your job, exactly as they were before the plug existed.
The other split that decides how an agent behaves in production is read vs write. Read tools only fetch — look up an order, search the documents — so they usually run automatically. Write tools change the world — issue a refund, send an email — so they need rules, limits, or human approval, they must be safe to retry, and every step must be logged so the loop can be replayed afterwards.
Key takeaways
- The model proposes; your code disposes — validate every tool request and reject unknown tools or bad inputs.
- The agent loop is think, act, observe, repeat — with step limits, time limits, and a stop-and-hand-over rule when the same tool fails twice.
- MCP standardises the connection, not trust: authentication, permissions, and logging are still your job.
- Sort tools into read vs write; require human approval for irreversible actions and make writes idempotent with a unique key.
- Return only the data a tool needs, never put secrets in prompts or tool results, and log every step so the loop can be replayed.
Try it · Episode 4 homework
Build the smallest agent you can — one read tool, one low-risk write tool, a five-step limit, and an approval pause before the write runs. Give it five real tasks, watch the log, and write down every time it surprises you.
Retrieval and RAG: How the Model Finds What It Does Not Know
The model can't look anything up — so your system looks it up first. Retrieval augmented generation (RAG) searches your own documents before the model answers, and grounds the answer in what it finds. This episode walks the whole pipeline, stage by stage.
The episode in two diagrams
The pipeline, stage by stage
Diagram · From documents to a grounded answer
Split documents into one-idea passages — by meaning, not just length.
Turn each chunk into coordinates that capture its meaning.
Find the nearest chunks to the question — hybrid keyword + vector, filtered by version and permissions. Top 20.
Read the shortlist carefully and keep only the best 5.
The model answers from those passages, with citations — or says it doesn't know.
Good retrieval is how the context budget from Episode 2 gets spent on purpose: five good chunks beat five hundred pages on cost, speed, and quality.
Key takeaways
- Chunk by meaning, not just length: respect headings, keep tables together, and overlap chunks so ideas aren't split across a boundary.
- Embeddings place similar meaning close together. They're a similarity tool, not a truth tool.
- Use hybrid search: keyword search for exact codes and names, vector search for paraphrases, plus filters for version and permissions.
- Rerank the shortlist — top 20 down to top 5 — before building the prompt.
- Grounding reduces hallucination but doesn't eliminate it: wrong retrieved documents produce confident wrong answers, so keep the index fresh and deduplicated.
Try it · Episode 3 homework
Take ten documents you actually use and ten questions people really ask. Chunk the documents two ways — once by fixed length and once by section headings — and for each question check whether the right passage lands in the top five retrieved.
What Is an FDE, and What Is AI, Really?
Start here. A Forward Deployed Engineer embeds with a customer and owns a working AI system end to end — and a language model, underneath all the jargon, is a probabilistic text function behind an API. Everything in the series builds on those two ideas.
The episode in two diagrams
How an FDE actually works
Diagram · The FDE working loop
Sit with users. Turn "we want an AI agent" into a spec you can measure: what counts as good, who approves it, what happens when it's wrong.
Build the smallest working version in days, with real users and real data — not a slide deck.
Build the eval set, add guardrails, handle permissions and audit trails, watch cost and latency.
Ship it, teach the customer team to run it, and feed what you learned back to the product team.
Then the loop starts again. The role was pioneered at Palantir and has spread to OpenAI, Anthropic, and Databricks — because a capable model alone doesn't become a working enterprise system. Someone has to close that gap.
The one mental model to keep
Diagram · A language model, in developer terms
Your prompt: instructions, conversation so far, background material
Predicts the most likely next token, one piece at a time, from patterns learned in training
Fluent, confident — and possibly wrong. It has no table of facts to check against
Wrap that call in a system you already know how to build: prompt → model → tools → data → output → evaluation. If you can draw that, you understand the architecture.
Four ways to make a model useful
Prompting, retrieval, fine-tuning, and agents aren't four technologies — they're four levers on the same model. Choosing between them is a core FDE skill, and the order matters: start with the cheapest lever.
What it is: better instructions, examples, and a structured answer format.
Reach for it: always first. It costs nothing to try and fixes more than beginners expect.
What it is: search your own documents first, put the relevant passages in the prompt, and let the model answer from them.
Reach for it: when answers must come from facts the model never trained on — runbooks, contracts, records.
What it is: training the model further on your examples so its behaviour shifts.
Reach for it: rarely, and late. Almost never the answer for simply adding new facts.
What it is: the model decides, calls a tool, reads the result, and decides again — inside limits you set.
Reach for it: when a task needs real actions. The hard part is stop conditions and keeping a human in control.
Key takeaways
- An FDE is an engineer who embeds with the customer and owns a working AI system end to end.
- A model is a probabilistic text function behind an API — prediction, not lookup.
- Your leverage comes from prompts, retrieval, tools, and evals — in that order.
- Evals are AI's version of tests. If you can't measure a change, you're guessing.
- Three beginner traps: collecting tools instead of building one small system, judging by one impressive demo, and automating a decision that should stay human.
Try it · Episode 1 homework
Take a task you do every week — triaging support tickets, summarising a meeting. Write the instructions you'd give a new colleague, paste them into a model with three real examples, and compare the output with what you'd have written. Then ask: what did it get wrong, what was missing from my instructions, and what would I need to check before trusting it? That gap is the whole job, in miniature.
Tokens, Context Windows, and the Budget You Cannot See
The idea that quietly controls almost every design decision you'll make: the context window. This episode treats it the way you'd treat any resource in a system you run — as a budget you spend on purpose, in money, time, and quality.
The episode in two diagrams
First, count in tokens
A token is the unit a model actually reads and writes — not a word, not a character, usually a chunk of a word. A short common word might be one token; a longer word, two or three. Code costs more tokens than plain English, because punctuation breaks into more pieces.
The rough rule for English text. An estimate only — other languages and code behave differently.
About a page and a half of single-spaced text.
The window size many current flagship models advertise — roughly 750,000 words, or a large codebase, in one request.
One window, everything competes for it
The context window is the model's working memory for a single request: the maximum tokens it can consider at once. Nothing gets its own space — every part of the call shares the one window, including the answer being written.
Diagram · What competes for the context window (schematic)
The model doesn't remember your last call — each request is a fresh start. If you want it to remember the conversation, your code resends the whole history, and you pay for those tokens again, every turn. A long chat is a growing bill.
One window, three budgets
You pay per token, in and out — output usually costs more than input. Stuffing the window makes every single call more expensive.
More context means a slower time to first answer, and long outputs take longer to generate. Latency is part of the budget, not a side effect.
A fuller window isn't a smarter one. Relevant facts can get lost in the middle of a huge prompt — spending context badly can make answers worse.
The episode works the arithmetic with round illustrative rates ($2 / $10 per million tokens) purely to show how the maths works — check the provider's pricing page for real numbers. The fix list is the practical part: measure real token usage, trim and summarise history, retrieve only what matters, cache what repeats, use the smallest model that passes your evals, and cap the output length.
Key takeaways
- A token is a chunk of text, roughly three quarters of a word — count real tokens when money is involved.
- The context window is working memory for one request, and everything shares it.
- Every call resends the history, so costs grow with conversation length.
- Context is not free space. Context is a budget — and a good engineer spends it on purpose.
Try it · Episode 2 homework
Measure the cost per thousand calls for something you already run (or a small test). Then try one change — summarise the history, retrieve fewer documents, or shorten the instructions — and measure again. You'll learn more from that afternoon of measuring than from any pricing page.
Review
Concept cards — tap to test yourself
Every key term from the episodes so far, in the words the show uses. Try to say the definition out loud before you reveal it.
Catch up
The story so far — and the reading version
Sixty-second recap
Five episodes in, the series has built its foundation, given the model hands — and learned how to check its work:
- The job: an FDE embeds with a customer and owns a working AI system end to end — discover, prototype, evaluate, deploy, repeat.
- The machine: a model is a probabilistic text function behind an API. Your levers, in order, are prompts, retrieval, tools, and evals.
- The budget: everything you send lives in one context window, billed per token, resent every turn. Spend it on purpose.
- The retrieval: chunk, embed, search, rerank, answer — the right five passages, found in your own documents, ground the answer.
- The doing: the model proposes tool calls, your code validates and runs them, and the loop repeats under step and time limits — reads run automatically, writes wait for approval.
- The checking: AI fails quietly, so a golden dataset runs on every change, hallucination is scored as groundedness claim by claim, drift is caught by grading real traffic weekly, and guardrails fence the model in code.
- Next: Episode 6 covers deployment and LLMOps — cost, latency, caching, observability, permissions, and logging, everything that keeps an AI system running well after launch day.
The Substack companion
Each episode also gets a written post — the same visuals, takeaways, and homework in article form — so the series can be read, searched, and shared outside the podcast feed. The audio stays on the FDE Podcasts feed; each post links to it.
- Episode 1 post — “What Is an FDE, and What Is AI, Really?” with both Episode 1 labeled diagrams.
- Episode 2 post — “Tokens, Context Windows, and the Budget You Cannot See” with both Episode 2 labeled diagrams.
- Episode 3 post — “Retrieval and RAG: How the Model Finds What It Does Not Know” with both Episode 3 labeled diagrams.
- Episode 4 post — “Agents and Tools: When the Model Starts Doing Things” with both Episode 4 labeled diagrams.
- Episode 5 post — “Evals and Guardrails: How You Know the AI Is Actually Working” with both Episode 5 labeled diagrams.