Skip to main content

Complete learning course

AI Engineer

From programming foundations to production AI systems

A step-by-step skill map for engineers building practical products with foundation models, RAG, agents, evaluation, and production infrastructure.

Do not merely complete the materials. Build a portfolio of production-ready artifacts: a reliable API client, evidence-first RAG, a governed agent, an evaluation gate, and a final AI system with runtime controls.

0%0/33 lessons

Progress is stored locally in your browser.

6–12 months6 modules33 lessons4 assessments

Study operating system

How to complete the course and retain a real result

1. Understand the concept

Read the core lesson material and explain in your own words what it solves, where it breaks, and which trade-offs it creates.

2. Reproduce it hands-on

Do not move on after one read. Build a minimum working artifact: code, a diagram, an eval set, a policy, or an observable workflow.

3. Break your own solution

Add at least one failure case: timeout, stale context, prompt injection, invalid schema, wrong tool call, or budget overrun.

4. Capture evidence

Save test output, a screenshot or trace, an acceptance result, or a short postmortem. The course evaluates proof that the system works, not the number of pages opened.

Course rules

Do not merely read it — prove it

  • Do not count a lesson as complete after reading alone; an artifact or evidence is required.
  • If a prerequisite is weak, go back. The course is a dependency graph, not a playlist.
  • Do not optimize cost before a quality baseline exists, and do not scale autonomy before evaluation and controls exist.
  • Finish every project or milestone with a short postmortem: what broke, what changed, and which signal you now monitor.

Module 1

Engineering foundations

The base required to keep AI products from turning into fragile demos.

OutcomeA reliable Python/API project with typing, tests, and controlled error handling.

  1. Production Python

    Core

    Typing, virtual environments, packaging, async, logging, and testing.

    Material →

  2. Typing and data contracts

    Core

    Types, dataclasses, Pydantic, and schema validation.

    Material →

  3. Async and API clients

    Core

    Concurrency, timeouts, retries, and rate limits.

    Material →

  4. Reliability patterns

    Recommended

    Idempotency, backoff, circuit breakers, and observability.

    Material →

  5. HTTP, APIs, and backend fundamentals

    Core

    REST, streaming, authentication, queues, and background jobs.

    Prerequisites: Production Python

    Material →

  6. Project: resilient AI API client

    Projectpractice + assessment

    A client with structured outputs, retries, timeouts, tracing, and tests.

    Prerequisites: Production Python, HTTP, APIs, and backend fundamentals

    Material →Practice →Assessment →

Checkpoint after module

Engineering baseline

  • ✓ API client passes tests
  • ✓ retries, timeouts, and idempotency are implemented
  • ✓ errors have controlled types and logs

Module 2

LLM engineering core

How models behave inside an applied system and how to control that behavior.

OutcomeA controlled LLM workflow with structured outputs, context management, and model routing.

  1. LLMs, tokens, context, and inference

    Core

    Transformer basics, tokenization, context windows, latency, and cost.

    Material →Resource →

  2. Prompt and context engineering

    Core

    System instructions, examples, constraints, context packing, and failure modes.

    Prerequisites: LLMs, tokens, context, and inference

    Material →

  3. Structured outputs and tool calling

    Core

    JSON Schema, validation, function contracts, and safe tool execution.

    Prerequisites: Prompt and context engineering

    Material →Resource →

  4. Model selection and routing

    Recommended

    Quality, cost, latency, fallbacks, and multi-model strategy.

    Prerequisites: LLMs, tokens, context, and inference

    Material →

Checkpoint after module

Controlled LLM workflow

  • ✓ structured output is validated against a schema
  • ✓ tool contracts have explicit boundaries
  • ✓ a model and fallback decision exists

Module 3

RAG and knowledge systems

Building systems that answer from owned data and verifiable evidence.

OutcomeProduction-ready RAG with citations, evaluation, freshness, and access control.

  1. Embeddings and semantic search

    Core

    Vector representations, similarity, and indexing.

    Material →

  2. Ingestion and chunking

    Core

    Parsing, metadata, chunk strategies, and document quality.

    Prerequisites: Embeddings and semantic search

    Material →

  3. Retrieval, hybrid search, and reranking

    Core

    Query rewriting, multi-query retrieval, filters, and rerankers.

    Prerequisites: Ingestion and chunking

    Material →

  4. Citations, freshness, ACL, and observability

    Core

    Evidence, versioned facts, permissions, traces, and evaluations.

    Prerequisites: Retrieval, hybrid search, and reranking

    Material →

  5. Project: evidence-first knowledge assistant

    Projectpractice + assessment

    An assistant with citations, ACL, regression evaluations, and a freshness policy.

    Prerequisites: Citations, freshness, ACL, and observability

    Material →Practice →Assessment →

Checkpoint after module

Evidence-first knowledge system

  • ✓ answers include citations
  • ✓ ACL and freshness are checked
  • ✓ retrieval regression cases exist

Module 4

Agents and tool ecosystems

Governed multi-step systems with tools, state, and human oversight.

OutcomeAn agent with explicit state, permission boundaries, audit logging, and evaluation.

  1. Agent loop and planning

    Core

    Goal, plan, action, observation, and stop conditions.

    Material →

  2. State and memory

    Core

    Working state, history, durable knowledge, and context limits.

    Prerequisites: Agent loop and planning

    Material →

  3. State machines, handoffs, and multi-agent systems

    Recommended

    Deterministic control flow, delegation, and coordination.

    Prerequisites: State and memory

    Material →

  4. MCP servers, clients, and authorization

    Core

    Standardized tool/data integration and secure contracts.

    Prerequisites: Agent loop and planning

    Material →

  5. Project: governed operations agent

    Projectpractice + assessment

    An agent with approvals, cost/time limits, tools, audit logging, and rollback.

    Prerequisites: State machines, handoffs, and multi-agent systems, MCP servers, clients, and authorization

    Material →Practice →Assessment →

Checkpoint after module

Governed agent

  • ✓ state and stop conditions are explicit
  • ✓ high-impact actions have an approval boundary
  • ✓ tool actions have an audit trail

Module 5

Evaluation, safety, and quality

Systematic verification of quality, safety, and regressions.

OutcomeAn evaluation pipeline that blocks unsafe or weak releases.

  1. Evaluation datasets and test cases

    Core

    Representative cases, golden sets, and production feedback.

    Material →

  2. Metrics and model graders

    Core

    Deterministic checks, LLM judges, calibration, and bias.

    Prerequisites: Evaluation datasets and test cases

    Material →

  3. Prompt injection, guardrails, and red teaming

    Core

    Threat models, tool isolation, data leakage, and adversarial tests.

    Material →

  4. Regression and release gates

    Milestonepractice + assessment

    Thresholds, baselines, CI integration, and rollback criteria.

    Prerequisites: Metrics and model graders, Prompt injection, guardrails, and red teaming

    Material →Practice →Assessment →

Checkpoint after module

Release quality gate

  • ✓ a representative eval dataset exists
  • ✓ quality thresholds are defined
  • ✓ a dangerous regression blocks release

Module 6

Production AI systems and runtime operations

Deployment, serving, observability, runtime evidence, model/provider change control, cost, reliability, and incident response.

OutcomeAn AI system with SLOs, traceable runtime truth, governed model/provider changes, known-good rollback, and verified economics.

  1. Serving, streaming, and batching

    Core

    Managed APIs, self-hosting, vLLM, queues, backpressure, quotas, and capacity limits.

    Material →

  2. Tracing, SLOs, and production monitoring

    Core

    End-to-end traces across model/retrieval/tool calls, latency, cost, quality, errors, drift, and alerts.

    Prerequisites: Serving, streaming, and batching

    Material →Resource →

  3. Runtime evidence and false-green prevention

    Core

    Capture the deployed commit/config fingerprint, authoritative postconditions, production verification, and separate truth for CI, deploy, and indexing.

    Prerequisites: Tracing, SLOs, and production monitoring

    Material →Resource →

  4. Model, prompt, retrieval, and provider change control

    Core

    A versioned release envelope for model revision, prompt, retrieval/index, tools, policy, and provider; replay, canary, holdback, and rollback as one change contract.

    Prerequisites: Runtime evidence and false-green prevention, Regression and release gates

    Material →Resource →

  5. Capacity, cost, and routing budgets

    Recommended

    Concurrency, rate limits, token/tool budgets, cost per successfully verified task, and fallback economics across models or providers.

    Prerequisites: Serving, streaming, and batching

    Material →Resource →

  6. Privacy, secrets, and governance

    Core

    PII, data policies, supply chain, sandboxing, policy engines, and least-authority tool access.

    Material →Resource →

  7. Failure injection, failover, and reconciliation

    Recommended

    Test 429/5xx, timeouts, provider outages, partial side effects, and duplicate delivery; reconcile first, retry second; verify fallback before a real incident.

    Prerequisites: Runtime evidence and false-green prevention, Model, prompt, retrieval, and provider change control

    Material →Resource →

  8. Incident response and rollback

    Recommended

    Runbooks, scoped kill switches, model/provider fallback, recovery to a known-good configuration, and postmortem → permanent regression test.

    Prerequisites: Failure injection, failover, and reconciliation

    Material →Resource →

  9. Capstone: production AI system

    Milestone

    An end-to-end system with contracts, RAG/agents, evals, a versioned release envelope, deployment gates, runtime evidence, failure drills, and verified recovery.

    Prerequisites: Incident response and rollback, Privacy, secrets, and governance, Capacity, cost, and routing budgets

    Material →Resource →

Checkpoint after module

Production readiness

  • ✓ SLOs and monitoring exist
  • ✓ rollback or kill switch is tested
  • ✓ cost and failure modes are measured

Capstone contract

Capstone: production AI system

Build one end-to-end system in which the LLM is only a component. It must accept real input, work with knowledge or tools, pass evals, provide observability, and behave predictably under failure.

What to submit

  • — architecture diagram and a short ADR covering key trade-offs
  • — repository or executable prototype with README and setup
  • — evaluation dataset plus baseline and regression results
  • — observability evidence: traces, latency, errors, and cost
  • — security and authority model for data and tool actions
  • — runbook covering failure, rollback, kill switch, and recovery

When it is ready

  • ✓ the happy path works from input to verified output
  • ✓ at least three failure scenarios are reproducible and handled predictably
  • ✓ a critical action cannot execute outside the defined authority boundary
  • ✓ the release can be stopped or rolled back without manual chaos
  • ✓ another engineer can understand the result from evidence rather than “it seems to work”

Final practice and assessment

The course ends with a practical artifact and an acceptance rubric. The result is considered complete after the acceptance criteria are met, not merely after reading the materials.

Project: resilient AI API client →Project: evidence-first knowledge assistant →Project: governed operations agent →Regression and release gates →Open roadmap →