The AI Engineering Skills Map

Building and deploying AI applications

Ng's framing is that AI systems produce unpredictable outputs, and that this is what makes building them different: you can't plan the process in advance, so you build a piece of it, look at what came out, and let the result decide the next step. The stated goal is to get reliable systems out of unreliable components — which is a fair description of the job and not a pleasant one.

The follow-up letter splits this item into six: LLM foundations, grounding models with data, building agentic systems, evaluation-driven development, operating in production, and machine learning foundations. Read quickly, it's another list of true things. Read slowly, the six aren't the same kind of object at all — three are menus of technique you can look up, one is a discipline, one is operations, and one is the field that was already there.

The first three are the easy half, and they are where most of the attention goes. Wiring a retrieval step to a prompt is an afternoon. What isn't an afternoon is the decision underneath it: which representation actually fits the data and the queries. RAG over a vector index was the answer most teams reached for first, and it has quietly become the default answer to all of them, including the ones where a join over structured records would be exact and a nearest-neighbour lookup is a guess. Ng's phrasing — that grounding is now a menu — is the useful part. If your customer data lives in tables, embedding it so a model can approximately find it again is a strange thing to do on purpose.

The harder question, for all six, is whether you can say how often the system is wrong, in what way, and whether that has changed since last week.

Anyone coming from machine learning already has this. In vision work you don't ship a detector because the demo looked good. You have a held-out set that resembles deployment rather than whatever was convenient to collect, numbers broken down per class and per condition so a regression has an address, and a fixed protocol so that "better" is a claim rather than an impression. Then you look at the failures. Not the aggregate — the actual images, in batches, sorted into causes before you touch anything.

None of that changes when the output is tokens instead of boxes. The labels are harder to get and the failures are more varied, but the loop is the same. What changes is the pull away from it. Prompt-tweaking gives you the feeling of progress in about four seconds, and evals give you nothing for a day and a half, so people tweak. The habit worth keeping is the boring one: take twenty failures, group them by cause, fix the largest group. That hour is usually the highest-leverage hour available and it is almost never scheduled.

Ng puts this more strongly than I expected: in his experience the trait that most separates people who are good at building AI systems from people who aren't is whether they can run a disciplined evals and error-analysis loop, and he calls it tricky to master because the right approach changes with the project and with the project's stage. Both halves match what I've seen. Early on, twenty examples looked at by hand beat any harness, because you don't yet know what to measure. Later, the same twenty examples are worthless and you need a set large enough that a two-point move means something. Teams get stuck by keeping the early-stage answer too long, or by building the late-stage machinery before they know what it should count.

The item on his list I'd underline is evaluating the evals. A judge prompt that agrees with you on the cases you built it from and drifts everywhere else is worse than no eval, because it produces numbers. The cheap defence is a small hand-labelled set kept aside to score the scorer, and re-scoring it whenever the judge or the model behind it changes. Nobody does this until an eval has lied to them once.

Machine learning foundations, sixth on the list, reads like a legacy entry — the field the author came from, kept there out of loyalty. The detector above is why I don't think it is. Ng's claim is that every engineer he knows who is good at building with LLMs also understands ML at some depth, and the reason isn't that you'll be training models. It's that bias/variance, error analysis and thinking about data as something you engineer rather than something you have are the frameworks for working with any system whose output is uncertain. Without them, "the model is wrong sometimes" isn't a diagnosis and there's nowhere to go next. With them it decomposes into questions with different answers: is the failure in the data, the retrieval, the instructions, or the metric.

The "deploying" half of that heading matters too, and it splits into two problems that get confused. A hosted model is an unreliable network dependency with latency variance, rate limits and a per-call cost, and everything the profession knows about those applies unchanged: timeouts, retries with backoff, circuit breakers, cost budgets, contract tests against a pinned version.

None of that machinery catches the failure specific to this dependency, which is a response that arrives quickly, with a 200, and is wrong. Retries don't help because nothing failed. That needs its own layer: validation of the output against a schema or a constraint, cross-checks against a source of truth where one exists, some way for the system to abstain, and a human in the loop where a wrong answer is expensive. The common shape is a service hardened carefully against the failures that announce themselves and not at all against the ones that don't.

Two things from the operating-in-production item are worth stating plainly because they're easy to defer past the point where deferring is cheap. The first is that observability here means traces you can actually read — inputs, retrieved context, tool calls, output — because the aggregate number tells you a regression happened and nothing about where. Sampling real traffic into the eval set is the closest thing to a free lunch on this list. The second is that cost and latency are budgets, not outcomes, and they belong in the acceptance criteria the same way a latency target does in any other system. When they're breached, the reflex is to shop for a cheaper model; the larger win is usually structural — one fewer round trip, a retrieval step that doesn't run when the answer is already in context, a step that didn't need a model at all. Model swaps and distillation are the levers you reach for after the workflow itself is honest.

Software engineering fundamentals

This is the item that gets read as reassurance. See, the old skills still count. I think that reading is too comfortable.

The claim isn't that fundamentals survived. It's that the moment you use them moved. You need to know what a mutex costs, what breaks an ABI, why that allocation is sitting in the hot loop — but increasingly you need it in order to state a constraint before anything is generated, and to recognise in a few seconds that the diff in front of you is wrong. Less for writing the code, more for specifying it and rejecting it.

An agent will hand you something that compiles and passes the tests you already have. Whether it holds a latency budget, respects the ownership model, keeps an internal header out of a public one, or avoids a race that only shows up under contention is not something the compiler will mention and not something the model checks itself. The knowledge is the same as it ever was. Where it gets applied has shifted to either end:

There's a consequence here I don't have a good answer to. Fundamentals were mostly learned by writing a lot of code badly and watching it fail — by being the person who introduced the race and then spent two days finding it. If the agent writes the code, that loop is gone, and the judgement it produced is exactly what the map says you now need. The best I have is that some amount of unassisted implementation has to be treated as practice rather than output, deliberately slower than necessary. That's an unsatisfying answer and I'd rather hear a better one.

Using coding agents

Ng lists context management, planning versus execution, verification, orchestration, and avoiding catastrophic failures, with the observation that the skill is knowing how much to intervene and how much to leave the agent alone.

That last part is the whole thing and it's a calibration problem, which means it can't really be written down. Step in too early and you have an expensive autocomplete. Step in too late and you're reading four hundred lines that rest on a decision that was wrong near the top. The right moment depends on the task, the codebase and the model, and all three keep moving.

What can be written down is the model underneath. A coding agent isn't one thing — it's a harness, an inference provider, a planner, a worker, and a place where the computation happens. I went through this at length in AI Coding Workflows: From Cloud to Local. Once those parts are separate in your head, using agents stops being about prompt phrasing and starts being a set of choices you can have opinions about.

Two of those opinions I'd defend. Context is a budget rather than a container: quality falls off as the window fills, well before anything overflows, so starting fresh is a technique and not an admission of defeat. And splitting work across subagents doesn't automatically save anything. Measured, it often costs more, because every handoff re-establishes context the previous session already had. What the split buys is an interface, and the interface is what lets you put a cheaper or local model behind it later.

The catastrophic-failures item reads like boilerplate until an agent runs something destructive against something real. Sandboxing, least-privilege tool permissions, a clean branch per run, no production credentials in the loop. It's the one item on the map where learning by failing isn't an option.

Shaping the build

The argument is that as agents get better at executing a specification, the scarce work becomes deciding what the specification says: product sense, business context, talking to whoever has the problem, owning the thing through the cycle instead of receiving a ticket.

I think this is the most important item and the one with the weakest advice attached, mine included. The other three have feedback loops that answer quickly. A compiler answers immediately and without diplomacy. A sanitizer, a profiler, a test suite, an eval set — all of them tell you today whether you were right. Shaping the build answers in months, gives you a handful of samples a year, and confounds every one of them with market timing and whatever else shipped that quarter. That's close to the worst environment a skill can be learned in, which is probably why product sense is so unevenly distributed among otherwise excellent engineers.

The things that seem to shorten the loop aren't reading about product management. Watch someone use the thing for half an hour; the gap between what people say they need and what they do is where most of the information is. Write the prediction down before shipping, because otherwise you'll remember having expected whatever happened and learn nothing. Ship smaller, since iterations are the only lever on a slow loop.

There's a version of this specific to applied ML that I find sharper than the general advice. Choosing what to measure is the product decision, wearing technical clothes. A detector tuned for mAP and a detector tuned so the operator doesn't stop trusting the alerts are different systems built by the same people from the same data, and the choice between them won't come up in any modelling discussion. No agent is going to raise it either.

The part that's missing

Skills two and three point in opposite directions, and the map presents them as a flat list.

Using agents well means delegating: let the thing work, orchestrate instead of typing. Fundamentals are the deep familiarity with the code and its constraints that lets you tell good output from plausible output. The more you do the first, the less raw material accumulates for the second. That isn't a paradox to be dissolved with a slogan, it's a real trade that has to be made on purpose.

The way I've come to think about it is that verification capacity sets the ceiling. You can delegate about as much as you can check — and check in the sense of something that fails when a property is violated, not in the sense of reading a diff and nodding. Which puts the order of operations somewhere unglamorous: write the constraints down where an agent can read them and where they outlive the session, put validation behind one command that rebuilds from source and returns an exit status, make the acceptance tests something the worker can't edit. Then increase how much you hand over, until the checks stop catching what matters.

Which side of the line a piece of work falls on is usually obvious once you ask what the check would be:

On the right-hand side, an agent producing something plausible tells you nothing about whether it's right.

Closing

Ranked by how hard they are to acquire rather than how much they matter, the order is close to the reverse of how much attention each one gets. The ranking tracks the feedback loop almost exactly:

The single thing I'd act on is improving what you can check, since that raises the delegation ceiling and frees attention for the judgement that's actually scarce. That would have been reasonable advice ten years ago too.

What I keep coming back to is the junior question. If the loop that produced this judgement was writing bad code and living with it, and that loop is now optional, it isn't obvious what replaces it. Practising unassisted feels like the right shape of answer but it depends on people choosing the slower path while the fast one is right there. I don't know how that resolves.