The AI Engineering Skills Map, Read from the Systems Side

1. The Map at a Glance

I use the table’s middle column as my own shorthand. The map connects implementation with specification, verification and product judgment; continuous learning runs through all four areas.

2. Building and Deploying AI Applications

Sub-skills from Ng’s letter, briefly paraphrased:

• LLM foundations — Understand tokenisation, context limits, caching and tool calling well enough to choose models and diagnose failures.
• Grounding models with data — Choose retrieval and representations — vector indexes, knowledge graphs or structured-data layers — and keep inputs fresh.
• Building agentic systems — Choose workflow structure, tools and memory; account for adversarial inputs and data-exfiltration risks.
• Evaluation-driven development — Use error analysis to decide the next useful experiment.
• Operating in production — Track drift and failures on real traffic, maintain regression checks, and manage cost and latency.
• Machine learning foundations — Understand learning methods, data requirements and model tradeoffs.

Ng explicitly includes code-based evaluations, model judges and human review. The emphasis is choosing and checking the evaluation method that fits the task, rather than adopting one universal toolkit.

3. Software Engineering Fundamentals

Sub-skills from Ng’s letter, briefly paraphrased:

• Building full-stack applications — Understand UI rendering, APIs, authentication, asynchronous processing and persistence around the AI capability.
• Managing data — Choose storage around access patterns; understand transactions, concurrency and data integrity.
• Designing system architectures — Choose service boundaries, state placement and dependencies to fit the project’s stage.
• Making systems secure and reliable — Design tests, failure handling and security into development.
• Scaling and operating in production — Use release automation, observability and incident response; adapt infrastructure to measured load.

The useful lesson for me is that delegating implementation still requires enough understanding to recognise the decisions being delegated. A working result can conceal an unsuitable tradeoff. Knowing the syntax is less useful than knowing why the system is slow, fragile or expensive.

4. Using Coding Agents

Sub-skills from Ng’s letter, briefly paraphrased:

• Directing the workflow — Balance planning, implementation and review effort; divide work into verifiable steps.
• Enabling agent autonomy — Choose delegation boundaries and permissions; carry changed assumptions forward in context.
• Reviewing the work — Check outputs and whether the tests establish the intended result.
• Customising the agent and its environment — Maintain project instructions, tools, hooks and shared conventions; prune obsolete context.
• Coding agent foundations — Understand retrieval, context limits and the harness; recognise overengineering and missing verification.

The letter groups the workflow into planning; execution, including testing and verification; and deployment and monitoring, with feedback between them. It also cautions against equating long autonomous runs with useful output: intervention and cost still matter.

5. Shaping the Build

Sub-skills from Ng’s letter, briefly paraphrased:

• Driving the build loop — Decide whether the next step needs an experiment, user feedback or implementation.
• Making product decisions — Connect requirements with user needs and economic constraints.
• Communicating and leading — Coordinate stakeholders and explain technical possibilities and limits.
• High-agency ownership — Identify worthwhile problems and take responsibility for outcomes.

Ng describes developers taking on parts of specification and product work, while PMs and designers increasingly participate in building. I read this as an invitation to make engineering judgment visible earlier. A developer can help define the problem and the acceptance criteria while working within the team’s priorities.

6. The Same Map, Read from the Systems Side

On a single-device perception pipeline, these skills meet a fixed resource budget and physical operating conditions. Here is where I would put the engineering effort: timing guarantees, representative datasets, hardware-aware verification and early feasibility decisions.

Unpredictability has a second axis

In a real-time pipeline, timing can also be part of correctness: a prediction arriving after its consumer’s deadline may no longer be useful. Measure end-to-end latency and deadline misses under representative load, including sustained operation after warm-up. Percentiles such as p99 help describe observed behaviour, but do not establish a hard worst-case bound. The specification must say whether occasional misses are acceptable and what the system does when they occur.

Evals without a judge model

For detection or pose estimation, my starting point would be labelled data, task-specific metrics, slice-based error analysis, and a curated regression set: low-light frames, occlusions, and unusual camera angles. A model judge cannot replace geometric or numerical correctness checks. The discipline transfers directly; much of the work is choosing representative cases and maintaining trustworthy labels.

Operating within a fixed compute budget

For a single edge device with a fixed power and compute budget, adding servers does not solve the local frame deadline. The relevant choices include model size, precision, operator fusion, buffer reuse and scheduling less important work less often. Batching may improve throughput while increasing waiting time, so it needs an end-to-end check. A fleet still needs deployment, observability and capacity planning; local optimisation is one part of operating it.

Data management includes the dataset

For a perception system, data management includes dataset versions, class balance, labelling protocols, capture provenance and sensor calibration. Calibration changes must be recorded so comparisons between captures remain meaningful. Gaps here can produce a model that fails silently in a condition nobody captured. Runtime data still matters too: frame timestamps, buffer ownership and telemetry determine whether the pipeline processes the right inputs and leaves enough evidence to diagnose a failure.

Make systems changes verifiable

For a generated CUDA kernel, the evidence should include numerical comparison against an independent reference, checks for memory and synchronisation errors, and performance measurements on the target hardware. Compared with a straightforward UI change, this can require more specialised infrastructure. The scope of the claim matters: passing on one GPU does not establish correctness or performance on every supported architecture.

Hardware choices belong in the specification

In embedded and robotics work, product decisions include the compute module, sensor, thermal design and power budget. These choices constrain the software and become more expensive to revise as board design and manufacturing progress. Engineers should make the consequences visible early: which accuracy and timing targets fit the proposed hardware, what remains uncertain, and which prototype would reduce that uncertainty before committing.

The two areas I would prioritise for this kind of work are evaluation-driven development and managing data. Together, they make it possible to distinguish a promising demonstration from a system that meets its intended operating conditions.

7. Use It as an Audit, Not a Reading List

Use the four categories to review a recent project. Assess your judgment through evidence, rather than inferring strengths from your job title:

• AI applications: Can you show a versioned evaluation set and an error analysis that changed your next experiment?
• Fundamentals: Can you show a resource budget or architecture decision with alternatives and measured consequences?
• Coding agents: Can you show an accepted patch, its independent checks, and a defect those checks caught?
• Shaping the build: Can you show a requirement changed by user feedback or a hardware feasibility experiment?

Pick one weak area and one real project. Define the missing evidence before starting, then review whether it improved a decision or exposed a failure. That gives the next learning step a concrete purpose.

Conclusion

For a systems engineer, this map is useful when each skill leads to something inspectable: requirements that name the operating conditions, data that covers them, and verification that tests the complete pipeline. Coding agents can help produce and use those artifacts, but someone still has to decide whether they answer the right question.

The detector from the opening needs more than an accuracy pass. It needs evidence that predictions arrive in time after the device heats up, and a defined response when they do not. Connecting that evidence to the product requirement is the engineering work I would use this map to strengthen.