As engineering managers, CTOs and tech leads, we see the hype every day.
People ask me what their teams should learn to “get into AI”, how to build proper AI products, or which course will turn a developer into an AI engineer in 30 days.
Honestly, I am tired of seeing people sold a lie.
The internet is full of content promising that anyone can “master Claude”, build an agent overnight, or become an AI engineer by learning a handful of prompts and connecting an API key to a web page.
That is not how serious engineering works.
A clever prompt can create an impressive demo. It can even create a useful prototype. But a prompt, an API key and a chat interface do not automatically create a durable product, a defensible business, or an AI engineering capability.
Equally, we should not swing too far in the other direction and tell every software engineer that they need a PhD in mathematics before they can contribute.
Both views are wrong.
The real skill is learning how to build reliable systems around probabilistic models. Systems with useful data, clear user value, proper evaluation, secure boundaries, human accountability and operational discipline.
AI is not magic. But it is not simply autocomplete either.
It is software engineering, data engineering, product judgement, statistics, security, operations and organisational change, all meeting in one messy place.
If you want to build a serious AI capability in your engineering organisation, I believe there are six foundations that matter.
1. Start with the problem, not the model
The first question should not be:
“Which model should we use?”
It should be:
“What user problem, decision or workflow would genuinely improve here?”
Too many teams start with a model and go looking for a use case. They build a chatbot because chatbots are easy to demo. They build an agent because every vendor is talking about agents. They add AI to a product because investors, competitors or senior leadership expect it.
That is how you end up with expensive features nobody trusts or uses.
A serious AI product starts with a clear problem:
What job is the user trying to do?
Where are they losing time, making errors or getting stuck?
What does good look like?
What happens when the system is wrong?
Should this be automated, assisted or kept entirely human-led?
Could a better process, search experience, workflow or rules engine solve this more reliably?
AI is valuable when it improves a real outcome, not when it generates an impressive screenshot.
For example, an internal support assistant should not be judged by whether its answers sound intelligent. It should be judged by whether it helps people resolve issues more quickly, points them to authoritative sources, reduces repetitive work and safely escalates when it is uncertain.
If you cannot define the user outcome and how you will measure it, you are not ready to choose a model.
2. Learn the fundamentals at the right depth
There is a persistent argument about whether people need maths to work in AI.
The honest answer is: it depends on what kind of work you want to do.
If you are researching models, training models, fine-tuning them, designing retrieval systems, improving ranking, managing inference infrastructure or working close to the machine-learning layer, then yes, mathematics matters a great deal.
You need to understand concepts such as:
Linear algebra: vectors, matrices, embeddings and similarity.
Probability and statistics: uncertainty, distributions, sampling, confidence and evaluation.
Optimisation: how models learn, how loss functions work and why training can fail.
Neural networks and deep learning: what a model is doing when it processes data.
You do not need to become a pure mathematician. But you do need enough understanding to know what is happening beneath the abstraction.
However, not every person building an AI-enabled product needs to derive matrix multiplication by hand.
A strong application engineer can build valuable AI features by understanding the practical behaviour of models, their limitations, their failure modes and how to engineer reliable systems around them.
The important thing is intellectual honesty.
Do not pretend an LLM is magic. Do not treat it as a human brain. And do not assume it knows what is true simply because it says something confidently.
At a basic level, language models predict likely next tokens based on patterns in their training data and the context they are given. That apparently simple mechanism can produce remarkable results across language, code, reasoning-like tasks and multimodal inputs.
But it does not make the output inherently factual, safe or appropriate for your organisation.
That is your engineering job.
3. Data is usually the real product
Most organisations do not have an AI problem.
They have a data problem.
Their information is duplicated, stale, scattered across tools, inconsistently named, inaccessible to the people who need it, or full of conflicting advice. Then they expect an LLM to somehow make it clean, current and trustworthy.
It will not.
If your AI product depends on company knowledge, customer information, policies, contracts, technical documentation or operational data, then data quality becomes a core engineering concern.
Your team needs to understand:
Where the source data comes from.
Who owns it and who is accountable for keeping it current.
Whether it is accurate, complete and authoritative.
Which version is the source of truth when documents conflict.
Who is allowed to access it.
Whether the data contains personal, confidential, regulated or commercially sensitive information.
How data is retained, deleted, audited and protected.
This is where retrieval-augmented generation, usually called RAG, can be useful.
RAG allows a system to retrieve relevant information at runtime and provide that information to the model as context. Instead of relying entirely on the model’s training data, the system can search internal documentation, policies, product catalogues, support records or knowledge bases before generating an answer.
That can improve relevance, freshness and grounding.
But RAG does not “fix hallucinations”.
It can still retrieve the wrong documents. It can retrieve incomplete or stale information. It can expose content the user should not see if permissions are badly designed. The model can misunderstand the evidence, ignore it or confidently invent an answer around it.
Good RAG is not just “put documents in a vector database”.
It requires source quality, document ownership, chunking, metadata, access controls, retrieval evaluation, relevance ranking, citations, clear uncertainty handling and continuous monitoring.
And do not throw away traditional search.
Keyword search remains excellent for exact error codes, policy clauses, product identifiers, customer names, technical acronyms and rare terms. In many real systems, the right approach is hybrid retrieval: combining keyword search, semantic search, metadata filters and reranking.
Choose based on evidence and evaluation, not ideology.
4. Build Systems, Not Prompt Demos
Prompting matters.
Good system instructions, examples, context design, tool descriptions and output constraints can make a substantial difference to quality. Prompt engineering is a real part of modern AI application work.
But prompting alone is not a durable technical strategy.
A thin interface over a general-purpose model is easy to copy. It is also exposed to vendor pricing changes, rate limits, model-version changes and the simple fact that the next model may make your basic feature available to everyone.
That does not mean using an API is a bad business model.
It means your value cannot be only the API call.
Durable AI products create value through things that are harder to replicate:
Deep integration into a real user workflow.
Proprietary or carefully governed domain data.
A strong user experience.
Reliable automation and orchestration.
Trust, safety and auditability.
Better operational processes.
Distribution and customer relationships.
A clear understanding of the domain problem.
This is also where conventional software engineering still matters enormously.
You need proper APIs, authentication, authorisation, databases, queues, retries, rate limiting, caching, error handling, observability, resilient fallbacks and sensible user interfaces.
You need to design for uncertainty.
A conventional system may fail because a server is down or a query times out. An AI system can fail while appearing completely healthy. It can return fluent, plausible, well-structured nonsense.
That is a different class of failure, and it needs a different engineering mindset.
5. Evaluation, Security and Governance Are Not Optional
This is the part too many AI discussions skip.
You cannot look at five impressive examples and conclude that a system works.
A proper AI product needs evaluation before it reaches users, and it needs monitoring once it is in production.
Your team should build representative test sets from real user tasks, known edge cases and previous failures. Then measure the things that matter for the use case:
Is the answer accurate?
Is it grounded in approved sources?
Does it cite or link to the right evidence?
Does it follow the required policy?
Does it select the right tool or workflow?
Does it know when to abstain or escalate?
Is it fast enough?
Is it affordable enough to operate?
Does it behave safely with hostile or misleading input?
Every change can affect performance.
A new prompt, model version, retrieval method, chunking strategy, system instruction, tool description or policy can improve one set of examples while making another worse.
That is why prompts, configurations and model choices should be versioned and tested like production code.
Structured output is also useful. If a model needs to pass information into an existing system, do not ask for loosely formatted prose and hope it parses correctly. Use schemas and structured fields where possible.
But valid JSON is not the same as a correct decision.
A system can return perfectly valid structured data that is factually wrong, operationally unsafe or unauthorised. Your application must still validate business rules and enforce permissions independently of the model.
Security matters just as much.
Teams need to think about:
Prompt injection, including malicious content hidden inside retrieved documents.
Data leakage through prompts, logs, tools or external model providers.
Excessive permissions for agents and tools.
Broken access controls in internal knowledge systems.
Unsafe automation of production, finance, customer or security actions.
Audit logging and incident response.
Human approval for consequential actions.
Clear kill switches and rollback plans.
The moment an AI system can take action rather than merely suggest one, the risk profile changes significantly.
Treat it accordingly.
6. MLOps, LLMOps and Reliable Operations
I used to think that teams had to master massive GPU clusters and highly specialised infrastructure before they could do anything meaningful with AI.
That is not true for every organisation.
Many teams can build useful AI applications with managed model APIs and existing cloud platforms. They do not need to train a foundation model or run their own inference fleet.
But they still need operational discipline.
For teams using hosted models, the important concerns often include:
Model and prompt version management.
Evaluation pipelines.
Request tracing and observability.
Token consumption and cost control.
Latency and timeout management.
Rate limits and provider outages.
Retrieval quality and document freshness.
Safety monitoring.
Tool-call reliability.
Vendor lock-in and model portability.
Incident response when an AI feature starts failing or behaving dangerously.
This is often described as LLMOps.
For teams training, fine-tuning or self-hosting models, the scope becomes broader. They may need traditional MLOps capabilities too: data pipelines, experiment tracking, model registries, training infrastructure, GPU scheduling, inference optimisation, model deployment, drift monitoring and capacity planning.
Neither replaces cloud engineering, SRE, security or platform engineering.
MLOps and LLMOps build on those disciplines.
It does not matter how good your model is if it is too slow, too expensive, unavailable during peak demand, impossible to debug, leaking sensitive data or making untraceable decisions.
Reliable AI is still reliable engineering.
There is no single AI career
One final point for people trying to build a career in this space.
“AI engineer” is too broad to be useful on its own.
There are several career paths, and they require different strengths:
AI application engineers build user-facing AI features, workflows, integrations and tools.
Machine-learning engineers build, train, evaluate and deploy ML systems.
Data engineers create the reliable, governed data foundations that AI systems depend on.
MLOps and platform engineers make AI development, deployment, security and operations repeatable at scale.
Research engineers and applied scientists work more deeply on modelling, optimisation and new capabilities.
Product leaders and engineering leaders choose use cases, create operating models, manage risk and help organisations adopt AI responsibly.
You do not need to be great at all of these.
But you do need to understand how they fit together.
The best organisations will not be staffed entirely by prompt engineers. Nor will they be staffed entirely by researchers.
They will have people who understand users, products, data, software, infrastructure, security, governance and the limitations of the models themselves.
You do not need to be an AI engineer to use AI well
Not everyone needs to become an AI engineer.
Most people will use AI in the same way they use spreadsheets, search engines, collaboration tools and office software: to make everyday work faster, clearer and less repetitive.
You do not need to understand neural networks to ask AI to improve an email, structure a plan, explain a difficult topic, summarise notes or create a first draft.
But you do need to understand the limits.
AI is good at generating, restructuring, summarising and suggesting. It is not automatically good at knowing what is true, what is current, what is confidential or what is appropriate in your organisation.
The basic skill is not learning secret prompts. It is learning how to give a clear brief.
Tell it what you are trying to achieve. Give it the relevant context. Specify the audience, tone, length and format. Ask it to show assumptions, identify uncertainty and suggest what needs human checking.
Then use your own judgement.
Do not paste sensitive data into tools your organisation has not approved. Do not accept important claims without checking them. Do not let an agent send, spend, delete, publish or change something consequential without appropriate controls.
Start with small, reversible tasks.
Use AI to draft the email, not automatically send it. Ask it to organise the meeting notes, not make personnel decisions. Let it suggest an incident checklist, not make production changes without review.
The more access and autonomy an AI system has, the more carefully it must be designed, governed and supervised.
That is AI literacy.
It is not about turning everyone into a machine-learning specialist. It is about helping people use powerful tools with enough understanding to get value from them without giving away their judgement, their data or their accountability.
A practical prompt formula
For everyday users, I would give a simple framework rather than talking about “prompt engineering” as though it were a mysterious technical skill:
Context + Task + Constraints + Output + Check
For example:
I am preparing an update for a non-technical executive team about an engineering incident.
Turn the notes below into a calm, factual update. Keep it under 250 words. Avoid jargon. Include: what happened, customer impact, what we have done, what remains uncertain and the next update time. Do not invent details. Flag anything that needs confirmation.
Notes: [paste approved, non-sensitive notes]
That is enough to make most people significantly better users of AI.
The goal is not to write magical prompts. The goal is to communicate clearly, retain ownership of the work and know when an answer needs checking.
In the end.
The real capability is learning to build dependable systems around uncertain models.
Start with a real problem. Understand the data. Learn the fundamentals at the right depth. Build proper software around the model. Evaluate continuously. Secure everything. Keep humans accountable for meaningful decisions. Operate the whole thing like a production system.
The hot tool will change.
The model provider will change.
The framework will change.
But the organisations that understand how to turn AI capability into trustworthy, measurable user value will keep winning.
About the Author
Diamantino Almeida is a tech leader, coach, and writer reshaping how we think about leadership in a burnout-driven world. With over 20 years at the intersection of engineering, DevOps, and team culture, he helps humans lead consciously from the inside out. When he’s not challenging outdated norms, he’s plotting how to make work more human, one verb at a time.
In my experience to lead AI in an organisation, you need three things, a strong mental model of how AI systems work in production, the engineering depth to build and run them safely, and the leadership skills to organise teams and strategy around them.
These five books cover all three layers. They are practical, not theoretical, and they match the kind of work you already do in cloud, security, and engineering operations.
1) Designing Machine Learning Systems – Chip Huyen
This book teaches you how to think about machine learning as a system, not just a model. Many engineers focus only on building a good model, but in the real world, models are only one part of a larger system that includes data, features, deployment, monitoring, and business impact. This book shows you the full loop, how to scope a problem, collect and validate data, engineer features, train and evaluate models, deploy them, monitor them in production, and then repeat the cycle as the world changes.
Why this is important for you.
As a leader, your job is to make sure AI creates real value, not just interesting experiments. This book gives you the language and framework to spot where projects will fail like when training data does not match production data, or when there is no plan for monitoring model performance over time. It helps you ask the right questions: What data do we need? How will we know if the model is still working? Who owns the feedback loop? For someone building a public voice on responsible AI, this book gives you a solid foundation to explain AI systems clearly to engineers and executives alike.
2) Machine Learning Production Systems – Robert Crowe and team (O’Reilly)
Where the first book gives you the mindset, this book gives you the engineering details. It walks through the entire lifecycle of production machine learning: how to collect and label data, how to build feature stores, how to train and tune models, how to serve models at scale, and how to monitor them once they are live. It also covers newer topics like large language model deployment patterns and MLOps at enterprise scale.
Why this is important for you
This book connects that knowledge to AI. It shows you how to design data pipelines that do not break, how to build feature stores that teams can reuse, and how to serve models in a way that is fast, safe, and cost-effective. For a fractional CTO or senior leader, this is the book you use when you need to audit an AI platform or design one from scratch. It helps you move from “we have a model” to “we have a reliable, scalable AI system that supports the business.”
3) AI Engineering: Building Production-Ready LLM Applications – James Hu
This book is about the new wave of AI, large language models and generative AI. It is written for software engineers who want to build real LLM applications that work in production. It covers evaluation frameworks for non-deterministic systems, retrieval-augmented generation (RAG) at scale, observability for LLMs, and the engineering patterns behind robust AI services.
Why this is important for you
You are already exploring LLMOps, RAG, vector databases, and knowledge graphs. This book speaks directly to that work. It helps you understand the new failure modes of generative AI: when models give wrong answers, when they leak sensitive data, when they are too slow or too expensive. As a leader, you will need to set standards for evaluation, safety, and cost control. This book gives you the tools to do that. It also helps you guide your team on how to build LLM apps that are not just demos, but reliable products.
4) Leading Enterprise AI Programs
This book is about the organisational side of AI. It shows you how to set up an enterprise AI program, how to pick the right use cases, how to build a community of citizen data scientists, and how to embed AI into the way the company creates value. It also covers portfolio management, cost and benefit analysis, and ethical and agile governance.
Why this is important for you
You can be a brilliant AI engineer and still fail to create value if the organisation is not set up to use AI well. This book helps you design the operating model: who owns what, how decisions are made, how use cases are prioritised, and how success is measured. For someone pursuing fractional CTO or COO work, this is essential. It helps you speak the language of business leaders and show them how AI fits into their strategy. It also helps you avoid common pitfalls, like too many pilots and no production, or AI projects that do not tie to business outcomes.
5) Team Topologies – Matthew Skelton and Manuel Pais (with Accelerate as a companion)
This book is not about AI specifically, but it is critical for leading AI teams. It teaches you how to design team boundaries around flow and cognitive load, rather than around job titles or org charts. It shows you how to structure teams so that work moves quickly and safely, without bottlenecks. A companion book, Accelerate, adds the DORA metrics (deployment frequency, lead time, change failure rate, and mean time to restore) that predict how well software teams perform.
Why this is important for you
AI work is complex and creates a lot of dependencies between teams. Without good team design, AI becomes a bottleneck: data teams wait on platform teams, product teams wait on data teams, and nothing ships. Team Topologies helps you design teams that can move fast without breaking things. Accelerate helps you measure whether your teams are actually improving. For a leader who wants to scale AI across an organisation, this is the book that helps you avoid organisational debt.
These five books cover some of the full stack, systems thinking, engineering depth, generative AI specifics, organisational design, and team structure.
Love this: “What user problem, decision or workflow would genuinely improve here?” I’ve been building agentic workflows for the past couple of months, and it never ends up how I thought it would at the start. That’s not a bad thing at all. It’s the process of identifying the problem to solve first, and then the best route to get to the best result. Agents? Maybe. But not necessarily.