In my experience to lead AI in an organisation, you need three things, a strong mental model of how AI systems work in production, the engineering depth to build and run them safely, and the leadership skills to organise teams and strategy around them.
These five books cover all three layers. They are practical, not theoretical, and they match the kind of work you already do in cloud, security, and engineering operations.
1) Designing Machine Learning Systems – Chip Huyen
This book teaches you how to think about machine learning as a system, not just a model. Many engineers focus only on building a good model, but in the real world, models are only one part of a larger system that includes data, features, deployment, monitoring, and business impact. This book shows you the full loop, how to scope a problem, collect and validate data, engineer features, train and evaluate models, deploy them, monitor them in production, and then repeat the cycle as the world changes.
Why this is important for you.
As a leader, your job is to make sure AI creates real value, not just interesting experiments. This book gives you the language and framework to spot where projects will fail like when training data does not match production data, or when there is no plan for monitoring model performance over time. It helps you ask the right questions: What data do we need? How will we know if the model is still working? Who owns the feedback loop? For someone building a public voice on responsible AI, this book gives you a solid foundation to explain AI systems clearly to engineers and executives alike.
2) Machine Learning Production Systems – Robert Crowe and team (O’Reilly)
Where the first book gives you the mindset, this book gives you the engineering details. It walks through the entire lifecycle of production machine learning: how to collect and label data, how to build feature stores, how to train and tune models, how to serve models at scale, and how to monitor them once they are live. It also covers newer topics like large language model deployment patterns and MLOps at enterprise scale.
Why this is important for you
This book connects that knowledge to AI. It shows you how to design data pipelines that do not break, how to build feature stores that teams can reuse, and how to serve models in a way that is fast, safe, and cost-effective. For a fractional CTO or senior leader, this is the book you use when you need to audit an AI platform or design one from scratch. It helps you move from “we have a model” to “we have a reliable, scalable AI system that supports the business.”
3) AI Engineering: Building Production-Ready LLM Applications – James Hu
This book is about the new wave of AI, large language models and generative AI. It is written for software engineers who want to build real LLM applications that work in production. It covers evaluation frameworks for non-deterministic systems, retrieval-augmented generation (RAG) at scale, observability for LLMs, and the engineering patterns behind robust AI services.
Why this is important for you
You are already exploring LLMOps, RAG, vector databases, and knowledge graphs. This book speaks directly to that work. It helps you understand the new failure modes of generative AI: when models give wrong answers, when they leak sensitive data, when they are too slow or too expensive. As a leader, you will need to set standards for evaluation, safety, and cost control. This book gives you the tools to do that. It also helps you guide your team on how to build LLM apps that are not just demos, but reliable products.
4) Leading Enterprise AI Programs
This book is about the organisational side of AI. It shows you how to set up an enterprise AI program, how to pick the right use cases, how to build a community of citizen data scientists, and how to embed AI into the way the company creates value. It also covers portfolio management, cost and benefit analysis, and ethical and agile governance.
Why this is important for you
You can be a brilliant AI engineer and still fail to create value if the organisation is not set up to use AI well. This book helps you design the operating model: who owns what, how decisions are made, how use cases are prioritised, and how success is measured. For someone pursuing fractional CTO or COO work, this is essential. It helps you speak the language of business leaders and show them how AI fits into their strategy. It also helps you avoid common pitfalls, like too many pilots and no production, or AI projects that do not tie to business outcomes.
5) Team Topologies – Matthew Skelton and Manuel Pais (with Accelerate as a companion)
This book is not about AI specifically, but it is critical for leading AI teams. It teaches you how to design team boundaries around flow and cognitive load, rather than around job titles or org charts. It shows you how to structure teams so that work moves quickly and safely, without bottlenecks. A companion book, Accelerate, adds the DORA metrics (deployment frequency, lead time, change failure rate, and mean time to restore) that predict how well software teams perform.
Why this is important for you
AI work is complex and creates a lot of dependencies between teams. Without good team design, AI becomes a bottleneck: data teams wait on platform teams, product teams wait on data teams, and nothing ships. Team Topologies helps you design teams that can move fast without breaking things. Accelerate helps you measure whether your teams are actually improving. For a leader who wants to scale AI across an organisation, this is the book that helps you avoid organisational debt.
These five books cover some of the full stack, systems thinking, engineering depth, generative AI specifics, organisational design, and team structure.
Love this: “What user problem, decision or workflow would genuinely improve here?” I’ve been building agentic workflows for the past couple of months, and it never ends up how I thought it would at the start. That’s not a bad thing at all. It’s the process of identifying the problem to solve first, and then the best route to get to the best result. Agents? Maybe. But not necessarily.
It's a very dynamic process, especially with AI agents, I keep in mind that I'm dealing with a probabilistic model, trying to attain a deterministic outcome, to manage my expectations.
In my experience to lead AI in an organisation, you need three things, a strong mental model of how AI systems work in production, the engineering depth to build and run them safely, and the leadership skills to organise teams and strategy around them.
These five books cover all three layers. They are practical, not theoretical, and they match the kind of work you already do in cloud, security, and engineering operations.
1) Designing Machine Learning Systems – Chip Huyen
This book teaches you how to think about machine learning as a system, not just a model. Many engineers focus only on building a good model, but in the real world, models are only one part of a larger system that includes data, features, deployment, monitoring, and business impact. This book shows you the full loop, how to scope a problem, collect and validate data, engineer features, train and evaluate models, deploy them, monitor them in production, and then repeat the cycle as the world changes.
Why this is important for you.
As a leader, your job is to make sure AI creates real value, not just interesting experiments. This book gives you the language and framework to spot where projects will fail like when training data does not match production data, or when there is no plan for monitoring model performance over time. It helps you ask the right questions: What data do we need? How will we know if the model is still working? Who owns the feedback loop? For someone building a public voice on responsible AI, this book gives you a solid foundation to explain AI systems clearly to engineers and executives alike.
2) Machine Learning Production Systems – Robert Crowe and team (O’Reilly)
Where the first book gives you the mindset, this book gives you the engineering details. It walks through the entire lifecycle of production machine learning: how to collect and label data, how to build feature stores, how to train and tune models, how to serve models at scale, and how to monitor them once they are live. It also covers newer topics like large language model deployment patterns and MLOps at enterprise scale.
Why this is important for you
This book connects that knowledge to AI. It shows you how to design data pipelines that do not break, how to build feature stores that teams can reuse, and how to serve models in a way that is fast, safe, and cost-effective. For a fractional CTO or senior leader, this is the book you use when you need to audit an AI platform or design one from scratch. It helps you move from “we have a model” to “we have a reliable, scalable AI system that supports the business.”
3) AI Engineering: Building Production-Ready LLM Applications – James Hu
This book is about the new wave of AI, large language models and generative AI. It is written for software engineers who want to build real LLM applications that work in production. It covers evaluation frameworks for non-deterministic systems, retrieval-augmented generation (RAG) at scale, observability for LLMs, and the engineering patterns behind robust AI services.
Why this is important for you
You are already exploring LLMOps, RAG, vector databases, and knowledge graphs. This book speaks directly to that work. It helps you understand the new failure modes of generative AI: when models give wrong answers, when they leak sensitive data, when they are too slow or too expensive. As a leader, you will need to set standards for evaluation, safety, and cost control. This book gives you the tools to do that. It also helps you guide your team on how to build LLM apps that are not just demos, but reliable products.
4) Leading Enterprise AI Programs
This book is about the organisational side of AI. It shows you how to set up an enterprise AI program, how to pick the right use cases, how to build a community of citizen data scientists, and how to embed AI into the way the company creates value. It also covers portfolio management, cost and benefit analysis, and ethical and agile governance.
Why this is important for you
You can be a brilliant AI engineer and still fail to create value if the organisation is not set up to use AI well. This book helps you design the operating model: who owns what, how decisions are made, how use cases are prioritised, and how success is measured. For someone pursuing fractional CTO or COO work, this is essential. It helps you speak the language of business leaders and show them how AI fits into their strategy. It also helps you avoid common pitfalls, like too many pilots and no production, or AI projects that do not tie to business outcomes.
5) Team Topologies – Matthew Skelton and Manuel Pais (with Accelerate as a companion)
This book is not about AI specifically, but it is critical for leading AI teams. It teaches you how to design team boundaries around flow and cognitive load, rather than around job titles or org charts. It shows you how to structure teams so that work moves quickly and safely, without bottlenecks. A companion book, Accelerate, adds the DORA metrics (deployment frequency, lead time, change failure rate, and mean time to restore) that predict how well software teams perform.
Why this is important for you
AI work is complex and creates a lot of dependencies between teams. Without good team design, AI becomes a bottleneck: data teams wait on platform teams, product teams wait on data teams, and nothing ships. Team Topologies helps you design teams that can move fast without breaking things. Accelerate helps you measure whether your teams are actually improving. For a leader who wants to scale AI across an organisation, this is the book that helps you avoid organisational debt.
These five books cover some of the full stack, systems thinking, engineering depth, generative AI specifics, organisational design, and team structure.
Thanks for these book recommendations!
Thank you, there are many more, still I think these are a solid ones.
Love this: “What user problem, decision or workflow would genuinely improve here?” I’ve been building agentic workflows for the past couple of months, and it never ends up how I thought it would at the start. That’s not a bad thing at all. It’s the process of identifying the problem to solve first, and then the best route to get to the best result. Agents? Maybe. But not necessarily.
It's a very dynamic process, especially with AI agents, I keep in mind that I'm dealing with a probabilistic model, trying to attain a deterministic outcome, to manage my expectations.
An excellent article. Thank you.
I found that is best to start with a why…
Thank you Kevin.