DevOps explained : The Phoenix project

DevOps is an important IT concept that is often misunderstood. DevOps is not just a better collaboration between development and operations. It is a new way of working between marketing, product management, development, test, operations and IT security. One of the few books that explain this well is The Phoenix Project. In this article we review the book and provide a summary of the core DevOps concepts.

Why DevOps?

Many businesses have a complicated IT landscape with many different systems, servers and databases. Running these systems is not a simple matter of turning them on- and off: each system needs to be configured, provided with data, monitored and possibly restarted or reconfigured in case of issues. The task of doing all this falls to IT operations, a department that is often invisible.

Because their work is often invisible, IT operations is often under-appreciated. Keeping all systems up and running has become more complex is recent years because many companies have more smaller systems that all need to be connected in the right way. There are also higher security and auditability requirements, more connections to outside systems, and an increased number of patches and security fixes for all components. Even without any software development, there is a lot of work to do.

Many organisations such as banks, utility companies and other digital businesses are experiencing firsthand that their operations environment has become a bottleneck: they are not able to keep up with a required changes and tests, and suffer smaller and larger incidents as a result. DevOps promises to solve this mess by forging a close collaboration between Dev (development) and Ops (operations). The term was introduced in 2008 and became widely known in 2009 from the famous ‘10+ deploys per day Dev and Ops collaboration at Flickr’ presentation. The ‘dev and ops’ image illustrates one idea from this presentation, notably that dev and ops should work closer together. Both dev and ops have the same goal, to enable business. For ops this means that they should not just keep systems stable: the business requires change so ops should enable change (and vice versa: developers should also be responsible for keeping systems stable).

The implicit promise here is that companies doing DevOps can release software improvements more than 10 times a day. Since most companies struggle to make one change without mistakes every two weeks, this is a very attractive proposition. DevOps promises to speed up business and reduce mistakes and outages. Most people are excited about this promise, but do not understand what exactly DevOps is and how to get started. The Phoenix project (one of our recommended project management books) was written to help any reader understand what exactly DevOps is and how it helps IT operations.

The Phoenix project – a business novel

The business novel is a book format where a usually-not-so-good story is used to explain business ideas. In this case, the story focuses on Bill, an IT middle manager in the company ‘Parts Unlimited’ and the story focuses on his struggle to get IT under control in parts unlimited. He is initially promoted, has to solve burning problems, makes changes with the help of its team, walks out of his job, walks back in and is ultimately successful.

The book is structured in two parts: Part 1 explains the problems of the Phoenix project, a large and prestigious project that is held up by a struggling operations department. It is a typical waterfall project that is way behind schedule. It is eventually launched into production with disastrous results.
The second part of the book deals with project Unicorn, a new project that is organized differently by Bill based on his new understanding of operations. Using the new knowledge that Bill and we learned from the first half, the system is designed with operational requirements in mind. As a result all operations issues are avoided and the Unicorn project is successful.

As a book it is not very exciting, but the business novel format does make it easy to read. We recommend the book as a high level introduction for any team or company that wants to start with agile, especially because of the first part of the book. Every developer that wants to start with DevOps, first needs to understand what operations exactly is and why it is challenging. Only when you understand normal IT operations, you can improve it with DevOps ideas.

What is operations and what is DevOps?

If you would have to summarize DevOps in one sentence, it would be that it is important to appreciate the complexity of IT operations. The operations department is the place where everything comes together: new systems, changes to existing systems, sales and marketing expectations, audit requirements and security concerns. The amount of work in operations is often not understood by other departments because they see only part of the work. The book defines four major types of work:

• Business projects: These are new projects initiated by the business to generate new value
• Internal IT projects: These projects are necessary for technical reasons, for instance upgrades of storage, replacing unsupportable hardware or migrating to new suppliers.
• Changes: Changes are small work packages that are executed in one go, for instance the replacement of a single server or change of one setting. Changes are often generated from the previous work types. It is important to plan and visualize each change, because each change can have unintended effects that must be resolved immediately.
• Unplanned work or recovery work: This type of work is recovering from incidents and fixing issues caused by poorly planned or executed changes. Ideally one should avoid this type of work. However if an Operation departement is overwhelmed with unplanned work, there often is no time to plan the other activities properly.

Many IT departments do not have a clear overview of all changes and this leads to all kinds of surprises and unplanned work. Many IT departments also have not documented their operations knowledge, and are therefore dependent on a small and overworked group of experts that have all required knowledge in their head. In such environment, the amount of unplanned work becomes a self-sustaining circle: due to the amount of unplanned work and limited supply of experts, there is no time for planning and preventive work. This leads to future unplanned work for the limited supply of experts.

The solution suggested in the book is to first make all changes visible with a complete change calendar. Secondly one must identify the bottleneck resources (often specific experts) and make sure they only work on the most valuable type of work. This way of working is actually quite similar to the way Lean manufacturing works. The authors of the book state this explicitly: in their view, DevOps is just the application of Lean Principles to the IT value stream. This is a fresh perspective because it means that DevOps can be applied to any IT department, not just to IT departments with only new systems.

Using this definition, it becomes clear that DevOps is not just more communication between developers and operations. It requires developers and other departments to respect the change calendar and listen to the operations department to understand the available change capacity. Vice versa it means that the teams in the operations departments that manage the change calendar know the value of each change. They need to understand business needs, security requirements and the reason of IT projects. In practice this means that people from other departments must work inside these teams and help operations make the right decisions. In theory this sounds simple, but in practice this is a completely different way of working for all departments involved. Just to give an example: marketing staff can no longer plan marketing activities independently: they must align the content and timing of any message with the operations department to make sure all systems are ready for both the marketing action itself and the expected response.

Developing for DevOps

The second part of the book is dedicated to development of new systems with DevOps in mind. In this part, we get an understanding of the technical changes that are made by DevOps teams when they design new systems. The following list of ideas give an example of what when can do when one masters DevOps. It is not the core of DevOps, but rather a list of possible effects:

• Make changes to IT systems more than ten times a day, by automating all deployment steps.
• Test changes in production by developing switches so that new features can be turned on and off instantly or be shown to a subset of customers
• Design the system from the start with audit controls and security requirements in mind. All information is logged automatically so that no additional reports are ever needed
• Make all infrastructure settings scripted, and use exactly the same scripts in development and production. If one uses virtual machines or containers, one can make the production and development environments exactly alike and all errors will be reproducible in all environments
• Use the infrastructure scripts as ‘active documentation’ so that anyone in the team can safely deploy changes in any environment. The scripts, all of which are stored in version control systems along with the core, replace the live experts.

This ‘radical DevOps’ story uses many agile ideas: the DevOps approach includes agile ideas such as talking with the business first before starting development. Also the idea of deploying software often is essentially an agile idea. What makes DevOps and this book differently, is that it acknowledges that agile only works if one resolves the operational issues.

Conclusion

The Phoenix project is a valuable book because it provides a very different introduction to DevOps. Instead of focusing on a single tool or a single method, it provides a high level introduction of what IT operations and DevOps is about. It avoids discussing individual tools, but focuses on understanding IT operations. The disadvantage of the book is that it only provides the introduction. The book does not offer any practical advice. Teams using this book will need help from experience DevOps experts to bring the ideas from this book in practice.