The Phoenix Project

I recently read “The Phoenix Project”. I have no commercial interests in it, but would like to recommend it. It is a book about Lean, Agile and in particular DevOps. It is not written like a text-book, but like a novel.  It wont win any literature prizes, but it’s quite entertaining – if you have the right background.

The book describes how Bill, a recently promoted manager for “IT Operations” in his company “Parts unlimited”, is completely bogged down in firefighting and politics. The company is screaming for the delayed Phoenix project to save their lost market shares, while they are hit by one disaster in IT after another. Bills Operations team is “Ops”, while the R&D team developing Phoenix naturally is “Dev”.

Gradually Bill and his team improve their habits and processes. They are helped by a strange new board-member who talks about dogmas described by sensei this and that. He helps Bill, using trips to the factory catwalk as the main instrument.

Here are some of my main learnings from the book.

The three ways of DevOps

• Understand the value stream and identify the flow (left to right).
  Make it fast and un-interrupted.
  Avoid the need to go backward for a redo.
  Find and relieve the critical resources – like Brent (hint: look for the queues). In classic Gantt-planning you look for the critical path before you execute. Here you look for the critical resource(s) – while you execute.
• Feedback from right to left.
  Improve based on learnings.
  Fix problems at the source. Only really possible if you have a fast turnaround time.
  Grow from workstation controller to factory manager.
• Daily improvements.
  More important than daily work is … improvement of daily work.
  A culture of no blame but transparency.
  Controlled experiments.

The four types of work

• Business Projects (like the Phoenix Project and later Unicorn)
• Changes (here in the sense of vendor upgrades etc)
• Internal IT Projects
• Unplanned work

Due to “technical dept” and terrible processes, most of the work done is “Unplanned Work”.

One of the main advantages in having many – planned – releases per day is that it’s a no-brainer to fix errors at the source. If the path is long you start “patching” – making changes to what is built, because you don’t have time for anything else. You might get something that works, but do you also get these fixes implemented back in the source?

Virtualization

Virtualization is just one of many instruments. It has many advantages:

• Same environment in Dev, QA, and Prod.
• Scaling (also to cloud)
• From 0 to 100% config of a PC/Server by copying a file

My thoughts on the book

At the back of the book you find “The three ways as excerpted from the DevOps handbook”, followed by an Introduction. You might want to read at least the introduction first. I guess its hidden at the back to help Amazon users, use the “Look Inside” feature to instantly get into the action. I must admit that it worked for me.

The main character, Bill, does many things intuitively right – like protecting his bottleneck resource (Brent), and reserving him for the most important stuff, leaving as much as possible to others.

However, there are also many things explained that are less intuitive – e.g. like the advantages of smaller batch sizes, or the fact that the better a critical resource is utilized – the longer the wait.

Hand-overs introduce delays – especially between departments (Dev & Ops).

When Dev delivers source-code into version-control, the job is far from done. There is still the build/run environments: OS version, memory size, database version and servers etc. Dev, QA and Ops each use different environments. We need somehow to include the environment in the delivery. The next step is thus to deliver a package with compiled code and environment. Virtual image or Docker…

In embedded software I once experienced separate images for CPU, DSP and several FPGAs in a suite of different – but similar – PCBs. We bundled it all in one complete downloadable image, which saved us a lot of problems. Simple load and no mixed versions.

All changes must be managed instead of letting the highest shouting “customer” win. When various changes are introduced “randomly” without warnings to stakeholders, they may affect each other where and when it hurts most.

In the book, the “Chief Information Security Officer” (John) gets a rough treatment, because he just stands there shouting all his impossible requirements while the brave IT guys are fighting to save the company. Somehow, they manage to survive turning the deaf ear to this guy, until he changes his ways and helps them instead. The morale is that security compliance must work with – not against the project.

The book is sprinkled with political fights and an “evil” person (Sarah) that constantly tries to convince the CEO that it’s all IT’s fault, while she destroys the progress actually made. Just for the record: there are also good women (like Patty).

Bill is sometimes too “good”, like when he moves the CFOs tossed-out paper to the compost waist-basket. Not that I am against this, it’s just a little bit too much.

It is tempting to think that “yeah, you can apply lean methods because its about ops. ‘Real’ Development has less repetitions, and requires more innovation, so you cannot use it here”. But that might just be repeating Bills first mistake when IT Ops is compared to the factory floor.

You can find the book on Amazon here.