Infrastructure as code IaC (tech pill)

Infrastructure as code (IaC) is the practice of defining/declaring the infrastructure we need for a system using some kind of machine-readable source files. These source files are used by a tool to provision, create or maintain in a defined state our infrastructure.

These definitions help to provision/create a different kind of resources, compute, storage, communication services, network

For cloud-based infrastructures, we can use these definitions to create "virtual resources" and to configure them to be in a certain state. For example, we can create a virtual machine with an initial  OS image and later install some software and configure it.

In a bare-metal environment, we can use the definition to configure a fixed number of machines and devices already defined in an inventory.

The goals of this practice are:

• Avoid server configuration drifting
• Avoid proliferation of Snow flake Servers.
• Reduce drastically the maintenance cost and the total cost of ownership.
• Allow easy and infrastructure evolution.

As a collateral effect, this practice also allows:

• Use development practices for the infrastructure (version control, testing, audit, collaboration, live documentation...).
• Create on-demand systems for development, QA, testing, and experimentation.
• Developers collaboration.

Cons:  really, it's 2018... no, seriously I don't find a good reason to not to create infrastructure as code. And when the problem is that these resources are difficult or impossible to define using code or some kind of definition, we should avoid them as much as possible.

General Approaches and styles

• Push. We execute a tool that parses the definitions, calculate the changes to do and execute them.
• Pull. Each node/device have its definition and execute all the time in a loop executing the needed changes.
• Push + Pull. A combination of the previous ones, so we can push a change (when we need to be sure or force some changes) or wait until each node/device update its configuration 

Sometimes we can restrict the IaC to provision the low-level infrastructure. For example, we can use this practice to create a PaaS (using Kubernetes, or similar), so the rest of elements are dynamically provision inside the PaaS we have created.

Related Tools:

• Terraform
• Puppet
• Chef
• Ansible
• SaltStack
• Cloudformation
• https://cloud.google.com/deployment-manager/

In summary, IaC is a core DevOps practice and is the base for a lot of the innovations and evolution that the cloud brings us. It is a must for modern development in the cloud and also very recommendable for on-premise deployments.

Other tech pills:

• http://www.eferro.net/2017/08/scalable-design-pattern-sample-tech-pill.html
• http://www.eferro.net/2017/09/pub-sub-swiss-army-knife-tech-pill.html
• http://www.eferro.net/2017/07/queues-vs-distributed-logs-tech-pill.html

Good talks/podcasts (May 2018 I)

These are some interesting talks/podcast that I've seen/heard during the past month:

• Cloud Native related:
• Anatomy of a Production Kubernetes Outage Oliver Beattie A great example of a complex outage and the learning derived from it.
• Altitude NYC 2018: Observability workshop Peter Bourgon Good workshop to understand how to instrument our applications for good telemetry, logging, and tracing.
• Kubernetes: Finally...A True Cloud Platform Sam Ghods A good description of kubernetes as a base abstraction for the cloud platforms.
• CraftConf 2018. Always a great content at this conference.
• Test-Driven: The Four-Step Dance Tim Ottinger Very interesting detail for improving our TDD practice.
• What is this cloud native thing anyway? Sam Newman A good summary of the topic.
• Five key challenges for software quality tomorrow Gojko Adzic This talk help us to understand the new challenges to test our systems in this new and changing world (ML, AI, cloud services, etc).
• Scaling Your Architecture with Events and Services  Randy Shoup New version of a classic talk from Randy. You can use as a map for dealing with data in microservices and to design using events.
• Simplify the stack  Matt Aimonetti Good talk about simplicity and how you can use "Requests For Comments" to discuss design ideas.
• Other
• The death of Agile Allen Holub (Related with the interesting blog post "Developers Should Abandon Agile") Thanks Fran Reyes for the reference.
• Computers for Cynics [full version] Ted Nelson. General historic information and deep ideas about technology.
• The Formula for Happiness from Solve for Happy An interesting interview with Mo Gawdat.
• The Next Generation of Data Products Hilary Mason. A good introductory talk for newbies like me.
• Old Is the New New Kevlin Henney Very interesting remainder about the bases and about how to identify these core ideas and the new improvements based on them. Understanding this process can help us to improve even faster if we learn and go deep in the bases.

DevOps talks

As a complement of the previous post DevOps concepts and learning path and in case that you prefer watching talks, these are some great talks about DevOps:

Bases for DevOps (Lean / Flow):

• The Efficiency Paradox Niklas Modig. Only 18m to understand the efficiency of flow vs efficiency of resources.
• Competing On The Basis Of Speed Mary Poppendieck
• The Value of Flow 14 09 17 Dan North 27m Great explanation for flow efficiency for soft delivery

DevOps:

• Devops and Dr Deming’s 14 Points John Willis' Ignite talk (Velocity NY 2014) 5mins to understand the relations between DevOps and the ideas of Dr. Deming.
• The (Short) History of DevOps Damon Edwards. 12m
• DOES17 London - The Key to High Performance What the Data Says

Related:

• Continuous Delivery Sounds Great But It Won’t Work Here Jez Humble. A new version of a classic one. A great talk.
• How Netflix Thinks of DevOps
• Fail Better: Radical Ideas from the Practice of Cloud Computing Tom Limoncelli
• Evolutionary Architecture and Fitness Functions
• The Virtuous Cycle of Velocity: What I Learned About Going Fast at eBay and Google Randy Shoup

Related posts:

• DevOps concepts and learning path

DevOps concepts and learning path

As Jez Humble said at leading a devops transformation DevOps is:

A cross-functional community of practice dedicated to the study of buildings, evolving and operating rapidly changing, secure, resilient system at scale. 

DevOps includes and enhance the ideas of the Agile Software Development giving a more end to end vision of the value stream for a technology-based company. And, you know, Every Company Is A Tech Company.

In summary, I think that DevOps should be the core of any technology-based company and the only option to reach/pursue the necessary business agility.

Main characteristics of DevOps culture:

• Collaboration between development and operations (avoiding SILOs and conflicting goals).
• Organize around the value stream optimizing for the flow efficiency (not resource efficiency). Learning to work in small batches.
• Remove waste (Non-Value Adding Activities).  Of course, we talk about value from the customer point of view.
• Build quality in.
• Create fast feedback loops.
• Maximize organizational learning (making safe to fail and learn).

Common practices:

• Infrastructure as code.
• Developers are involved in operations of the system (you build it, you run it).
• Operations involved in the development from the beginning (introducing/facilitating specific, nonfunctional requirements to create a system easy to operate and monitor).
• Automation (to avoid errors and to facilitate short iterations).
• Continuous Delivery.

Books / Learning path:

If you are interested in learning about this culture and you like reading books, this is the learning path I recommend:


 

The Phoenix project: As a great and easy to read an introduction to Lean in an IT environment. (My review)


 

 The DevOps handbook: The complement for The Phonix Project that explains step by step the typical practices and the strategy to introduce DevOps. A short of practitioner’s guide.


 

Continuous Delivery: To learn the principles and technical practices that enable rapid and incremental delivery of high-quality, valuable software to our customers.




Accelerate: To understand how to build and scale high performing technology organizations creating this DevOps culture. This book is also an analysis of the data from the State of DevOps reports that give an idea about the importance of these practices.

 

 

 

Team Topologies: To have a common vocabulary and organize the different types of teams that suit the business needs. The recommended team organizations try to optimize for end to end flow.

If you are involved in any technology company, do you a favor and learn about DevOps... As the State of DevOps report indicates is the key to be a high-performance organization.

Related posts:

• https://www.eferro.net/2018/05/devops-talks.html

Good talks/podcasts (April 2018)

These are some interesting talks/podcast that I've seen/heard during the past month:

• DevOps related:
• How Netflix Thinks of DevOps
• DOES17 London - The Key to High Performance What the Data Says 
• a16z Podcast: Feedback Loops  Company Culture, Change, and DevOps with Nicole Forsgren, Jez Humble, and Sonal Chokshi
• Cloud related:
• Five Cloud Native Ops Superpowers: Yes, You Can Do That! Dave Bartoletti, Forrester
• Fail Better: Radical Ideas from the Practice of Cloud Computing Tom Limoncelli
• Other topics:
• Emotional Intelligence for Engineers  April Wensel
• a16z Podcast: Improv’ing Leadership with Dick Costolo and Peter Levine
• Podcast. Interview Ron Jeffries (Agile.FM) Very interesting interview (darkscrum, agile adoption failures, number agile coaches vs the number of agile developers).

Don't blame the Manifesto for Agile Software Development

From time to time I hear people saying that Agile is not the same thing as "technical practices" and software delivery, that the Agile manifesto makes no sense for the whole company...  And I agree and disagree entirely.

A bunch of software developers created the famous manifesto. And the complete name is Manifesto for Agile Software Development. The title is not Agile Manifesto or Manifesto for Business Agility. It was created by developers, to improve how we develop software and it was an excellent improvement on software creation 20 years ago. Right now it is not good enough and should be adapted to the industry evolution (perhaps including DevOps, Continuous Delivery or ideas from Modern Agile).

If you want an adaptable business that's optimized for this VUCA world, you need business agility and in a lot of cases these days, agile software development. The good news is that all of these concepts are grounded on the same roots, and principles and need a common culture (respect for people, continuous learning, continuous improvement, etc.).

So don't blame the Manifesto for Agile Software Development or the agile software developers for not solving problems that are outside the original scope and need other skills and practices. Complement this way of doing software development with other great disciplines that have common roots and principles and are very compatible.

When you are trying to do the right thing, learn about lean product development, design sprints, lean startup, customer research, dual track agile, etc. When you need to do the thing right (assuming software based company), learn about Extreme Programming (XP), DevOps, Continuous Delivery, Scrum, etc.

So focus on Continuous Incremental Improvement and Respect for People. Do the right thing and do the thing right.

References and Related posts

Agile is not a recipe for success...

The two pillars of agile software

Remove the excuses for not being a professional

http://www.extremeprogramming.org/

http://modernagile.org/

Continuous Delivery