Friday, December 07, 2018

Tejones Conf 18 / the next step of the tribe

The Joke

As I have described in previous posts, TheMotion's original team evolved into a kind of tribe with a shared culture (honeybadgers from team to tribe). A fundamental characteristic of this culture is sharing and learning as a team. So the next logical step in this evolution was to generate a space for just that... learning and sharing... our invitational conference, or rather, unconference.  Our Open Space, the HoneyBadgersConf / TejonesConf18.

It started almost as a joke, but we quickly convinced ourselves that it was an excellent opportunity to enjoy together and learn at the same time.

It took us a long time to agree on a date, but a month ago we were finally able to book a country house to spend the whole weekend and organize the openspace.



The Open Space / Sessions

We made a single track openspace. But although we were few we had many proposals so in the end some talks/sessions were left out of the panel.
  • Craftsmanship. We saw together the talk Run 2018 - Masterclass: Craftsmanship (talk in Spanish from Xavi Gost) and discuss the ideas present in the talk. A very interesting discussion. I proposed this discussion to confront some ideas about agile, the dark side of agile and how to transmit the idea of agile software delivery.
  • Show me your data pipeline. In this session, we talked about data processing workflows. We didn't focus on technology. The focus was on organizational and team design to develop and operate these data pipelines. You know, Conway's law. :) The session was proposed and conducted by Jorge Jardines.
  • Traceability and observability. Fran Ortiz introduced us to the concepts of distributed tracing and observability for applications. We talked about opentracing, zipkin, and honeycomb and were able to make a small demo using opentracing.
  • My life as a junior developer. Georgina gives a talk about her exciting evolution in our profession. It was a very inspiring talk and full of insights.  We talked about power relations, diversity, team culture, power dynamics, etc.
  • A light introduction to Terraform. In this session, Ivan explained how to use Terraform. He detailed how it compares to other tools and approaches for systems/resources provision. There was a lot of diversity in the degree of knowledge about systems architecture, cloud, etc. This diversity generated a fascinating conversation that we continued in the next session.
  • Kubernetes. In this last session Abel, Ivan and I explained the basic concepts of kubernetes and how they fit with the idea of applications that follow 12 factors.

The idea was to have more sessions, but the truth is that the lunches and dinners were longer than expected :)


Conclusions

The result of the experiment was remarkable, and although due to date conflict some of the Badgers could not come, we want to repeat the next year and even invite some friends.

On the one hand, I feel part of the tribe and its culture, and on the other hand an astonished spectator of this process of growth. For me, this process has been as surprising as it has been rewarding.

Related posts and Info

Saturday, November 17, 2018

Good talks/podcasts (Nov 2018 I)


Some good talks that I show lately:

Sunday, October 28, 2018

My Software Crafters Barcelona 2018

Dates: 29th, 30th September
Official site: http://scbcn.github.io/

A month ago I had the pleasure of attending the Software Crafters Barcelona 2018. I think it's the fourth time I've been to this conference and it's among my favorites.

I loved this year's edition. The first two years the conference had around 80 attendees and an excellent environment for networking and learning in community.
Last year the conference, and we were around  200 participants, which was a major logistical challenge. Although I liked last year's decision, the truth is that we lost some of the networking and community feeling. However, this year the organizers have overcome all the challenges and achieved the same atmosphere that we had in the first editions.

The venue was in the center of the city, very well communicated and comfortable. The rooms used for the talks and workshops were, I think, very suitable. The only thing I think could be improved is that the English track was a little difficult to access. Still, I think the venue was a great choice.

Another thing I loved this year, is that the open space part was done on Saturday afternoon, instead of Sunday, as was done before. This decision has allowed more people to participate in these open space sessions.



The sessions I attended:


A cowgirl journey Slides Meritxell Calvo Video
Entertaining and inspiring talk about the path she has taken within the industry and the milestones that have marked her. Very interesting to see the importance of having women referents in technology, community and mentoring.

Communication Antipatterns and where to find them Slides Emma Baddeley
As an introductory talk it went very well. He did some emphasis on distributed teams but it wasn't the main focus.

The Science of Not Estimating Slides Anemari Fiser, Javier Sánchez Rois.
An excellent summary of the problems with estimations and how to use some other alternatives as probabilistic forecasting based on real data from the team. It was fascinating how they used Montecarlo simulations based in the past cycle time and lead time of the user stories delivered by the team.
Some interesting references:


Reactive microsystems Slides Ignasi Marimon-Clos 
The talk described how to evolve from Rest Microservices to more reactive microservices using CQRS. For each step of the evolution, Ignasi described the pros and cons and the actual options for the next step. The content of the talk was great, but the following conversation was even more significant.

Kubeflow (Workshop) Laura Morillo
We follow the lab "Introduction to Kubeflow on Google Kubernetes Engine"
It was an introduction to kubeflow and concepts needed to use it (kubernetes, ksonnet). It was perfect for me because it was not to depth but let me understand the architecture and the development flow.

Parallel Changes. Big changes in small steps.
I facilitated this workshop and from the feedback I got, it was quite good. :) Slides and additional information in the following blogpost.




Timeless Streams Slides Javier Salinas Polo https://twitter.com/MuSTa1nE
Interesting session on how to solve the problem of time use in systems that use event streams. Finally, they came up with a solution by sending timestamps as events. They used these marks to demarcate periods so that they could test without the need to depend on time.


Open space Sessions:

Going to production, Help! Session moderated by Laura Lacarra
During the session, we shared different ideas about how to push to production and operating software systems. We could see how there were two groups, one more aware of the DevOps culture that considered production as part of value delivery and another group that still saw a clear separation between development and operations. We talked about incident reports and other learning tools (game days, chaos engineering). We also talked about the DORA report, and some reference books.


When there is diversity, do you know how to manage it? Session moderated by Raquel Lainde
Some ideas that I extracted from the session:
  • We should provide training on diversity (because it is a great asset, but we usually are not prepared to manage it).
  • Learn about biases and social psychology help us to understand the problems and our reactions.
  • It is much easier for someone with privileges to correct or point out non-inclusive behavior to another person with privileges (less risk involved).
  • It is essential to identify and try to be attentive to situations of discrimination.
They recommended this book that I already have on my reading list:

Other notes and stuff:

As every year I leave with a lot of new ideas, experiments to do, some devirtualizations, a lot of interesting conversations and a great sense of community. I am increasingly proud to have a profession that considers sharing and learning to be fundamental. 

Thank you very much to the organization for the impressive work you do. Each edition, I left Barcelona with the desire to return the following year. 



Saturday, October 27, 2018

Good talks/podcasts (Oct 2018 II)


Some good talks that I show lately:

Saturday, October 13, 2018

No Business Agility without Agile Software Delivery

Business agility is not possible only doing agile software delivery. Furthermore, business agility is impossible without doing agile software delivery.



The bad news is that using scrum without agile development practices is not agile software delivery (and will collapse faster than you think).

Agile software delivery:
- Requires to be capable to put in production and operate a continuous train of product increments during all the product life.
- Fast feedback for each step of the value stream.
- Requires automatic testing and requires to include quality in the process.
- Good feedback for the system in production (monitoring, observability, operability, etc).

So if you are a tech company (why every company is a technology company) and you want to improve your business agility, don't forget to improve your agile software delivery capacity. If you have doubts, DevOps and XP are a great start to complement your scrum process.

Related posts




Monday, October 01, 2018

Slides Taller Parallel Changes (Barcelona Software Crafters 2018)

Slides que he usado para el taller sobre cambios paralelos en la Barcelona Software Crafters





Presentación original:
Taller Parallel Changes (Barcelona Software Crafters 2018)




Esta ha sido la cuarta versión de este taller. Gracias a todos los asistentes por el valiosisimo feedback que me habeis dado y que me va a permitir mejorar el taller.

Muchas gracias a la organización por haber contado conmigo y por todo el apoyo.



Si has estado en el taller y quieres dejar feedback: https://joind.in/event/software-crafters-barcelona-2018/cambios-en-paralelo-cambios-grandes-pasos-pequeos-workshop

Monday, September 10, 2018

Good talks/podcasts (Sept 2018 I)


Some good talks that I show lately:

Friday, August 31, 2018

Good talks/podcasts (Aug 2018 II)

Some good talks that I show lately:

Sunday, August 19, 2018

Cloud mindset (tech pill)

To create a significant software system using the cloud we need to think differently than when we are building a system to run in our data center or using VPS from a classic hosting provider.



Outside the Cloud (data centers and VPS),  the underlying computing resources are:

  • Effectively fixed because obtaining new resources requires months.
  • They are scarce because we try to buy the minimum resources needed.
  • Expensive since we have to pay them entirely and amortize them.

But in the Cloud, the underlying computing resources are abundant and fast to provision and this change the rules of the game and deserve a mindset change.



In the Cloud we should:

  • Consider the servers/disks/databases as disposable resources, treated as "just one more". This concept is called Pets vs Cattle and even if I don't like the analogy too much, it is true that it is interesting to read about it: The History of Pets vs Cattle and How to Use the Analogy Properly 
  • Always assume that anything will fail and design for it. The Cloud is based on commodity hardware that is not under our control and allows us to create massive scale systems. Due to the scale, we experience partial failures all the time (machines go down, disks broke, network fail...). As we can not predict at this scale, the best approach is to design for failure and improve the resilience of our system continuously. 
  • Define the infrastructure as code to allow rapid iteration. You can learn more about this concept in this previous blog post  Immutable infrastructure (tech pill).
  • Design to take advance of the elasticity of the Cloud. This includes architect for horizontal scalability so you can scale during peak hours without paying for this capacity during low activity hours.
  • Pay per use and rent not buy. When we design for the Cloud, we should try to use the resources as much as possible and deal with the variability in the load using the elasticity. The mindset should be, rent the most suited resources for the task and release them as soon as possible.





In summary, understanding the Cloud providers APIs and services is only a small part of your journey to the Cloud. The most critical step is to change your mindset and accept new underlying assumptions.

A new engineering challenge 
Construct a highly agile and highly available service from ephemeral and assumed broken components. (Adrian Cockcroft)

Related:

Tuesday, August 07, 2018

Good talks/podcasts (Aug 2018 I)

Some good talks that I show lately:

Monday, August 06, 2018

Honey Badgers, from Team to Tribe

A team is a group of individuals working together to achieve a goal. However, a tribe (in anthropology) is a human social group. That is: a group of people who interact with one another, share similar characteristics and have a collective sense of unity, and identity.

As already explained in several previous blog posts, all the people that worked in the tech/dev department at The Motion formed a cohesive team, called the Honey Badger Team, with shared goals and a common mindset.

The Motion was a typical example of an early stage startup: the platform itself was working well, but we weren't able to reach the product market fit, at least with the initial idea. This led to, the company, the product, and the business pivoting several times in the search for monetization. And although the current model seems to work better than the initial one, the result of all these changes, was that part of the team, including myself, gradually left. Motivation for leaving was different for each engineer, but in my case, I had already fostered a great team and had helped build a scalable platform, and was now looking for a new challenge. And I found that new challenge I was looking for at Nextatil, leading the agile retail revolution, but that is another story for a different blog post.

The important thing was that The Honey Badger Team was no more, and at first, this made me sad. It was one of the cons about changing jobs.
However, (and I'm really not sure about how this happened), we seemed to have created a social group. With our mindset. With our shared history. With our way of working.
And we evolved from being a team to forming a tribe. A tribe with our own communication channels, our jokes, our standard way of solving problems. With a grown/agile mindset, aligned with the XP/DevOps values and practices.

It is incredible, but we are now spread over more than five different startups, and we still all thinking about how to spread our values, our way of thinking, our way of solving problems. Sharing knowledge, sharing values, sharing solutions.
So it's clear that instead of losing a great team, what really happened was I got a tribe instead. :)

Currently, we have two telegram channels, which we use to communicate or to obtain great solutions to any problem that we share.
Moreover, we are thinking about creating our invitational conference: the “Tejones Conf", an event for ourselves, our families and friends of us. If we aren't a tribe, I don’t know what we are.

References

The history of the Honey Badger team:

Blogs from other other Honey Badger tribe members:

Sunday, July 29, 2018

Tuesday, July 17, 2018

Agile is counterintuitive

A good Agile culture for a technology-based company is counterintuitive for most people, as we were educated to use cost accounting and resource efficiency instead of flow efficiency. Neither do we know how to collaborate and work in teams, as well as it being very common to see Engineering as a team to control: as a service provider, instead of as part of the business.


At the organization and team level, at least, the following practices are counterintuitive:
  • Increase Focus and reduce the work in progress (WIP).
  • Organize people around the work minimizing hands offs (requires generalists and collaboration), instead of in functional silos (need specialists) with central coordination.
  • Self-organized autonomous cross-functional product teams that require trust, power, and team responsibility, instead of a culture of Command and Control. 
  • The NoEstimates movement to avoid waste.
  • A continuous flow of small increments (instead of large batches of work).
  • Maximizing the amount of work not done, that can often be confused with laziness, without keeping in mind that this is an excellent strategy to avoid complexity, reduce waste and reduce maintenance costs.
  • And related to the previous one, Maximize the outcomes and minimize the outputs.



To reach technical excellence, and with the focus on software development, a lot of practices are initially counterintuitive. For example:

  • Test Driven Development (or for some people even automatic testing).
  • Pair programming. This practice is a classic one. It's very easy to measure the cost of two developers. But is very difficult to estimate the cost of a bad design, the reduction of the bus factor, the improvement in the knowledge about the code, the system, or about a technical practice. So if we try to analyze from a cost efficiency perspective in a command and control culture that doesn't understand our profession, it is nearly impossible to understand the real benefits of this practice for the product/ the team/ the business.
  • Continuous integration (the practice) that encourages trunk based development or working in short-lived branches (less than one day). 
  • Continuous Delivery or even Continuous Deployment that requires technical excellence, self-testing code, and proper tooling for recovery in case of failure.
  • Introduce operability, security, quality in the process instead of having specific teams and phases to validate these activities.
  • Optimize for fast recovery (MTTR) instead of increasing the mean time between failures (MTBF).


As we can see, in general, Agile is counterintuitive... and this is one of the reasons for the vast amount of fake agile cultures that are doing a lot of harm to a lot of companies.

In summary, agile software development is counterintuitive because it requires a culture of respect for people, collaboration and is incompatible with the classic low trust Command and Control organization. The problem is that being adaptable to changes is no longer optional...
Embrace change, Embrace failure, Embrace uncertainty...

It is not necessary to change. Survival is not mandatory. W. Edwards Deming

Resources and related posts:

Thanks for the feedback and/or for the contributions to:

Sunday, June 24, 2018

Good talks/podcasts (May 2018 II)



These are some interesting talks/podcast that I've seen/heard during this month:

Saturday, June 16, 2018

Pamplona Software Crafters 2018

This post is in Spanish because this conference and all the content was in Spanish.

Otro año más (y van tres), he tenido la gran suerte de poder asistir a la conferencia Pamplona Crafters. Esta conferencia mezcla charlas preestablecidas con open space y este año, muy acertadamente, también han incluido una serie de talleres.

Aunque en todas las conferencias a las que voy, aprendo y disfruto mucho, tengo que confesar, que en estos últimos dos años, la Pamplona Software Crafters es la que más me gusta. Supongo que es la combinación de la gente que vamos, las charlas y la tematica, con que me gusta Pamplona y por supuesto la impecable organización de los amigos de 540.



En cuanto a la conferencia en si, he asistido a las siguientes charlas/talleres:

  • Modelos mentales, complejidad y software (Arturo Herrero) (Video) Esta es una de las charlas que más me gustaron de toda la conferencia. Es un tema que me interesa, puesto que al igual que a Arturo he descubierto que el tema de interesarme por la complejidad y los modelos mentales me permite entender y relacionarme mucho mejor. Arturo ha escrito un muy interesante blogpost al respecto, incluyendo las slides que utilizo: https://arturoherrero.com/mental-models-complexity-and-software/ 
  • One Page Product Design (Jorge Barroso flipper83) Interesante charla sobre las ventajas de compactar, visualizar, gran cantidad de información mediante diagramas, screenshots, diseños en una única página/diagrama de forma que la podamos pegar por las paredaes, poner en alguna zona de paso, y la riqueza de conversaciones e impacto que esto genera… Una imagen vale más que mil palabras, y esa imagen pegada al lado de la máquina de café 
  • Sobreviviendo en producción. Construyendo una cultura de aprendizaje. (eferro) A petición de Modesto San Juan, propuse esta sesión para el Open Space. En ella, hable de como generar una cultura de aprendizaje comenzando a partir de la gestión de las inicidencias y usando como el Blameless Incident Report como medio. Esta aproximación la he usado en TheMotion y ahora en Nextail y la verdad es que ha funcionado muy bien. Tengo pendiente un blog post al respecto.
  • Taller Refactoring, aprendizaje y economía (Luis Artola, Guillermo Galmazor). Disfrute mucho en este taller, puesto que por necesidad temporal no le estoy pudiendo dedicar mucho tiempo al desarrollo, asi que cualquier oportunidad de hacer algo técnico lo disfruto muchisimo. El taller en si pretendia resolver la kata trivia de Jbrains (https://github.com/jbrains/trivia) aplicando Golden Master o Subclasses.  Como siempre con Luis y Guille, el disfrute viene de lo didacticos que son. Me gusto mucho y me quede con varias notas sobre la estrategia Golden Master para cuando tienes código legado cuyo compartamiento sólo puedes observar viendo su salida por consola.
  • El código como una forma de expresión (patxangas) Divertidisima e interesante charla sobre el uso creativo del código como medio de expresión. Muy impactante. Me trajo muchos recuerdos e ideas a la cabeza (Demoscene, videoarte, música experimental, etc). Además del contenido, siempre me encanta como se expresa y trasmite @patxangas sobre los temas más variopintos. Slides: https://slides.com/interzonas/el-cdigo-como-una-forma-de-expresin#/
  • Mide más y sufre menos (Modesto San Juan) Charla introductoria sobre la necesidad de instrumentar nuestro software de forma que dispongamos de información real para tomar decisiones. Quizás se me quedo algo corta, pero por lo que comente con Modesto, yo no era el público objetivo puesto que ya estoy convencido. En cualquier caso es importante que se mande de forma continua este mensaje ya que muchos de los desarrolladores se centran sólo en la parte funcional sin tener en cuenta que eso es sólo una parte de su trabajo.
  • Evolutionary Architecture (María Gómez) Esta charla me gusto mucho en contenido. Complementa desde un punto de vista algo más práctico a las charlas relacionadas de algunos de sus compañeros de ThoughtWorks. La charla se basa en lo descrito en el libro Building Evolutionary Architectures  (http://evolutionaryarchitecture.com/).
  • Cómo no ser un tech lead (Gamarra). Esta sesión de open space se organizó como una charla abierta en la que se comentó algunas ideas sobre cómo incentivar una a tus compañeros a formarse, crecer, etc. Cogi algunas ideas interesantes de lo que estaban haciendo en Tuenti y en Liferay, que espero poder poner en práctica.
  • ¿Arquitectura sin arquitectos? (German Del Zotto) Esta sesión planteo varias decisiones y balances que se tienen que tomar cuando estamos deciendo temas de artquitectura. German los describio como desciones a tomar, no por un arquitecto, sino como un habilidad que deberian poseer cualquier desarrollador. Me quede con ganas de más, pero seguro que puede quedar con German y profundizar en el tema. :)



Como siempre lo que más me ha gustado y lo que más me ha enriquecido han sido las grandes converaciones. Me vienen a la cabeza unas cuantas:

  • Con cataflu durante uno de los desayunos (además es una gran desvirtualizacion). 
  • Con nabaroa durante las cervezas del jueves (otra genial desvirtualización). Además hizó una impresionante lightning talk  acerca de las subidas y bajadas que experimentamos durante nuestra carrera y de cómo las ha gestionado ella.
  • Con Arturo Herrero sobre el mercado para desarrolladores de software en UK y sobre el value investing.
  • Con Helder y Marco Alacot sobre aprender/enseñar a programar, agilidad, y otras hierbas.
  • Con bberrycarmen  después de la sesión de Arturo sobre modelos  mentales. Con luisrovirosa sobre la charla que dio Martin Fowler en Madrid la semana anterior y de cómo “hemos sido engañados” con el tema de la agilidad, puesto que podemos resumirla en XP+DevOps.

Me resulto gracioso que rafaoe y algún que otro “milenial” tuviese curiosidad de cómo desarrollabamos en los 90s. :)
También disfrute de largas e interesantisimas conversaciones con jaceguzigiella, y mucha más gente…

Por supuesto disfrute como un enano con tejones como Saski, Fran Ortiz, amigotes de Alea como Apa, Ruben y Nestor y con muchisima gente más (sharpbitesalfredodev, Toño, jorgeuriarte, alfredocasado, ...)
Me quedé con pena por no haber podido hablar y disfrutar con todos, pero llega un punto en el que es prácticamente imposible.

He estado repasando la lista de asistentes (https://twitter.com/pamplonaswcraft/lists/asistentes-2018) y he estado tentado de pegarla por aquí, puesto que compartir tiempo con todos ellos ha sido un autentico placer...


Bola extra:
Este año no he podido ir con familia, lo que por un lado me da algo de pena, pero que por otro lado me ha permitido aprovechar el viaje de otra forma. Me he organizado para poder ir con Alberto Paz, Modesto San Juan y Jerolba. La verdad que ha sido como tener dos dias más de confencia…  Grandes conversaciones y mucho aprendizaje.

Foto de @benjymontoya (usada con autorización)

Muchas gracias a los organizadores (@wricka69, @pabloalbizu, @gorkma, Mikel Ros) … Sois muy Top!!!

PD: Iré actualizando el post según vaya recopilando recursos.

Saturday, June 02, 2018

Immutable infrastructure (tech pill)

Immutable infrastructure is a pattern to change/evolve processes without modifying the running code, actual configuration, and the base components (library, auxiliary processes, SO configuration, SO packages, etc).  In summary, avoid any manual or automatic configuration change to the running systems.
But if we don't allow changes, how can we evolve our application and service?  Easy, change the whole piece (function, container, machine) in one shot...



So to make any kind of change (code, configuration, SO) we don't connect to the target box to execute commands, we "cook" a new artifact (container image or machine image) and use it to create a new instance (container or machine) with the changes. In the past, the cost/time to create a new instance was huge, so in order to optimize the process, we tend to execute the minimal change needed to run the new version. I mean, use a manual or automatic process to ssh to the box, install the needed packages, change the configuration, update the code, etc.

But...
¿What happens when the ssh dies in the middle of an update?
¿And if we have a problem installing a package?
¿How can we be sure about the actual state of a machine?
¿How can we calculate the changes to execute if we are not sure about the actual state of a machine?

Making changes in a machine using ssh, is not a transactional operation so we can have a failure in the middle of the process.

The solution, Immutable Infrastructure...
Create a new artifact and run it. Without intermediate states. Only "Not Ready" or "Ready". Simple.  And if something is wrong, destroy the artifact and try again.

This pattern is at the core of the principals container-orchestration systems and PaaS (kubernetes, swarm, mesos, heroku, open shift, Deis, etc).

Why is a good idea

  • Simplicity. Is an order of magnitude easier to destroy a resource and create a new one from scratch than to calculate the deltas to apply and execute them. 
  • If we need scalability we need to support this patter anyway.
  • Right now is easy to implement with the help of the different clouds and PaaS providers.
  • Very easy to return to the previous version.
  • Very easy to troubleshoot a problem, because there are no intermediate states. You can know the exact content of a running version (the SO state, the concrete conf, the concrete code, etc).
  • With this approach, there is no difference between languages and running environment (python, jruby, java, all the same...).  Is a general solution for all the technology your systems requires.

What are the downsides

  • In some cases, the bootstrapping/setup time to use a new machine is larger than modifying an existing one. But the time is improving continuously and if we have a scalable architecture we already are dealing with it. Another solution is to use the patter at a different level, for example, functions or containers instead of at the machine level.
  • This pattern requires more and longer steps that the classic approach so it is not practical to do it without automation. But not automatize this kind of task is shooting in your foot anyway.

Implementation Samples

As a general implementation strategy we need to be capable to make the following steps:
  1. Start a new running process (without processing workload).
  2. Detect when this running process is ready to accept real workload.
  3. Connect the running process to start processing workload.
  4. Disconnect the old running process to stop receiving new workload.
  5. The old running process completes the workload that already has.
  6. Detect that all the pending workload of the old process is completed.
  7. Destroy the old process.


If we talk about web servers the steps can be:
  1. Start a new machine/container running the web server in an internal endpoint.
  2. Detect when the web server is ready using the health check endpoint.
  3. Connect the new web server internal endpoint to the load balancer with the external endpoint.
  4. Inform the old web server to stop processing new traffic and disconnect from the load balancer.
  5. Detect when the old web server finished.
  6. Destroy the old web server.
If we talk about a background queue job processor the workflow can be:
  1. Start a new machine/container running the background queue job processor.
  2. Detect that the new machine is processing jobs from the queue.
  3. Inform the old background queue job processor to not get new jobs.
  4. Detect when the old background queue job processor has completed its pending work.
  5. Destroy the old background queue job processor.
We can think similar steps for other kinds of processes.

Design Notes

As we can see, this pattern requires collaboration from our code and support from the platform, but I assume that this is part of designing scalable systems that make great use of the capabilities of the cloud. For a good reference about designing applications for the cloud, please read The 12 Factor Apps

We can apply the same pattern at different levels. Virtual machines, Containers, Functions... The ideas are the same but the granularity is different.

Using Infrastructure as Code is recommended prerequisite to implement this pattern.

Conclusions

  • The platforms are going in this direction so they have a lot of help to implement it.
  • This is the fundamental pattern for:
    • Scale up and down in the cloud.
    • Advance patterns for deploy with zero downtime (blue-green deploy, rolling deploy, canary releasing, etc).

Other tech pills:


Saturday, May 26, 2018

Infrastructure as code IaC (tech pill)

Infrastructure as code (IaC) is the practice of defining/declaring the infrastructure we need for a system using some kind of machine-readable source files. These source files are used by a tool to provision, create or maintain in a defined state our infrastructure.

These definitions help to provision/create a different kind of resources, compute, storage, communication services, network



For cloud-based infrastructures, we can use these definitions to create "virtual resources" and to configure them to be in a certain state. For example, we can create a virtual machine with an initial  OS image and later install some software and configure it.

In a bare-metal environment, we can use the definition to configure a fixed number of machines and devices already defined in an inventory.

The goals of this practice are:

  • Avoid server configuration drifting
  • Avoid proliferation of Snow flake Servers.
  • Reduce drastically the maintenance cost and the total cost of ownership.
  • Allow easy and infrastructure evolution.

As a collateral effect, this practice also allows:
  • Use development practices for the infrastructure (version control, testing, audit, collaboration, live documentation...).
  • Create on-demand systems for development, QA, testing, and experimentation.
  • Developers collaboration.

Cons:  really, it's 2018... no, seriously I don't find a good reason to not to create infrastructure as code. And when the problem is that these resources are difficult or impossible to define using code or some kind of definition, we should avoid them as much as possible.

General Approaches and styles


  • Push. We execute a tool that parses the definitions, calculate the changes to do and execute them.
  • Pull. Each node/device have its definition and execute all the time in a loop executing the needed changes.
  • Push + Pull. A combination of the previous ones, so we can push a change (when we need to be sure or force some changes) or wait until each node/device update its configuration 
Sometimes we can restrict the IaC to provision the low-level infrastructure. For example, we can use this practice to create a PaaS (using Kubernetes, or similar), so the rest of elements are dynamically provision inside the PaaS we have created.

Related Tools:


In summary, IaC is a core DevOps practice and is the base for a lot of the innovations and evolution that the cloud brings us. It is a must for modern development in the cloud and also very recommendable for on-premise deployments.



Other tech pills:



Sunday, May 20, 2018

Good talks/podcasts (May 2018 I)

These are some interesting talks/podcast that I've seen/heard during the past month:


Wednesday, May 09, 2018

DevOps talks


As a complement of the previous post DevOps concepts and learning path and in case that you prefer watching talks, these are some great talks about DevOps:



Bases for DevOps (Lean / Flow):
DevOps:
Related:

Related posts:

Monday, May 07, 2018

DevOps concepts and learning path

As Jez Humble said at leading a devops transformation DevOps is:
A cross-functional community of practice dedicated to the study of buildings, evolving and operating rapidly changing, secure, resilient system at scale. 
DevOps includes and enhance the ideas of the Agile Software Development giving a more end to end vision of the value stream for a technology-based company. And, you know, Every Company Is A Tech Company.




In summary, I think that DevOps should be the core of any technology-based company and the only option to reach/pursue the necessary business agility.


Main characteristics of DevOps culture:

  • Collaboration between development and operations (avoiding SILOs and conflicting goals).
  • Organize around the value stream optimizing for the flow efficiency (not resource efficiency). Learning to work in small batches.
  • Remove waste (Non-Value Adding Activities).  Of course, we talk about value from the customer point of view.
  • Build quality in.
  • Create fast feedback loops.
  • Maximize organizational learning (making safe to fail and learn).



Common practices:

  • Infrastructure as code.
  • Developers are involved in operations of the system (you build it, you run it).
  • Operations involved in the development from the beginning (introducing/facilitating specific, nonfunctional requirements to create a system easy to operate and monitor).
  • Automation (to avoid errors and to facilitate short iterations).
  • Continuous Delivery.

Books / Learning path:


If you are interested in learning about this culture and you like reading books, this is the learning path I recommend:

The Phoenix project: As a great and easy to read an introduction to Lean in an IT environment. (My review)

The DevOps handbook: The complement for The Phonix Project that explains step by step the typical practices and the strategy to introduce DevOps.  A short of practitioner’s guide.

Continuous Delivery: To learn the principles and technical practices that enable rapid and incremental delivery of high-quality, valuable software to our customers.

Accelerate: To understand how to build and scale high performing technology organizations creating this DevOps culture.  This book is also an analysis of the data from the State of DevOps reports that give an idea about the importance of these practices.









If you are involved in any technology company, do you a favor and learn about DevOps... As the State of DevOps report indicates is the key to be a high-performance organization.

Related posts: