Showing posts with label 3s. Show all posts
Showing posts with label 3s. Show all posts

Sunday, September 26, 2021

Code reviews (Synchronous and Asynchronous)

There are different types of Code Reviews with different objectives. This article only refers to those Code Reviews that another team member does before incorporating the change in the shared code repository.

The objective of these reviews is to improve the quality, avoid errors introduced in the shared code repository, and prevent issues from going out to production.

Therefore the code reviews we are talking about:

  • Avoid errors/bugs by acting as a safeguard.
  • Occur within the flow of making changes to production.

In this blog post, we will not discuss other types of Code Reviews that are done outside of the normal development flow.

Most common: Asynchronous code reviews.

In this context, a very common (and for some, recommended) way of performing code reviews is that a developer (A) works individually on a task (d) and upon completion of the general change, creates a Pull Request (PR)/ Merge Request (MR) that another person in the team must review asynchronously.

We can see an example of this way of working in the following diagram. In it, we can see how developer A generates PR/MRs that developer B reviews, generating proposals for changes that developer A should incorporate.

In the example, the feature/change is composed of increments d1, d2, d3, and we see that we deploy it to production at the end of the code review of increment d3.



We can see that A's development flow is constantly interrupted to wait for the corresponding feedback for each code review. This way of working delays the release to production, increasing the time it takes to receive real feedback from customers.

Because each individual code review takes so long, there is an unconscious and natural tendency for developers to work in increasing larger increments. At least, this is what I found in many teams I have worked with.

It is a curious situation because, on the one hand, the industry says that working in small safe steps is a recommended practice. On the other hand, many companies use async reviews that tend to discourage them.

So the most common way of working looks like the following diagram:


Moreover, in this case, by generating large PRs/MRs, the code review loses meaning and, in many cases, becomes a pantomime (LGTM). (See: how we write/review code in big tech companies)
With this last way of working, the lead time to production improves a little, but at the cost of taking bigger steps and losing the benefits of detailed code reviews.

When working with asynchronous code reviews, the reality is usually more complex than what we have seen in the previous diagrams. The reality usually includes many developers, simultaneous code reviews and cross flows in which there is a strong tendency for individuals to work alone and do continuous context switching.




In fact, there is no incentive for the team to focus and collaborate on a single user story at the same time since everyone would be busy with their tasks and doing asynchronous code reviews for each other.

Another problem that often occurs is that code reviews do not have sufficient priority and generate long queues of pending reviews. These queues cause the lead time to grow and create deployments with larger batches of changes (more risk).

Is there a better way?

Fortunately, there are synchronous code reviews and continuous code reviews.

Synchronous, high priority code reviews

The first step is to give the highest priority to code reviews and to do them immediately and synchronously between the person who developed the increment and the person in charge of doing the review.

This way of working reduces the total lead time of each increment and encourages working in small increments.

As a side effect, the code reviews should be more straightforward because the developer themself explains them and can express in detail why they have taken each of the decisions.


This way of working requires a lot of coordination between team members, and it is not easy to make them always synchronous, but of course, it is a simple step if we are already doing asynchronous code reviews.

It is as simple as making PRs/MRs top priority, and as soon as you have one ready, coordinate with someone else to review it together on a single computer.

Pairing/Ensemble programming: Continuous Code Review

Pair programming: "All code to be sent into production is created by two people working together at a single computer. Pair programming increases software quality without impacting time to deliver. It is counter intuitive, but 2 people working at a single computer will add as much functionality as two working separately except that it will be much higher in quality. With increased quality comes big savings later in the project.". http://www.extremeprogramming.org/rules/pair.html
 
When we work using pair programming, as recommended in extreme programming, two people design and develop the code. This way of working generates a continuous review of the code, so it no longer requires a final review phase. 

Of course, from the point of view of coordination, it is the simplest system, since once the pairs have been formed, each pair organizes itself to make increments, which are implicitly reviewed continuously.

In this case, the diagram representing this way of working would be: 



A new way of working known as mob/ensemble programming has appeared as an evolution of the pair programming practice. In this modality, the whole team works simultaneously on a single task and using a single computer. 

The whole team collaborates in the design and development, limiting the context switching and minimizing the lead time for the task. As in pair programming, mob/ensemble programming performs a continuous and synchronous review of the code.

Of the ways of working that we have seen, pair or group programming has the shortest lead time for each change, fewer context switching interruptions, and requires less coordination.

Even so, pair and group programming is not a simple practice and requires effort and a lot of practice. IMHO it is clear that this way of working takes the quality benefits usually associated with code reviews to the extreme.

The industry vision, my personal vision

The industry seems to agree that code reviews are a good practice and should be implemented by all development teams.
On the other hand, it is also recommended in most cases to work in small increments so that the complexity and risk of each change can be better controlled.

With these two points in mind, it seems a good idea to use Code Reviews and make them as synchronous and continuous as possible to work on those small increments of low complexity and low risk.

In my case, I have been successfully pushing pair or mob programming for a few years now, creating a stream of small increments that are reviewed continuously.

In both Alea Soluciones and TheMotion, we used pair programming by default and mob programming occasionally. Both companies' deployment frequency was daily, and we could make considerable changes in small and safe steps. In addition to this constant and sustainable development flow, we got attractive additional benefits such as the ease of spreading domain and technical knowledge or make fast onboarding of new members. In the case of Alea Soluciones, we hired people who already had experience in pairing and XP practices, so it was easier. In the case of TheMotion, however, we had to introduce some of the agile development practices (pairing, TDD, etc.), so we hired technical coaching help.
In Nextail there was a mix of teams doing pairing and continuous reviews and others using asynchronous code reviews but prioritizing to avoid queues.
At Clarity AI, some teams use asynchronous Code Reviews but prioritize them to minimize the lead time. There are already two teams doing continuous code reviews (using pairing or mob programming) and we are gradually expanding this way of working to other teams.

Of course, the statistical value of my experience is zero, and it is much more interesting to analyze the results of more rigorous analyses such as DORA (https://www.devops-research.com/research.html). In them, we can see how the following practices are associated with high-performance teams:

Related:

Saturday, August 07, 2021

Revisiones de código (Síncronas y Asíncronas)

Actualización 30/8/2021: 
Ampliada seccion "Visión de la industria y personal" y añadida referencias adicionales.


¿Qué es una Code Review?

Aunque existen distintos tipos y con distintos objetivos en este articulo nos vamos a referir como Code Review a aquellas revisiones de un cambio de codigo/configuracion/esquema BD que solicitamos que nos haga otro compañero antes de incorporar el cambio en el repositorio común de código. El objetivo de estás revisiones es controlar la calidad e intentar evitar que se introduccan errores en el repositorio común de código y por tanto evitar que el cambio salga a producción.

Por tanto las code review de las que hablamos:

  • Intentan evitar errores/bugs actuando como salvaguarda.
  • Se producen dentro del flujo para llevar cualquier cambio a producción.

Existen otros tipos de code review que no se hacen en el flujo normal de desarrollo y que suelen usarse para aprendizaje compartido del equipo. En este articulo no las vamos a tratar.

Lo más común: las Code reviews asíncronas

En este contexto, una forma muy común (y para algunos recomendada) de realizar las code reviews, consiste en que un desarrollador (A) trabaja de forma individual en una tarea (d) y al terminar el cambio general una Pull Request / Merge Request que otra persona del equipo debe revisar asíncronamente.

Podemos ver un ejemplo de esta forma de trabajar en el siguiente diagrama. En él se puede ver como el desarrollador A genera PR/MRs que el desarrollador B revisa, generando propuestas de cambios que el desarrollador A deberá incorporar.

En el ejemplo, el desarrollo a realizar está compuesto por los incrementos d1, d2, d3, y vemos como es desplegado a producción al finalizar el la code review del incremento d3.


Podemos ver que el flujo de desarrollo de A se interrumpe constantemente para esperar el feedback correspondiente a cada code review. Esto hace que el tiempo total que pasa hasta que hacemos una release a producción y conseguir feedback de verdad es muy largo.

Con esta forma de trabajar, es probable que inconscientemente exista la tendencia a trabajar en incrementos más grandes, para evitar la espera por la code review. De hecho, esta tendencia a hacer pasos más grandes es lo que me he encontrado en muchos equipos con los que he trabajado. Es una situación curiosa, por que por un lado se transmite la necesidad de trabajar en pasos pequeños y por otro lado se usan prácticas que tienden a desincentivarlos.

Por lo que la forma de trabajo más común se parece al siguiente diagrama:


Además, en este caso, al generar PRs/MRs grandes, la code review, pierde sentido y en muchos casos se convierte en una pantomima (LGTM). (Ver: how we write/review code in big tech companies)

Con esta última forma de trabajar se mejora un poco el lead time hasta poner en producción, pero a costa de dar pasos más grandes, y de perder los beneficios de las code reviews serias.

Cuando se trabaja con code reviews asíncronas la realidad no suele ser tan sencilla como lo que hemos visto en los diagramas. La realidad suele incluir muchos flujos simultáneos y cruzados en el que existe una tendencia fuerte al trabajo individual y al cambio de contexto continuo.

De hecho, no se genera ningún incentivo para que el equipo trabaje con foco en una sola historia de usuario al mismo tiempo, puesto que cada uno estaría ocupado con sus propias tareas y haciendo code reviews asíncronas para los demás.

Otro problema que suele suceder es que las code reviews no tengan suficiente prioridad y se generen colas. Esto hace que el lead time crezca y que se desplieguen a producción cambios más grandes con el incremento de riesgos que ello conlleva.

¿No hay otras opciones?

Por suerte sí que las hay, las code reviews síncronas y las code reviews continuas.

Code review síncronas y con prioridad

El primer paso puede ser dar la máxima prioridad a las code reviews y hacerlas de forma inmediata y síncrona, entre el que ha desarrollado el incremento y el encargado de hacer la review.

Con esta forma de trabajar se consigue que el lead time total de cada incremento sea menor y se incentiva trabajar en pasos pequeños.

En este caso, como efecto colateral, las code reviews deberían ser más fáciles, puesto que la explica el propio desarrollador y puede expresar con detalle por qué ha tomado cada una de las decisiones.

Esta forma de trabajar requiere mucha coordinación entre la los miembros del equipo y no es sencillo hacerlas siempre síncronas, pero desde luego es un paso sencillo si ya estamos realizando code reviews asíncronas.

Realmente es tan simple como que las PRs/MRs tengan la máxima prioridad y que en cuanto tengas una preparada, te coordines con alguien para revisarla los dos juntos en un único ordenador.

Pairing/Ensemble programming: Code review continua

Programación en parejas: "Todo el código que se envía a producción es creado por dos personas que trabajan juntas en un mismo ordenador. La programación en parejas aumenta la calidad del software sin que ello afecte al tiempo de entrega. Es contraintuitivo, pero dos personas trabajando en un mismo ordenador añadirán tanta funcionalidad como dos trabajando por separado, salvo que será de mucha mayor calidad. El aumento de la calidad supone un gran ahorro en la fase posterior del proyecto". http://www.extremeprogramming.org/rules/pair.html 


Cuando trabajamos usando programación en parejas tal y como se recomienda en programación extrema, el código se diseña y desarrolla de forma conjunta por dos personas. Esta forma de trabajar genera una revisión constante del código por lo que no requiere una fase final de revisión. 

Desde luego, desde el punto de vista de coordinación, es el sistema más sencillo, puesto que una vez que se han formado las parejas, cada pareja se organiza para ir haciendo incrementos, que de forma implícita van siendo revisados de forma continua.

En este caso, el diagrama que representa esta forma de trabajo sería: 


A partir de la programación en parejas y viendo sus beneficios, ha aparecido una nueva forma de trabajar conocida como programación en grupo (mob/ensemble programming). En esta modalidad todo el equipo trabaja al mismo tiempo en un único cambio y usando un único ordenador. De esta forma todo el equipo colabora en el diseño de cada pieza de código limitando al máximo los cambios de contexto y minimizando el lead time de cada uno de los cambios. Al igual que en la programación en parejas, la programación en grupo realiza una revisión continua y sincrona del código.

De las formas de trabajar que hemos visto, la programación en pareja o en grupo, es la que tiene un menor lead time para cada cambio, menos cambios de contexto e interrupciones y requiere menos coordinación.

Aun así la programación en parejas y en grupo no es una práctica sencilla y requiere esfuerzo y mucha práctica. Lo que sí tengo claro es que lleva al extremo los beneficios de calidad que se suelen asociar con las revisiones de código.

Visión de la industria y personal

Si en una cosa parece estar de acuerdo la industria es que hacer code reviews es una buena práctica y que deberían hacer todos los equipos de desarrollo. Por otro lado también se recomienda en la mayoría de los casos trabajar en incrementos pequeños, de forma que se pueda controlar mejor la complejidad y el riesgo de cada uno de los cambios.

Teniendo en cuenta estos dos puntos, parece una buena idea usar Code Reviews y que estás sean lo más sincronas y continuas posible, de forma que podamos trabajar en esos incrementos pequeños de baja complejidad y bajo riesgo.

En mi caso, llevo unos cuantos año empujando la programación en parejas o en grupo, creando un flujo de incrementos pequeños que se revisan de forma continua.

Tanto en Alea Soluciones como en TheMotion, utilizábamos programación en parejas por defecto y programación en grupo ocasionalmente. La frecuencia de despliegue de ambas empresas era diaria, y podíamos hacer cambios considerables en pasos pequeños y seguros (3s). Además de este flujo de desarrollo constante y sostenible, obtuvimos atractivas ventajas adicionales, como la facilidad para difundir el dominio y el conocimiento técnico o hacer una rápida incorporación de nuevos miembros. En el caso de Alea Soluciones, contratamos a personas que ya tenían experician en programación en parejas y en prácticas XP, por lo que fue más fácil. Sin embargo, en el caso de TheMotion, tuvimos que introducir las prácticas XP (TDD, programación en parejas, etc.), por lo que contratamos ayuda de coaching técnico.

En Nextail había una mezcla de equipos que hacían programación en pareja y revisión continua y otros que utilizaban revisiones de código asíncronas pero las priorizando para evitar las colas.

En Clarity AI, algunos equipos utilizan revisiones de código asíncronas pero las priorizan para minimizar el lead time. Ya hay dos equipos que hacen revisiones de código continuas (utilizando la programación en parejas o en grupo) y estamos ampliando gradualmente esta forma de trabajar a otros equipos.

Por supuesto el valor estadistico de mi experiencia es nulo y es mucho más interesante analizar los resultados de analisis más serios como el DORA (https://www.devops-research.com/research.html). En ellos podremos ver como se asocian a equipos de alto rendimiento las siguientes prácticas:


Relacionado:

Monday, May 03, 2021

API contract & Expand. (Small Safe Steps)

In the latest Toughtworks tech radar (Vol 24 April 2021), API Contract & Expand appears as a technique to be adopted. I couldn't agree more.
We often use complex versioning mechanisms even if a more straightforward API expand and contract strategy works perfectly for most cases.

In general, API expand and contract should be enough for internal APIs. For external APIs, expand and contract should work for small changes, and API versioning is a better option for more significant changes.

If you want to practice this and other strategies to reduce risk and slice technical changes, please check out this Small Safe Steps workshop.

References:

Wednesday, March 03, 2021

Talk: (3s) Small Safe Steps: The secret to agility



Ayer tuve el gran honor de compartir cartel con dos grandisimas profesionales, María Berenguer y Angélica Lozano en el Software Development Summit organizado  por el grupo Voxel y presentar la charla  "(3s) Small Safe Steps: The secret to agility".
 
Muchas gracias a @vgaltes por la invitación. 
 

Video

 

 

Resumen Visual

Resumen Visual by @Aitortxu

Slides

He añadido a la presentación una slide inicial para explicar el contexto que aplica en la misma y que uso para los ejemplos. En la presentación hablamos sobre todo de entornos en los que es posible hacer ciclos de feedback cortos.


Documento Original (Con Notas)


Referencias:


Thursday, January 07, 2021

Small batches for the win / Continuous Delivery

In software product development the batch size deeply affects flow efficiency and resource utilization. In this post, we will try to explain the underlying principles that make a good strategy to use small batches when we are developing software.

Let’s start with some basic concepts:

  • Batch: A group of items that move together to the next step in the process. In this case, is a group of changes that are deployed to production at the same time.
  • Item/change: Each one of the individual units of work composes a Batch. In our case any kind of change that affects our system, including new features, code improvement, configuration changes, bug fixes, and experiments.
  • Holding cost: The sum of the cost associated with delaying the deployment of each one of the items. In summary, the cost of delaying the feedback or the value delivered by each item. For example the cost of delay of a new feature or the cost associated with not putting in production a bugfix, etc.
  • Transaction cost: the cost associated with the deployment. The cost of executing a deployment (cost of people, cost of infrastructure, etc).


Batch size / Direct cost

If we only take into account the transaction cost the optimal solution is to make huge batches, as we only pay the transaction cost when we deploy. For example, deploying once a year.

If we only take into account the holding cost, the optimal solution is to use batches with only one item, to avoid delaying any kind of holding cost.

The reality is that if we try to optimize the two variables at the same time we have a U-Curve optimization problem.


U-curve graph




We can see that from the optimal batch size the total cost only grows and that before the optimal batch size we have a penalty due to our transaction cost. So a good strategy is always to minimize the batch size until the transaction cost makes smaller batches inefficient.

Batch size / Risk management


When we develop software each component is coupled with other components of the system. That is, that any part of the code has relations with other parts, such as static or runtime dependencies, common messages, data model dependencies, etc. If we invest in  good internal quality, we will minimize the unneeded coupling, but in the worse scenario, each part of the system can be potentially coupled with another part.

When we create a batch with a size of N changes to be deployed at the same time the potential interactions that can happen are:

  • Each change with the current version of the system. N
  • Each change with the rest of the changes in the batch. That is all the 1-to-1 relations between each change. (N*(N-1))/2


Potential Interactions (I)
Batch size (N)

I = N + (N*(N-1))/2

Whereas this formula describes the number of potential interactions, in general not all of those combinations are possible.

Iterations graph




The basic problem with the batch size for software development is that the universe of potential interactions (I) grows very fast. In fact, it is a quadratic function.

We can quickly conclude that the following problems grow depending on the size of the universe (quadratic):

  • The probability of an error or a negative impact (functional, performance, cost, etc).
  • The cost of detecting/troubleshooting an error.


At the same time we can see that the size of the batch (the number of changes) affects (linearly) the following:

  • The number of teams/people required to coordinate/synchronize the deployment (code freeze, communication, testing, etc).
  • The possibility of having a change difficult to revert (data model changes, high volume migrations, etc).

 
Let's illustrate how fast the problem grows with an example
If we have an error in 1 of 100 interactions we can see how fast grows the possibility of having an error:



Probability error. Batch size 5

Probability error. Batch size 25

Probability error. Batch size 50

With 25 changes, we already have 88% chance of having at least an error, and with 50 is near sure (99%). And we previously saw that these errors are more difficult to diagnose, and have more possibilities of not being easy to revert.

So clearly, for software development, increasing the size of the deployments (batch size) greatly increases (much more than linearly) the risk associated (risk of having an outage, loose availability, and frustrating our customers).

Batch size / Indirect cost and consequences

In the previous sections, we have seen the direct impact of the batch size on a product development process;
  • An increase in the total cost for a batch size greater than the optimal size.
  • A near quadratic growth of the risk associated with a deployment. 
  • An increasing number of production outages.

In addition to those direct costs, these are other indirect costs and consequences when batch size is large:
  • Lots of multitasking and the corresponding productivity and focus lost. The normal flow of work is frequently interrupted with problems that come from previous deployments.
  • A tendency to disconnect from the operation and impact of our changes in production. i.e.: when deploying something that you did several weeks ago.
  • Low psychological safety, because of the amount of risk and the probability of outages associated with the way of working.
  • Worse product decisions because there are fewer options to get fast feedback or to design new experiments to get more feedback.
  • Lack of ownership is a derived consequence of the previous points.

Conclusions

As we have shown, the size of the batch has many effects:
  • Risk and outage probability are proportional (and worse than linear) to the number of changes included in a deployment.
  • Our batch size should be as small as our transaction cost (deployment cost) allow us.
  • Large batches generate important indirect costs and consequences (lack of ownership, multitasking, low psychological safety, etc).

Consequently, we should invest as much as possible to try to work in small batches. If we detect that we already reached the optimal batch size for our current transaction cost but we are still having the same problems that we had before, perhaps the next step is to invest heavily in lowering the transaction cost (deployment automation, independent deployments per team, deployment time, etc).

In short:

Small batches -> faster feedback
Small batches -> Customer Value sooner
Small batches -> Reduce direct cost
Small batches -> Reduce deployment risk
Small batches -> Reduce errors
Small batches -> Reduce mean time to recover
Small batches -> Improve psychological safety


Small batches for the win!

The good news is that there is already an engineering capability focused on working in small batches. Continuous Delivery!

"The goal of continuous delivery is to make it safe and economic to work in small batches. This in turn leads to shorter lead times, higher quality, and lower costs." 

The importance of working in small batches has been validated statistically in the studies conducted by DevOps Research and Assessment (DORA) since 2014. You can see the details of these studies in the book Accelerate (Nicole Forsgren, Jez Humble, Gene Kim).


References:

Sunday, November 08, 2020

Small Safe Steps workshop

 


 

If your team wants to have better strategies to manage large and risky changes, improve their slicing skills, or practice techniques as parallel changes, branch by abstraction, look at this Small Safe Steps workshop.

I have prepared information so that anyone can easily facilitate the workshop.

Please use the material, run the workshop, give me improvement feedback.  And if you need advice on running or adapting it, contact me, and I will be happy to help.

 

Small Safe Step workshop

 

Additional references:

Recommended readings:


 
 


Sunday, July 12, 2020

Continuous learning. What / How


“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
“Working software is the primary measure of progress.”

But making software is complex and requires a good understanding of the customer's needs, the market (WHAT), and the reactions of the system we are developing (HOW). Let's remember that current software systems are complex systems, and that implies that they have emerging behaviors that we don't know when we create them.
“The main idea behind complex systems is that the ensemble behaves in way not predicted by the components. The interactions matter more than the nature of the units. ... This is called an “emergent” property of the whole, by which parts and whole differ because what matters is the interactions between such parts. And interactions can obey very simple rules.” 
Nassim Nicholas Taleb

Therefore the bottleneck to create software that meets customer needs is continuous learning. But learning is useless if we bring the system down or piss off our customers.

In a software system each of the deployments/changes we make, allows us to learn and make decisions for the next steps. Therefore we are interested in shortening the learning and decision cycles as much as possible.

What is Agile delivery's strategy for doing so?

Agile Delivery's strategy is to deliver increments in small safe steps (3S), each of which must attempt to generate learning about the WHAT (user needs) or the HOW (technology/system).

Learning WHAT / User needs / Valuable software

If we have a lean product development process, we will try to validate and learn this aspect by generating as little software as possible. This implies generating hypotheses and trying to achieve validated learning using everything we have at our disposal: data analysis, prototypes with or without code, or software increments. For the latter, we will have to divide using Vertical Slicing in increments so small as possible.
Of course, to learn about user behavior, it is also essential that we instrument the product to learn continuously from the use that the customer makes of the system.

Learning HOW / Technology/System / Working software

Software is a complex system. This means that we can't foresee the final behavior when we develop each of its parts independently. That is why it is essential to make small safe changes (technical Slicing) that allow us to learn from the behavior of the system in production without affecting the user. Of course, in order to get these learnings, we need to decouple the deployments (putting into production a change) from the releases (activating a change for the end-user).


Summary

In summary, the sure way to achieve this continuous learning of the WHAT and HOW that agile delivery offers is the Continuous Delivery of Small Safe Steps (both purely technical increases and increases in value for the user).
It seems that the data confirm that this is the right approach for making digital products.



References:

Small Safe Steps

Vertical Slicing

Continuous Delivery