Be humble, no Rockstars allowed

Continuous delivery is based on the idea that when dealing with uncertainty and complex problems (as described in the Cynefin framework), the best approach is to assume we don’t already know the solution. Instead, we must commit to continuously learning about both the problem and the optimal solution. Moreover, continuous delivery encourages us to implement systems that effectively manage errors—because we are bound to make mistakes—while minimizing their impact.

If we assume that we do know the solution, or if the problem is straightforward (or just "complicated" within the Cynefin framework), continuous delivery and iterative learning might not be the most efficient approach. In such cases, a direct solution delivered by a team with prior experience can work better. However, in this scenario, don’t expect to gain new insights or improve the process further. And if the problem has already been solved before, reproducing the solution could simply be wasteful—something we should avoid at all costs.

As Kent Beck aptly put it:

"The only way it's all going to go according to plan is if you don't learn anything."

My experience and context

In my case:

• I assume that the solutions I come up with are rarely the best ones.
• I acknowledge that we (myself included) make frequent mistakes—even with something as simple as writing "Hello, World" correctly on the first try.
• I recognize that we still have a lot to learn—about the domain, the business, tools, needs, and everything else.
• I have never worked in a simple or purely complicated context where a ready-made solution was available. And if I ever did encounter such a case, I would avoid reinventing the wheel to minimize waste.

Within this context, the discipline of lean product development, coupled with practices like continuous delivery (focusing on continuous flow), continuous learning, and sustainable quality, remains the best strategy I know.

For Self-Sufficient Experts

If you consider yourself an expert in everything, someone who rarely makes mistakes and feels like a 'rockstar' in their field, you might believe that you don’t need continuous delivery. I suspect that humility and the recognition that EVERYONE (including yourself) makes mistakes might be lacking in this case. Perhaps I’m wrong, and you’re some mythological being who never errs and doesn’t need to learn continuously. If that’s the case, my apologies… :)

However, I won’t be able to work with you since I do make mistakes and need to be part of a team that uses techniques enabling us to develop products despite our errors, all without living in constant stress.

In summary

I believe the general trick of this profession is to work as a team in the best way we know how, using the best techniques we have, and still assuming that:

• We are going to make mistakes.
• We have to keep learning.
• We need to improve continuously.
• We need a level of quality that allows us to work sustainably.

The discipline of continuous delivery (CD, TBD, TDD, etc.) provides all of that for me.

Please...

• Be a great team player. Collaborate and get involved.
• Create a safe learning environment for your team.
• Program as well as you can but assume you will fail, so build an environment where failing is manageable and has minimal impact on customers.
• Make the best architectural decisions you can, but assume they will need to evolve and improve over time.
• Seek continuous delivery as a goal—it fosters the right discipline and creates a safe, sustainable work environment.
• Be humble and accept failure, uncertainty, and your own limitations.

I don’t want to work with Rockstars, nor with x10 developers… I want to work in x10 teams and environments where learning is constant, questioning is welcomed, errors are accepted (because they are low-cost), and improvement is continuous.

Remember:

We need professionals, not heroes.

“I’m not a great programmer; I’m just a good programmer with great habits.” — Kent Beck

Harnessing Efficiency: The Theory of Constraints in Software Development

This micro post was previously published at linkedin

Navigating the multi-faceted domain of software development often presents a series of bottlenecks that could hinder project momentum and delivery timelines. The Theory of Constraints (TOC) serves as a beacon, guiding teams to identify, address, and overcome these bottlenecks, thereby unlocking a pathway to streamlined processes and enhanced productivity.

Here's a snapshot of how TOC unfolds in software development:

1.  Identify the most important Constraint: Pinpoint the process, resource, or technology bottleneck obstructing progress (lack of quality, knowledge silos, convoluted deployment process, individual ownership about parts of the code or processes, etc.).
2. Exploit the Constraint: Maximize the efficiency of the identified constraint without additional resources (reducing the WIP, redirecting people to help with the bottleneck, etc.).
3. Subordinate Everything Else to the Constraint: Ensure all other processes are aligned to support and work around the constraint.
4. Elevate the Constraint: Allocate necessary resources to alleviate or eliminate the constraint, promoting better throughput (investing in automation, test automation, pair/ensemble programming to spread knowledge, feature toggles to reduce risk, etc.).
5. Repeat the Process: Embark on a cycle of continuous identification and resolution of constraints to foster a culture of ongoing improvement.

For a deeper dive into TOC and its application in IT landscapes, 'The Goal' by Eliyahu M. Goldratt and 'The Phoenix Project' by Gene Kim, Kevin Behr, and George. Spafford are essential reads.

Using the Theory of Constraints (TOC) can help software development professionals and teams deal with problems smartly. It helps turn these problems into opportunities for improvement, growth and faster delivery.

Indeed, the Theory of Constraints relates to Lean Software Development and agile software development methodologies.

The lightning talk "Stop Starting and Start Finishing" by Jason Yip is one of the best explanations I know for understanding how to use TOC and other Lean ideas to optimize the sustained delivery flow of a software development team.

#TheoryOfConstraints #SoftwareDevelopment #ContinuousImprovement #TheGoal #LeanSoftwareDevelopment

The virtuous loop of software development

This micro post was previously published at linkedin

The ultimate aim is Continuous Delivery (CD), a goal that enables fast flow with rapid iterations and continuous feedback. At the same time, this goal promotes technical excellence and good design.

Continuous Integration (CI) is required (integrating at least once a day, also known as Trunk Based Development), along with a strong focus on Test-Driven Development (TDD) or other similar practices to ensure high confidence and emphasis on excellent and simple design. This approach is closely linked to Extreme Programming and the DevOps mindset, which emphasizes collaboration and continuous improvement. By following these principles, software development teams can enhance their efficiency and deliver high-quality products to customers.

Here are some related resources that you might find interesting:

• Continuous Delivery book by Jez Humble and Dave Farley
• Extreme Programming Explained book by Kent Beck
• Continuous Delivery YouTube channel by Dave Farley
• Lean Software Development by Mary and Tom Poppendieck
• Many More Much Smaller Steps (MMMSS) by GeePaw Hill
• MMMSS – The Intrinsic Benefit of Steps by GeePaw Hill 
• Minimum Viable CD 

Why Lean Product Development is Essential for Effective Product Teams

 In today's fast-paced and competitive business landscape, it is crucial for companies to have effective product teams that can deliver high-impact solutions. However, many product teams often struggle to achieve this goal due to various challenges and constraints. In this blog post, we will explore the importance of lean product development from the perspective of engineering and how it can enable effective product teams. We will also discuss the role of system architecture in facilitating product delivery&discovery and highlight some common pitfalls that hinder the success of product teams.

The Risks of Product Development

Before diving into the details, let's first understand the risks associated with product development. According to a blog post by Marty Cagan, there are four major risks: value, usability, feasibility, and business viability.

• Value: Will customers buy or use our software? This risk pertains to whether the customer sees value in the product and is willing to adopt it.
• Usability: Can users figure out how to use the software? This risk focuses on the user experience and whether the software is intuitive and easy to use.
• Feasibility: Can our engineers build the software within the given constraints? This risk considers the technical feasibility of implementing the desired features.
• Business Viability: Does the solution align with the overall business strategy and goals? This risk examines whether the solution is viable from a business perspective.

It is important to address these risks to ensure the success of a product. However, many product teams often overlook or underestimate these risks, leading to failures and wasted resources.

Learning from Industry Leaders

To emphasize the significance of lean product development, let's take a look at some insights from industry leaders. Microsoft conducted research on their own product development process and found that only one-third of their ideas actually improved metrics. Similarly, Amazon's success rate was below half of their ideas. These findings highlight the importance of validating ideas and the need to filter out ineffective ones.

It is crucial to recognize that even highly successful companies like Microsoft and Amazon face challenges in generating impactful ideas. Therefore, assuming that all ideas are great without proper validation can be detrimental to a product team's success.

Not only are many of the ideas not positive, but also the ones that are, require several iterations (with real feedback) to begin generating positive impact.

The Vicious Cycle of Software Development

To understand why lean product development is essential, we need to examine the vicious cycle of software development. In this cycle, a product team has a fixed capacity to develop software. While some features may generate positive impact, others may go unused or fail to meet user expectations. Additionally, the software incurs maintenance costs, such as debugging, monitoring, and infrastructure support (See: Basal Cost of Software).

Over time, the accumulation of software and its associated maintenance costs can deplete the team's capacity for innovation. This cycle hinders the team's ability to focus on high-impact initiatives and leads to a decrease in overall productivity.

Furthermore, the cost of maintaining software is not linear with its size. Doubling the size of an application does not double the maintenance cost; it increases it even more. This exponential increase in maintenance costs further limits the team's capacity for innovation.

Viewing Software as a Means to an End

To break free from the vicious cycle of software development, it is crucial to view software as a means to an end rather than the end itself. Software should be seen as a liability and an expensive asset that should be minimized. The focus should be on maximizing outcomes, impact, and return on investment while minimizing functionality, requirements, and maintenance costs.

By adopting this mindset, product teams can prioritize high-impact initiatives and avoid wasting resources on unnecessary features. This shift in perspective enables teams to make better decisions and deliver more value to customers.

This way of thinking also helps protect against the Feature creep that many products suffer from.

Effective Product Teams and Lean Product Development

To achieve high performance, product teams need to embrace effective lean product practices. These practices involve building the right thing and making the thing right.

Building the right thing involves minimizing the amount of software while maximizing the impact. It requires filtering out unnecessary ideas and focusing on initiatives that align with the overall business strategy. This approach ensures that the team's efforts are directed towards high-impact solutions.

Making the thing right involves continuous learning, hypothesis experiments, and user stories. It requires a collaborative team effort, with everyone involved in the product discovery&delivery process. By working together, teams can validate assumptions, reduce risks, and deliver elegant and impactful solutions.

This process of continuous learning, filtering ideas, continuously experimenting, and delivering in small, safe steps that allow us to learn (through feedback), minimizing waste as much as possible, is what we would call Lean Product Development.

The Role of System Architecture

System architecture plays a crucial role in facilitating lean product delivery. It should be designed to optimize learning, experimentation, and evolution. Here are some key considerations for system architecture:

• Decoupling Release and Deployment: To enable business-driven decisions, release should be decoupled from deployment. Feature flags can be used to control the release of new features and enable experimentation. This ability to release to a group of users independently enables the team to progressively validate hypotheses with different customer groups, thus managing the risk assumed in each case.
• Product Instrumentation: The system should be instrumented to gather relevant metrics and operational data. This data provides valuable insights into user behavior and helps in making informed decisions. Assuming that a team can be autonomous and make good decisions when flying blind and without instruments makes no sense. Product-level instrumentation and observability are not a luxury, but an essential part of any effective product team.
• Operational Data Exploration: It is essential to have systems in place to explore operational data effectively. This ensures that the collected data is utilized to its full potential and helps in identifying patterns and trends. This point complements the previous one; having good instrumentation is useless if it is not easy to exploit for making product decisions or learning about our users' behavior.
• Safe-to-Learn Environment: The system should provide a safe environment for experimentation and learning. A blameless culture should be fostered, where failures are seen as opportunities for improvement rather than as a cause for blame.
• Modular/Decoupled Architecture: A modular architecture enables multiple product teams to work autonomously on different parts of the system. It reduces the blast radius of failures and allows for more efficient development and maintenance. This architecture should also allow the team to easily compose new applications, proof of concepts, and demos based on the available modules and components. When the design quality is good, it is much easier to reuse components for experiments and as a basis for new use cases.

By incorporating these architectural considerations, product teams can create an environment that enables continuous evolution, learning, and experimentation. Ultimately, it all comes down to having an adaptable system that we can trust and that is optimized for making small, low-risk changes at high speed.

Depending on the type of system, we can even include architecture decisions that specifically help us experiment and learn more quickly. For example, if it is a product that can be considered a platform, having all capabilities and use cases accessible through APIs can allow us to experiment and create prototypes very quickly using those endpoints from Low Code or No Code systems.

Overcoming Challenges and Maximizing Outcomes

To build effective product teams, it is essential to address common challenges and pitfalls. Here are some key points to consider:

• Autonomy and Decision-Making: Product teams should have the autonomy to make decisions and generate ideas. Stakeholders should trust the expertise of engineers and involve them in the product discovery process.
• Technical Debt and Stability: Addressing technical debt and ensuring system stability are crucial for successful product delivery. Without a stable foundation, teams will struggle to innovate and deliver high-impact solutions.
• Focus on Outcomes: Engineers should shift their focus from technology to outcomes. The value and impact delivered to customers should be the primary concern, rather than the features or technology used.
• Continuous Learning and Improvement: Product teams should embrace a culture of continuous learning and improvement. This involves validating assumptions, running experiments, and constantly seeking feedback from customers and the system.
• Collaboration and Ownership: Effective product teams require collaboration and shared ownership. Product managers, designers, and engineers should work together as a cohesive unit, leveraging their respective expertise to deliver high-impact solutions.
• Praise Continuous Impact and Learning: To generate the appropriate dynamics within the team, we must celebrate the impacts we achieve (changes in user behavior, metric improvements, cost reductions, etc.) and not so much the functionalities we make available (which would only be the means). It is also essential that we celebrate the learnings since they allow us to make better decisions and increase future impact.

Conclusion

In conclusion, lean product development is a critical component of effective product teams. By prioritizing product discovery&delivery and incorporating it into system architecture and practices, teams can maximize outcomes, minimize waste, and deliver high-impact solutions. It is essential for engineers to embrace a product mindset and actively participate in all aspect of the product development process (discovery, delivery). By doing so, they can contribute to innovation and drive the success of their product teams.

If you want to learn more about product discovery&delivery and effective product teams, I recommend reading the following books:

• "Accelerate: The Science of Lean Software and DevOps" by Nicole Forsgren, Jez Humble, and Gene Kim
• "The Principles of Product Development Flow: Second Generation Lean Product Development" by Donald G. Reinertsen
• "The Lean Startup" by Eric Ries
• "The Lean Mindset: Ask the Right Questions" by Mary&Tom Poppendieck
• "Escaping the Build Trap" by Melissa Perri
• "Inspired: How to Create Tech Products Customers Love" by Marty Cagan

Or if you prefer talks, I recommend these ones:

• Competing On The Basis Of Speed by Mary Poppendieck 
• Keynote: Friction by Mary Poppendieck 
• Agile Adria 2013 keynote: Make Impacts Not Software by Gojko Adzic 
• Beyond Features: rethinking agile software delivery by Dan North 
• Creating Value and Flow in Product Development by John Cutler 

Remember, product development and continuous delivery go hand in hand. To achieve success, product teams must embrace both and strive for continuous learning and improvement.

References and related content

• https://www.svpg.com/empowered-product-teams/
• https://www.eferro.net/2020/06/technology-at-core-of-product-discovery.html
• https://www.eferro.net/2022/08/software-development-art-of-postponing.html
• https://productdeveloper.net/from-software-engineer-to-product-engineer/
• https://productdeveloper.net/iterative-incremental-development/
• https://www.eferro.net/2021/02/basal-cost-of-software.html
• https://jpattonassociates.com/dont_know_what_i_want/

Code reviews (Synchronous and Asynchronous)

There are different types of Code Reviews with different objectives. This article only refers to those Code Reviews that another team member does before incorporating the change in the shared code repository.

The objective of these reviews is to improve the quality, avoid errors introduced in the shared code repository, and prevent issues from going out to production.

Therefore the code reviews we are talking about:

• Avoid errors/bugs by acting as a safeguard.
• Occur within the flow of making changes to production.

In this blog post, we will not discuss other types of Code Reviews that are done outside of the normal development flow.

Most common: Asynchronous code reviews.

In this context, a very common (and for some, recommended) way of performing code reviews is that a developer (A) works individually on a task (d) and upon completion of the general change, creates a Pull Request (PR)/ Merge Request (MR) that another person in the team must review asynchronously.

We can see an example of this way of working in the following diagram. In it, we can see how developer A generates PR/MRs that developer B reviews, generating proposals for changes that developer A should incorporate.

In the example, the feature/change is composed of increments d1, d2, d3, and we see that we deploy it to production at the end of the code review of increment d3.

We can see that A's development flow is constantly interrupted to wait for the corresponding feedback for each code review. This way of working delays the release to production, increasing the time it takes to receive real feedback from customers.

Because each individual code review takes so long, there is an unconscious and natural tendency for developers to work in increasing larger increments. At least, this is what I found in many teams I have worked with.

It is a curious situation because, on the one hand, the industry says that working in small safe steps is a recommended practice. On the other hand, many companies use async reviews that tend to discourage them.

So the most common way of working looks like the following diagram:

Moreover, in this case, by generating large PRs/MRs, the code review loses meaning and, in many cases, becomes a pantomime (LGTM). (See: how we write/review code in big tech companies)

Small PRs tend to generate a lot of churn, people commenting etc. while bloated PRs just sail through -- no one wants to touch them with a 10 foot pole so they just push the Approve button.

— Alex Bunardzic, white non-colonizer (@alexbunardzic) July 21, 2021

With this last way of working, the lead time to production improves a little, but at the cost of taking bigger steps and losing the benefits of detailed code reviews.

When working with asynchronous code reviews, the reality is usually more complex than what we have seen in the previous diagrams. The reality usually includes many developers, simultaneous code reviews and cross flows in which there is a strong tendency for individuals to work alone and do continuous context switching.

In fact, there is no incentive for the team to focus and collaborate on a single user story at the same time since everyone would be busy with their tasks and doing asynchronous code reviews for each other.

The problem are not branches or PRs per se, but the async, and thus delayed code review that chockes the flow of changes through the system.#continuouscodereview

— Dragan Stepanović (@d_stepanovic) August 4, 2021

Another problem that often occurs is that code reviews do not have sufficient priority and generate long queues of pending reviews. These queues cause the lead time to grow and create deployments with larger batches of changes (more risk).

Is there a better way?

Fortunately, there are synchronous code reviews and continuous code reviews.

Synchronous, high priority code reviews

The first step is to give the highest priority to code reviews and to do them immediately and synchronously between the person who developed the increment and the person in charge of doing the review.

This way of working reduces the total lead time of each increment and encourages working in small increments.

As a side effect, the code reviews should be more straightforward because the developer themself explains them and can express in detail why they have taken each of the decisions.

This way of working requires a lot of coordination between team members, and it is not easy to make them always synchronous, but of course, it is a simple step if we are already doing asynchronous code reviews.

It is as simple as making PRs/MRs top priority, and as soon as you have one ready, coordinate with someone else to review it together on a single computer.

Pairing/Ensemble programming: Continuous Code Review

Pair programming: "All code to be sent into production is created by two people working together at a single computer. Pair programming increases software quality without impacting time to deliver. It is counter intuitive, but 2 people working at a single computer will add as much functionality as two working separately except that it will be much higher in quality. With increased quality comes big savings later in the project.". http://www.extremeprogramming.org/rules/pair.html

 

When we work using pair programming, as recommended in extreme programming, two people design and develop the code. This way of working generates a continuous review of the code, so it no longer requires a final review phase. 

Of course, from the point of view of coordination, it is the simplest system, since once the pairs have been formed, each pair organizes itself to make increments, which are implicitly reviewed continuously.

In this case, the diagram representing this way of working would be: 

A new way of working known as mob/ensemble programming has appeared as an evolution of the pair programming practice. In this modality, the whole team works simultaneously on a single task and using a single computer. 

The whole team collaborates in the design and development, limiting the context switching and minimizing the lead time for the task. As in pair programming, mob/ensemble programming performs a continuous and synchronous review of the code.

Of the ways of working that we have seen, pair or group programming has the shortest lead time for each change, fewer context switching interruptions, and requires less coordination.

Even so, pair and group programming is not a simple practice and requires effort and a lot of practice. IMHO it is clear that this way of working takes the quality benefits usually associated with code reviews to the extreme.

The industry vision, my personal vision

The industry seems to agree that code reviews are a good practice and should be implemented by all development teams.

On the other hand, it is also recommended in most cases to work in small increments so that the complexity and risk of each change can be better controlled.

With these two points in mind, it seems a good idea to use Code Reviews and make them as synchronous and continuous as possible to work on those small increments of low complexity and low risk.

In my case, I have been successfully pushing pair or mob programming for a few years now, creating a stream of small increments that are reviewed continuously.

In both Alea Soluciones and TheMotion, we used pair programming by default and mob programming occasionally. Both companies' deployment frequency was daily, and we could make considerable changes in small and safe steps. In addition to this constant and sustainable development flow, we got attractive additional benefits such as the ease of spreading domain and technical knowledge or make fast onboarding of new members. In the case of Alea Soluciones, we hired people who already had experience in pairing and XP practices, so it was easier. In the case of TheMotion, however, we had to introduce some of the agile development practices (pairing, TDD, etc.), so we hired technical coaching help.

In Nextail there was a mix of teams doing pairing and continuous reviews and others using asynchronous code reviews but prioritizing to avoid queues.

At Clarity AI, some teams use asynchronous Code Reviews but prioritize them to minimize the lead time. There are already two teams doing continuous code reviews (using pairing or mob programming) and we are gradually expanding this way of working to other teams.

Of course, the statistical value of my experience is zero, and it is much more interesting to analyze the results of more rigorous analyses such as DORA (https://www.devops-research.com/research.html). In them, we can see how the following practices are associated with high-performance teams:

• Trunk Based Development (https://cloud.google.com/architecture/devops/devops-tech-trunk-based-development), where they talk about working in small batches with synchronous code reviews. (See: https://twitter.com/jezhumble/status/1415908836439773185) 
• Continuous Integration (https://cloud.google.com/architecture/devops/devops-tech-continuous-integration) that implies integrate any code change in the main branch of code at least daily (https://services.google.com/fh/files/misc/state-of-devops-2015.pdf#page=20).  In this context, continuous and synchronous code reviews enormously facilitate the introduction of this practice.

Related:

• Those pesky pull request reviews
• Pairing Guidelines
• You Must Be CRAZY To Do Pair Programming Pair Programming
• Mob Programming and the Power of Flow • Woody Zuill • GOTO 2019  
• Dan North The Value of Flow
• DORA (devops-research)

Revisiones de código (Síncronas y Asíncronas)

Actualización 30/8/2021: 

Ampliada seccion "Visión de la industria y personal" y añadida referencias adicionales.

¿Qué es una Code Review?

Aunque existen distintos tipos y con distintos objetivos en este articulo nos vamos a referir como Code Review a aquellas revisiones de un cambio de codigo/configuracion/esquema BD que solicitamos que nos haga otro compañero antes de incorporar el cambio en el repositorio común de código. El objetivo de estás revisiones es controlar la calidad e intentar evitar que se introduccan errores en el repositorio común de código y por tanto evitar que el cambio salga a producción.

Por tanto las code review de las que hablamos:

• Intentan evitar errores/bugs actuando como salvaguarda.
• Se producen dentro del flujo para llevar cualquier cambio a producción.

Existen otros tipos de code review que no se hacen en el flujo normal de desarrollo y que suelen usarse para aprendizaje compartido del equipo. En este articulo no las vamos a tratar.

Lo más común: las Code reviews asíncronas

En este contexto, una forma muy común (y para algunos recomendada) de realizar las code reviews, consiste en que un desarrollador (A) trabaja de forma individual en una tarea (d) y al terminar el cambio general una Pull Request / Merge Request que otra persona del equipo debe revisar asíncronamente.

Podemos ver un ejemplo de esta forma de trabajar en el siguiente diagrama. En él se puede ver como el desarrollador A genera PR/MRs que el desarrollador B revisa, generando propuestas de cambios que el desarrollador A deberá incorporar.

En el ejemplo, el desarrollo a realizar está compuesto por los incrementos d1, d2, d3, y vemos como es desplegado a producción al finalizar el la code review del incremento d3.

Podemos ver que el flujo de desarrollo de A se interrumpe constantemente para esperar el feedback correspondiente a cada code review. Esto hace que el tiempo total que pasa hasta que hacemos una release a producción y conseguir feedback de verdad es muy largo.

Con esta forma de trabajar, es probable que inconscientemente exista la tendencia a trabajar en incrementos más grandes, para evitar la espera por la code review. De hecho, esta tendencia a hacer pasos más grandes es lo que me he encontrado en muchos equipos con los que he trabajado. Es una situación curiosa, por que por un lado se transmite la necesidad de trabajar en pasos pequeños y por otro lado se usan prácticas que tienden a desincentivarlos.

Por lo que la forma de trabajo más común se parece al siguiente diagrama:

Además, en este caso, al generar PRs/MRs grandes, la code review, pierde sentido y en muchos casos se convierte en una pantomima (LGTM). (Ver: how we write/review code in big tech companies)

Small PRs tend to generate a lot of churn, people commenting etc. while bloated PRs just sail through -- no one wants to touch them with a 10 foot pole so they just push the Approve button.

— Alex Bunardzic (@alexbunardzic) July 21, 2021

Con esta última forma de trabajar se mejora un poco el lead time hasta poner en producción, pero a costa de dar pasos más grandes, y de perder los beneficios de las code reviews serias.

Cuando se trabaja con code reviews asíncronas la realidad no suele ser tan sencilla como lo que hemos visto en los diagramas. La realidad suele incluir muchos flujos simultáneos y cruzados en el que existe una tendencia fuerte al trabajo individual y al cambio de contexto continuo.

De hecho, no se genera ningún incentivo para que el equipo trabaje con foco en una sola historia de usuario al mismo tiempo, puesto que cada uno estaría ocupado con sus propias tareas y haciendo code reviews asíncronas para los demás.

The problem are not branches or PRs per se, but the async, and thus delayed code review that chockes the flow of changes through the system.#continuouscodereview

— Dragan Stepanović (@d_stepanovic) August 4, 2021

Otro problema que suele suceder es que las code reviews no tengan suficiente prioridad y se generen colas. Esto hace que el lead time crezca y que se desplieguen a producción cambios más grandes con el incremento de riesgos que ello conlleva.

¿No hay otras opciones?

Por suerte sí que las hay, las code reviews síncronas y las code reviews continuas.

Code review síncronas y con prioridad

El primer paso puede ser dar la máxima prioridad a las code reviews y hacerlas de forma inmediata y síncrona, entre el que ha desarrollado el incremento y el encargado de hacer la review.

Con esta forma de trabajar se consigue que el lead time total de cada incremento sea menor y se incentiva trabajar en pasos pequeños.

En este caso, como efecto colateral, las code reviews deberían ser más fáciles, puesto que la explica el propio desarrollador y puede expresar con detalle por qué ha tomado cada una de las decisiones.

Esta forma de trabajar requiere mucha coordinación entre la los miembros del equipo y no es sencillo hacerlas siempre síncronas, pero desde luego es un paso sencillo si ya estamos realizando code reviews asíncronas.

Realmente es tan simple como que las PRs/MRs tengan la máxima prioridad y que en cuanto tengas una preparada, te coordines con alguien para revisarla los dos juntos en un único ordenador.

Pairing/Ensemble programming: Code review continua

Programación en parejas: "Todo el código que se envía a producción es creado por dos personas que trabajan juntas en un mismo ordenador. La programación en parejas aumenta la calidad del software sin que ello afecte al tiempo de entrega. Es contraintuitivo, pero dos personas trabajando en un mismo ordenador añadirán tanta funcionalidad como dos trabajando por separado, salvo que será de mucha mayor calidad. El aumento de la calidad supone un gran ahorro en la fase posterior del proyecto". http://www.extremeprogramming.org/rules/pair.html 

Cuando trabajamos usando programación en parejas tal y como se recomienda en programación extrema, el código se diseña y desarrolla de forma conjunta por dos personas. Esta forma de trabajar genera una revisión constante del código por lo que no requiere una fase final de revisión. 

Desde luego, desde el punto de vista de coordinación, es el sistema más sencillo, puesto que una vez que se han formado las parejas, cada pareja se organiza para ir haciendo incrementos, que de forma implícita van siendo revisados de forma continua.

En este caso, el diagrama que representa esta forma de trabajo sería: 

A partir de la programación en parejas y viendo sus beneficios, ha aparecido una nueva forma de trabajar conocida como programación en grupo (mob/ensemble programming). En esta modalidad todo el equipo trabaja al mismo tiempo en un único cambio y usando un único ordenador. De esta forma todo el equipo colabora en el diseño de cada pieza de código limitando al máximo los cambios de contexto y minimizando el lead time de cada uno de los cambios. Al igual que en la programación en parejas, la programación en grupo realiza una revisión continua y sincrona del código.

De las formas de trabajar que hemos visto, la programación en pareja o en grupo, es la que tiene un menor lead time para cada cambio, menos cambios de contexto e interrupciones y requiere menos coordinación.

Aun así la programación en parejas y en grupo no es una práctica sencilla y requiere esfuerzo y mucha práctica. Lo que sí tengo claro es que lleva al extremo los beneficios de calidad que se suelen asociar con las revisiones de código.

Visión de la industria y personal

Si en una cosa parece estar de acuerdo la industria es que hacer code reviews es una buena práctica y que deberían hacer todos los equipos de desarrollo. Por otro lado también se recomienda en la mayoría de los casos trabajar en incrementos pequeños, de forma que se pueda controlar mejor la complejidad y el riesgo de cada uno de los cambios.

Teniendo en cuenta estos dos puntos, parece una buena idea usar Code Reviews y que estás sean lo más sincronas y continuas posible, de forma que podamos trabajar en esos incrementos pequeños de baja complejidad y bajo riesgo.

En mi caso, llevo unos cuantos año empujando la programación en parejas o en grupo, creando un flujo de incrementos pequeños que se revisan de forma continua.

Tanto en Alea Soluciones como en TheMotion, utilizábamos programación en parejas por defecto y programación en grupo ocasionalmente. La frecuencia de despliegue de ambas empresas era diaria, y podíamos hacer cambios considerables en pasos pequeños y seguros (3s). Además de este flujo de desarrollo constante y sostenible, obtuvimos atractivas ventajas adicionales, como la facilidad para difundir el dominio y el conocimiento técnico o hacer una rápida incorporación de nuevos miembros. En el caso de Alea Soluciones, contratamos a personas que ya tenían experician en programación en parejas y en prácticas XP, por lo que fue más fácil. Sin embargo, en el caso de TheMotion, tuvimos que introducir las prácticas XP (TDD, programación en parejas, etc.), por lo que contratamos ayuda de coaching técnico.

En Nextail había una mezcla de equipos que hacían programación en pareja y revisión continua y otros que utilizaban revisiones de código asíncronas pero las priorizando para evitar las colas.

En Clarity AI, algunos equipos utilizan revisiones de código asíncronas pero las priorizan para minimizar el lead time. Ya hay dos equipos que hacen revisiones de código continuas (utilizando la programación en parejas o en grupo) y estamos ampliando gradualmente esta forma de trabajar a otros equipos.

Por supuesto el valor estadistico de mi experiencia es nulo y es mucho más interesante analizar los resultados de analisis más serios como el DORA (https://www.devops-research.com/research.html). En ellos podremos ver como se asocian a equipos de alto rendimiento las siguientes prácticas:

• Desarrollo en/desde Trunk (Trunk Base Development) (https://cloud.google.com/architecture/devops/devops-tech-trunk-based-development), donde hacen referencia a trabajar en lotes pequeños y hacer code reviews sincronas. (Ver: https://twitter.com/jezhumble/status/1415908836439773185) 
• Integración Continua (https://cloud.google.com/architecture/devops/devops-tech-continuous-integration) que implica integrar el código en la rama principal al menos diariamente (https://services.google.com/fh/files/misc/state-of-devops-2015.pdf#page=20). En este contexto, hacer code reviews continuas y sincronas, facilita enormemente la introducción de esta práctica.

Relacionado:

• Those pesky pull request reviews
• Pairing Guidelines
• You Must Be CRAZY To Do Pair Programming Pair Programming
• Mob Programming and the Power of Flow • Woody Zuill • GOTO 2019  
• Dan North The Value of Flow
• DORA (devops-research)

Small batches for the win / Continuous Delivery

In software product development the batch size deeply affects flow efficiency and resource utilization. In this post, we will try to explain the underlying principles that make a good strategy to use small batches when we are developing software.

Let’s start with some basic concepts:

• Batch: A group of items that move together to the next step in the process. In this case, is a group of changes that are deployed to production at the same time.
• Item/change: Each one of the individual units of work composes a Batch. In our case any kind of change that affects our system, including new features, code improvement, configuration changes, bug fixes, and experiments.
• Holding cost: The sum of the cost associated with delaying the deployment of each one of the items. In summary, the cost of delaying the feedback or the value delivered by each item. For example the cost of delay of a new feature or the cost associated with not putting in production a bugfix, etc.
• Transaction cost: the cost associated with the deployment. The cost of executing a deployment (cost of people, cost of infrastructure, etc).
  

Batch size / Direct cost

If we only take into account the transaction cost the optimal solution is to make huge batches, as we only pay the transaction cost when we deploy. For example, deploying once a year.

If we only take into account the holding cost, the optimal solution is to use batches with only one item, to avoid delaying any kind of holding cost.

The reality is that if we try to optimize the two variables at the same time we have a U-Curve optimization problem.

We can see that from the optimal batch size the total cost only grows and that before the optimal batch size we have a penalty due to our transaction cost. So a good strategy is always to minimize the batch size until the transaction cost makes smaller batches inefficient.

Batch size / Risk management

When we develop software each component is coupled with other components of the system. That is, that any part of the code has relations with other parts, such as static or runtime dependencies, common messages, data model dependencies, etc. If we invest in  good internal quality, we will minimize the unneeded coupling, but in the worse scenario, each part of the system can be potentially coupled with another part.

When we create a batch with a size of N changes to be deployed at the same time the potential interactions that can happen are:

• Each change with the current version of the system. N
• Each change with the rest of the changes in the batch. That is all the 1-to-1 relations between each change. (N*(N-1))/2

Potential Interactions (I)
Batch size (N)

I = N + (N*(N-1))/2

Whereas this formula describes the number of potential interactions, in general not all of those combinations are possible.

The basic problem with the batch size for software development is that the universe of potential interactions (I) grows very fast. In fact, it is a quadratic function.

We can quickly conclude that the following problems grow depending on the size of the universe (quadratic):

• The probability of an error or a negative impact (functional, performance, cost, etc).
• The cost of detecting/troubleshooting an error.

At the same time we can see that the size of the batch (the number of changes) affects (linearly) the following:

• The number of teams/people required to coordinate/synchronize the deployment (code freeze, communication, testing, etc).
• The possibility of having a change difficult to revert (data model changes, high volume migrations, etc).

 
Let's illustrate how fast the problem grows with an example
If we have an error in 1 of 100 interactions we can see how fast grows the possibility of having an error:

With 25 changes, we already have 88% chance of having at least an error, and with 50 is near sure (99%). And we previously saw that these errors are more difficult to diagnose, and have more possibilities of not being easy to revert.

So clearly, for software development, increasing the size of the deployments (batch size) greatly increases (much more than linearly) the risk associated (risk of having an outage, loose availability, and frustrating our customers).

Batch size / Indirect cost and consequences

In the previous sections, we have seen the direct impact of the batch size on a product development process;

• An increase in the total cost for a batch size greater than the optimal size.
• A near quadratic growth of the risk associated with a deployment. 
• An increasing number of production outages.

In addition to those direct costs, these are other indirect costs and consequences when batch size is large:

• Lots of multitasking and the corresponding productivity and focus lost. The normal flow of work is frequently interrupted with problems that come from previous deployments.
• A tendency to disconnect from the operation and impact of our changes in production. i.e.: when deploying something that you did several weeks ago.
• Low psychological safety, because of the amount of risk and the probability of outages associated with the way of working.
• Worse product decisions because there are fewer options to get fast feedback or to design new experiments to get more feedback.
• Lack of ownership is a derived consequence of the previous points.

Conclusions

As we have shown, the size of the batch has many effects:

• Risk and outage probability are proportional (and worse than linear) to the number of changes included in a deployment.
• Our batch size should be as small as our transaction cost (deployment cost) allow us.
• Large batches generate important indirect costs and consequences (lack of ownership, multitasking, low psychological safety, etc).

Consequently, we should invest as much as possible to try to work in small batches. If we detect that we already reached the optimal batch size for our current transaction cost but we are still having the same problems that we had before, perhaps the next step is to invest heavily in lowering the transaction cost (deployment automation, independent deployments per team, deployment time, etc).

In short:

Small batches -> faster feedback
Small batches -> Customer Value sooner
Small batches -> Reduce direct cost
Small batches -> Reduce deployment risk
Small batches -> Reduce errors
Small batches -> Reduce mean time to recover
Small batches -> Improve psychological safety

Small batches for the win!

The good news is that there is already an engineering capability focused on working in small batches. Continuous Delivery!

"The goal of continuous delivery is to make it safe and economic to work in small batches. This in turn leads to shorter lead times, higher quality, and lower costs." 

Lean Enterprise (Jez Humble, Joanne Molesky, Barry O'Reilly)

The importance of working in small batches has been validated statistically in the studies conducted by DevOps Research and Assessment (DORA) since 2014. You can see the details of these studies in the book Accelerate (Nicole Forsgren, Jez Humble, Gene Kim).

References:

• The Principles of Product Development Flow  (Book)
• Limiting Work in Process (WiP) and Having Small Batches is Antifragile
• Understanding Feedback Loops in DevOps
• Lean Enterprise (Book)
• Accelerate (Book)

Tools used:

• https://www.desmos.com/calculator
• https://www.getguesstimate.com