Thursday, November 16, 2017

Good talks/podcasts (October 2017 Part II)


These are some interesting talks/podcast that I've seen during October:

Monday, November 13, 2017

Crónica de mi CAS 2017 Sevilla

Últimamente escribo mis blogposts en Inglés, pero en este caso, como se trata de una crónica para una conferencia mayoritariamente en Castellano, voy a hacerlo en este idioma (y así me cuesta algo menos :) )



Primero, quiero agradecer a la organización todo el esfuerzo puesto que la verdad es que hacer un evento de estas caracteristicas entre voluntarios es realmente INCREIBLE! Asi que muchas gracias por el esfuerzo...

Y ahora mi crónica:


Charlas a las que asistí:

  • Keynote: Transformación digital y humana Ramón Cabezas Sin duda esta fue la sesión que menos me gusto de todo el evento, me parecio un publireportaje además sin demasiada relación con Agile. Lo siento, pero creo que no fue una keynote a la altura de una CAS.
  • Escalando la Cordillera de la Autogestión Toni Rodríguez Lezcano Charla muy interesante sobre un tema que me apasiona, la autogestión de los equipos. Creo que el orden definido por Toni partiendo de la experiencia real de Magento es de gran valor. 
  • Discusiones y decisiones: herramientas para la efectividad Toño de la Torre Charla muy interesante y que me aporto conceptos interesantes sobre las dinamicas, los juegos ágiles y la facilitación.
  • Valores y principios en el diseño del software Fran Reyes Me encanto, una gran charla. Cada una de las slides, es para leerla con calma y rumiarla. Mucho contenido para cualquier desarrollador avido de mejorar. Me ha convencido para que adelante en mi orden de lectura el Implementation Patterns de Kent Beck.
  • Keynote: La voz, la clave del éxito en tu comunicación Marta Pinillos Creo que no fue adecuada para una CAS. La relación con agile es muy lejana, pero por lo menos en este caso fue divertida. Creo que es interesante buscar keynotes que no sean directamente de la comunidad, pero creo que el mensaje debe resonar con Agile (Ejemplo Koldo Saratxaga en una edición anterior).
  • Keynote: The 8 Stances of a Scrum Master  Barry Overeem No pude verla entera puesto que tenia que preparar mi charla, pero la parte que vi, me resulto interesante. Creo que caracterizo bien los diversos comportamientos típicos de un SM y sus efectos en un equipo. Interesante y divertida.
  • Agilidad (Hacia la entrega continua ¿Qué te lo impide?) Eduardo Ferro Para mi charla he creado un blog post especifico donde incluiré toda la info relacionada http://www.eferro.net/2017/11/agilidad-hacia-la-entrega-continua-que.html
  • Gestionando el roadmap de producto de forma efectiva Vanesa Tejada La vi empezada, pero lo que vi me gusto mucho. La idea de afrontar el roadmap y los retos que se le plantearon volviendo a los principios y valores de agile me parecio genial. Además la aproximación tan Lean que tiene Vanesa para la gestión del producto me parece un ejemplo a seguir.
  • Yo soy Dev, yo soy Ops y somos dos en un equipo Luis Fraile Leo Díaz Una buena presentación de un tema tan desconocido como el DevOps. Creo que hicieron un gran trabajo en introducir el concepto y trasmitir la importancia que tiene. Además fue bastante divertida :)
  • Keynote: Agile works when it's not about Agile - A Paradox Neil Killick Una buena keynote, interesante y me hizo plantearme algunas cosas. Si tengo que decir algo para mejorarla es que el ritmo fue algo lento (creo que ayudado por algún problema técnico que tuvo con el portatil).

Talleres a los que asistí:

Hay cierto tipo de aprendizajes que son más sencillos y profundos si los interiorizas mediante experiencias, bien sea mediante un juego o mediante algún tipo de taller. Por eso creo que para algunos conceptos del mundo agile, un taller es el mejor formato de aprendizaje, así que en esta ocasión intente aprovecharme de la gran cantidad de talleres que se dieron.
Asistí a los siguientes:
  • Liderazgo para la transformación Jose Ramón Díaz Interesante taller en la que se generaron buenas conversaciones sobre el tema en cada uno de los grupos. No fue muy novedoso para mi puesto que ya hice con Josera un taller similar de dos jornadas. En cualquier caso siempre salen conversaciones interesantes. En este tipo de talleres en los que me doy cuenta de soy un poco Anarco :)
  • It’s more than drawing… become a visual facilitator today! Jaume Durany Vendrell y Juan Antonio Sosa A dibujar se ha dicho... la verdad es que ya estoy en el proceso de aprender a hacer facilitación gráfica y a usar los dibujos para expresarme, así que este taller fue una buena sesión de práctica.
  • Kanban system design - Toma el control para que puedas empezar a mejorar Gerard Chiva Pablo Domingo Este taller me aporto mucho, puesto que me ha afianzado algunos de los conceptos de lean que no tenia claros. Muy buen formato y muy útil.
  • Auto organización a escala. Adrian Perreau Gran experiencia de auto organización para resolver un problema. Muy interesante ver como nos comportabamos y las dinamicas que se generaban. Posteriormente, Adrian repaso los comportamientos que suelen aparecer y algunos de los antipatrones que se pueden dar. Muy interesante. Una gran experiencia.


Feedback para la organización


Positivo:
  • El sitio espectacular, tanto Sevilla como el Fibes. Muy cómodo bien comunicado y el palacio de congresos, muy amplio y cómodo para ir de sala a sala.  Además las salas estaban muy bien.
  • El soporte de la organización a los ponentes en cuanto a preparación de sala, medios, acceso a internet, etc, perfecto.
  • Buena organización general de tiempos, comunicación y servicios.
  • Me gusto la idea de reservar para los talleres... Creo que hay que darle una vuelta a ver como gestionarlo, pero en general me parece buena idea.
A mejorar:
  • Los keynotes en general no me han gustado mucho, ya he comentado en el punto anterior cada una, pero las del primer dia me decepcionaron bastante. Aunque no sean directamente sobre Agile, deben estar relacionadas de una forma clara.
  • La comida no me convencio (y eso que soy adicto al queso). Para un intolerante a la lactosa debe haber sido complicado.
  • El sistema de reserva de talleres se deberaia haber comunicado con más antelación.
  • Las salas de talleres se han quedado algo pequeñas.
  • El track técnico estaba etiquetado como iniciación cuando muchas de las charlas eran avanzadas.

En cualquier caso, todos estos comentarios son para mejorar, partiendo de que considero impresionante el trabajo que se hace para organizar un evento de este calado. Así que mis más profundas GRACIAS para toda la organización y para Agile Spain por apoyar y organizar este tipo de eventos.

Aprovecha para hacerte socio de Agile Spain https://agile-spain.org/asociate/ y colabora con la organización de la CAS, el AOS y multitud de conferencias como la Pamplona Software Craftsmanship o la Barcelona Software Craftsmanship. Cuesta poco y aporta mucho



Desvirtualizaciones

Otra cosa que me estoy acostumbrando a hacer en las conferencias es a desvirtualizar a gente interesante... Aunque parezca que no, soy muy tímido (aunque me lo estoy curando a golpes).
En esta CAS he tenido la suerte de desvirtualizar a:


  • Seguia a Aritz Suescun desde que en un AOS presento dos maginificas sesiones  sobre Design Sprints y Product Discovery.
  • Con Rafael Luque comparto la pasión por la OO (de verdad :) ) y me ha encantado poder conversar con él sobre el tema. Espero tener suficientes puntos familias para poder acercarme al meetup sobre Pharos Smalltalk que va ha dar dentro de poco :).
  • Samuel Casanova es un gran Coach Agile y me encanta su blog. Además, como también le da al visual thinking he usado alguno de sus dibujos para alguna de mis presentaciones (http://www.eferro.net/2017/10/charla-acelerando-la-cultura-devops.html).
  • Parece que Dani de la Cruz y yo coincidimos bastante en como vemos el desarrollo de software y ha sido un placer poder desvirtualizarle por fin.
  • He coincido con Adrian Perreau en varios Socracan y aunque habiamos hablado en el pasado, nunca nos habiamos presentado formalmente, así que he aprovechado en esta ocasión.
  • Seguia Vanesa Tejada por su interesante enfoque sobre desarrollo de producto basado en Lean y por los post sobre productividad personal.

Desvirtualizaciones Pendientes

Al final por falta de tiempo y organización no pude desvirtualizar a Carlos Iglesias al que admiro por el trabajo que hace en la comunidad Agile Barcelona y a Jerónimo Palacios que me parece que es una de las grandes figuras que tenemos por aquí... Me quedan pendientes para la próxima ;)

La gran pregunta

¿QUÉ ES LA CAS?
La CAS 2017 es un evento generado por personas que viven el desarrollo de software de una manera diferente.
Es un punto de encuentro entre profesionales del sector donde se comparten conocimientos y experiencias en torno a las metodologías ágiles.
Esta cita está extraida de la web de la conferencia, y como se puede ver, habla de desarrollo de software y metodologias ágiles, pero mi sensación en la conferencia es que cada vez es menos relacionado con eso. Así que como comunidad tenemos una pregunta que hacernos, ¿Cómo queremos que sean las próximas CAS? ¿Queremos que sean orientadas al Coaching? ¿Queremos que integren a los desarrolladores? ¿Queremos que exploren diversos ambitos partiendo del desarrollo de software como indica el manifiesto?...

Creo que debemos hacernos este tipo de preguntas, puesto que ahora mismo el resultado es bastante confuso y me apena escuchar desarrolladores diciendo que no van a la CAS por que Agile no tiene relación con el desarrollo de software :(

Lo que si que me alegra ver es que como consecuencia de esta CAS se está abriendo algo de debate sobre como deberia evolucionar la comunidad y la propia asociación...  Veamos que sale :) Inspeccionemos y adaptemonos.

Enlaces Relacionados:



Friday, November 10, 2017

Agilidad. Hacia la entrega continua. ¿Qué te lo impide? CAS2017




Slides de mi charla para la CAS 2017
Agilidad. Hacia la entrega continua. ¿Qué te lo impide?

Las slides sin la explicación son de poca ayuda. Pero actualizaré este post con el video, referencias y cualquier otra cosa relacionada.

Feedback de la charla




Otras charlas previas relacionadas:

Tuesday, October 31, 2017

Good talks/podcasts (October 2017 Part I)

These are some interesting talks/podcast that I've seen during October:


Sunday, October 22, 2017

Book Review: Reinventing organizations





Reinventing organizations

A Guide to Creating Organizations Inspired by the Next Stage of Human Consciousness

by Frédéric Laloux

This is an incredible book that has had a deep impact on my thinking… It gives a great historical view of the different organizational model used by the humans and the correspondence of each model with each new stage of human consciousness.
It also explains how we are changing to a new stage and that a new kind of organizational model is appearing.
After giving this context and the explanation for each model, the book describes, this new "Teal" organizational model. The examples are very well documented and include a lot of details about the internals of the organizations selected.


The author also describes the environment from a human angle that allows this kind of organization to flourish… This new organization, the "Teal" organization, is optimized to deal with actual complexity.

An essential book for anyone interested in discovering the organizations of the future for this complex world.

A wonderful book.

Related resources:

Good talks/podcasts (September 2017)

These are some interesting talks/podcast that I've seen during September:

Sunday, October 15, 2017

My Software Craftsmanship Barcelona 2017


Dates: 7th, 8th October
Official site: http://scbcn.github.io/



This past weekend (7-8th Oct) I was at the Barcelona Software Craftsmanship Conference 2017. This is the second time attending this conference (http://www.eferro.net/2014/10/software-craftsmanship-barcelona-2014.html).

This conference is a two days conference organized by the Barcelona Software Craftsmanship community and this year congregate near two hundred passionate software crafters.

First, I want to thanks a lot the effort of the organization and the sponsors. This was the first year that they changed the venue and open the conference to more the attendants. Organizing an event of this magnitude is difficult, but this time, due to the political situation in Catalonia, it was even more difficult. So my more sincere congratulations for the effort and for the result.






I only was able to attend the Saturday sessions and to the first one of the Sunday (some family related logistics issues). I any case I enjoyed a lot the sessions and fully charged my batteries to research some of the things I saw there :)

The sessions I attended:

Other notes and stuff

As part of my program to not being a technological amoeba, I am trying to de-virtualize interesting people in this kind of conferences. This time I have the great luck to de-virtualize:
Lot of thanks for sharing a portion of your time with me, it was a real pleasure :)

I am also very proud that some colleagues from TheMotion have come to this conference. I know that for a lot of people, being in craftsmanship conference is a turning point in their career... so I liked to be involved in such important moment. :)

It was also a great pleasure to share my time with old friends and colleagues. 

Tuesday, September 26, 2017

Focusing my learning... (Js, Serverless, Aws, basic frontend)



I have some obsessive tendency to accumulate info/resources for learning (talks, presentations, blog post, books, podcast, etc).
From time to time I need to carry out an exercise to identify the things that I want to learn to complement my actual skills... This exercise helps me to narrow (temporarily) my scope for learning and gain speed, focus and motivation.

The process:


  1. I bring together a lot of topics that I am interested in (from languages, and technologies, to soft skills, and cultural topics).
  2. I classify each topic in several levels (my interest, my actual experience, level of alignment with my personal path/mission, etc.).
  3. I try to discard as many topics as possible... for example because I am already proficient or good enough. In these cases, I'd like to learn more, but I will not make any special effort.
  4. I select three or four topics.
  5. I remove mercilessly unrelated items from my list of resources (books, talks, blog posts, articles, etc...).  It is important to remove as much as possible because we don't have enough energy and if something is really important it will reappear in the future... :)




The results:


  1. Learning Javascript (the language). I will try to improve my Javascript knowledge up to a basic knowledge. Focused on the language itself and the most common execution environment (node and browser).
  2. Serverless. I see the serverless based architectures as the next step for a PaaS that force us to think in terms of events and create cloud-native designs. I think that this technology will be very important in the near future.
  3. Advanced AWS usage. In this case, I decided that even if I learned a lot about cloud technologies in my actual job I need to go deeper in AWS to really understand and gain more confidence.
  4. Frontend minimal stuff. I have developed from Linux kernel drivers up to cloud-native applications, but only a few times I needed to develop customer-facing apps (or at least in the actual frontend apps sense)... So my idea in this regard is to have the minimal experience with HTML5, CSS and the minimal amount of javascript needed to implement small SPAs.

I also take other actions to remove inputs for distraction:
  • Unfollow a lot of people on twitter (sorry, not enough time...).
  • Remove sync for email on my mobile.
  • And remove a lot of apps that generate push notifications.

This process also helps a lot with my impostor syndrome :)




And remember, when you buy a book, you don't buy the time to read it

Related content:

Saturday, September 09, 2017

Good talks/podcasts (August 2017)


These are some interesting talks/podcast that I've seen during August:

Tuesday, September 05, 2017

Pub-Sub the swiss army knife (tech pill)

Pub-Sub / Publish-Subscribe

"In software architecturepublish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are." wikipedia Publish-Subscribe pattern
As the Wikipedia explains, the pub-sub patterns allow decoupling the publishers (P) from the possible subscribers or consumers (C). So the publisher doesn't need to have any knowledge about the components interested in receiving the message.
To implement this pattern we need a central service in charge to receive the messages from the publisher and resend the message for each one of the interested subscribers. Usually, this central service is called broker (B).

Some common characteristics for a broker are:
  • Allow broadcasting message to several consumers.
  • Allow the consumers to configure in which messages are interested using some kind of consumer defined rule.
    • By topic
    • By regular expression over a topic 
    • By a query over some attributes of the message
  • Allow the publisher to send messages and add some meta info to it
    • Attributes
    • A destination topic
    • Other meta info as priority, TTL, etc
  • Allow several consumers to receive the same message
Another typical characteristic, and in fact, the most important one is that for each consumer the broker usually has a queue to maintain the pending messages.
And this queue has all the characteristics that we explain in the Queues vs Distributed log post... For example, the broker can load balance the messages between a group of consumers.



In summary, the main characteristics of a pub-sub system are:
  • Publishers:
    • They don't know the final destination of the messages (decoupling).
    • They send the messages to a topic/exchange/abstract destination.
    • They can add attributes to the messages.
  • Consumers/Subscribers:
    • Each one informs the broker the in which messages are interested (using the name of the topic, a regular expression over the topic or a combination of attributes of the message).
    • Each message can be consumed by several consumers (broadcasting).

Pros:

  • Great decoupling between publishers/consumers.
  • Allow easy creation of flexible communication topologies.
  • All the pros of Queues.
    • Easy to implement.
    • Unlimited/Easy horizontal scalability.
    • Fault tolerance is very easy to implement (time out and re-queue of the message).
    • Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons (same as queues):

  • The order is not guaranteed.
  • Usually, we can have duplicates.
  • Requires more resources from the broker some in some scenarios it has worst scalability than other solutions (not important for the majority of the cases).

Use a Pub-Sub system when:


For any use case that requires flexibility, broadcasting of messages and doesn't require that order of the messages is guaranteed. In these scenarios, a pub-sub system is a great solution because it includes all the use cases of a queue and all the flexibility of sending the same message to several queues/consumers.

The real potential of this kind of solution is when you combine several brokers (using federation or replication)  to create flexible topologies that can communicate several systems and services.


Use cases / Examples:

  • Any good scenario for queues.
  • Async processing of requests.
  • When we can allow losing some data (or having some delays), Pub-Sub systems are great for:
    • Log / Monitoring info distribution and processing.
    • Asynchronous processing of email requests.
    • Processing of independent batch jobs.
  • Using Federation
    • Info replication and distribution between data centers.
    • Global distribution of periodic information.
    • Global distribution of reference info.

Implementations:

Related content:

Monday, August 21, 2017

Good talks/podcasts (Purging the queue III)

Friday, August 18, 2017

Good talks/podcasts (Purging the queue II)



Sunday, August 13, 2017

Books I have read lately and Reads In Progress (RIP)

Continuing with my process of recovering my previous reading habit (books-i-have-read-in-last-12-months http://www.eferro.net/search/label/books) these are the books that I read lately:

  • "Nonviolent Communication: A Language of Life"  Marshall B. Rosenberg Review
  • "The Power of Habit: Why We Do What We Do in Life and Business" Charles Duhigg
  • "Los 7 Habitos de la Gente Altamente Efectiva" Stephen R. Covey
  • "The Toyota Way" Jeffrey Liker
  • "Algorithms to Live By: The Computer Science of Human Decisions" Brian Christian, Tom Griffiths
  • "El Arte de Vivir: Meditación Vipassana tal y como la enseña S.N. Goenka [The Art of Living]" William Hart
  • "The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life" Mark Manson
  • "Tribes: We Need You to Lead Us" Seth Godin
  • "The Five Dysfunctions of a Team: A Leadership Fable" Patrick M. Lencioni Review
  • "The Happiness Hypothesis: Finding Modern Truth in Ancient Wisdom" Jonathan Haidt
  • "Team of Teams: New Rules of Engagement for a Complex World" General Stanley McChrystal, Tantum Collins, David Silverman, Chris Fussell Review
  • "Focus: A Simplicity Manifesto in the Age of Distraction" Leo Babauta
  • "The Sales Bible: The Ultimate Sales Resource" Jeffrey Gitomer 
  • "The Ideal Team Player: How to Recognize and Cultivate the Three Essential Virtues: A Leadership Fable" Patrick M. Lencioni
  • "Ego Is the Enemy" Ryan Holiday
  • "Barking up the Wrong Tree: The Surprising Science Behind Why Everything You Know About Success Is (Mostly) Wrong" Eric Barker
  • "Scrum" Jeff Sutherland, JJ Sutherland


Read in progress (RIP) :)

  • "The Lean Product Playbook: How to Innovate with Minimum Viable Products and Rapid Customer Feedback" Dan Olsen
  • "This is Lean: Resolving the Efficiency Paradox" Niklas Modig, Par Ahlstrom
  • "Implementation Patterns" Kent Beck (2º reading)
  • "The real startup Book" v0.3 

Friday, August 11, 2017

Scalable design pattern sample (tech pill)


As a complement to the previous blog post Queues vs Distributed logs (tech pill) this blog post describes how to solve a concrete business problem using a distributed log and a queue.... I used this design several times in production with good results.

Any feedback or improvements of the design will be more than welcomed :)

The problem:

We have to implement a business process with the following characteristics:
  • The business process (Job1) can be divided into three different sequential phases (S1, S2, S3). For example, we can think about the generation and mailing of all the invoices for all the customer of one account manager.
  • The second phase (S2) can be divided in several (hundreds or thousands) individual and independent sub-jobs (S2.1, S2.2, ...). This sub-jobs can be executed/processed in any order. For example, generate and email one invoice. Each sub-job require one minute of process time.
  • The complete process should complete in less than 90 minutes without being affected by the number of sub-jobs of the second step (up to 10k sub-jobs).




We also need to:
  • Notify when each step starts and when each step finished.
  • Generate some statistics of each the process.
  • Access to the detailed status of the process at any moment.
The solution should have the following characteristics:
  • We can have several numbers of concurrent business processes of this type for each customer.
  • For the sub-jobs at step 2:
    • We have retries for the sub-job execution.
    • We can balance the time of the process and the cost.
    • We have horizontal scalability.

The Design:

After analyzing the requirements and make a fast web storming session we consider the following events:


If we need to include more information about the detailed state of the process we can also signal the starting point of each step using a StepXStarted event. But these Starting Events are not required because we already know when a step started (just when the previous step finished).

Implementation


As we want to implement several functionalities for the same events and we want to have each one completely separated, we can use a distributed log / stream that allows us to design a simple solution to manage the workflow, calculate statistics, compute a detailed status.
The distributed log / stream and the corresponding services can scale using as partition/sharding key the job id or the customer id (assuming that each customer can generate several jobs at the same time).



The Step2 require additional design.
It can be subdivided in several individual jobs (S2.2, S2.2, ...), this jobs can be processed in any order and in parallel so we can have a queue for dispatching this subjobs to a pool workers that can be scaled out if needed.
We can balance the number of workers (and the corresponding cost) with the duration of the Step2 that we want. Each worker get a job description from the queue, execute the job, and include the result in an event (Step2SubjobCompleted) published in the distributed log / stream.



The Workflow Manager should implement a response for each event and generate corresponding events to make the job progress.

The responses for each event are:

Job1Requested:

  • Execute Step1
  • Publish Step1Completed

Step1Completed:

  • Split the Step2 in several subjobs (S2.1, S2.2, ...)
  • Store the identifiers of the jobs created (S2.1, S2.2, ...)
  • Send the subjobs to the queue of S2 subjobs

Step2SubjobCompleted:

  • Mark the id of the job as no longer pending
  • Validate if we have jobs pending
  • if there is no more jobs pending, publish Step2Completed

Step2Completed:

  • Execute Step3
  • Publish Step3Completed

Step3Completed:

  • Do any garbage recollection needed
  • Publish Job1Completed

Job1Completed:

  • Nothing, Everything is already done  :)

For the same events, other functionalities as Statistics, reports or notifications, will define other reactions/implementation to process each event... Compute and store statistics, generate emails or push notification, create a view with detailed information on the status of the job, etc...


Notes and complementary designs:

  • The queue allows duplication so the workflow should be prepared to receive several events Step2SubjobCompleted for the same subjob.
  • We can have retries for the queue jobs, so we should include a mechanism to detect when we should stop waiting for Step2SubjobCompleted. For example, we can use a periodic tick event and use this event to decide if we should continue with the next step (for example marking a subjob as an error).
  • Is also possible to continue receiving Step2SubjobCompleted even if we are at Step3, the easy solution is to ignore this messages.
  • If we select the Job id as the sharding/partition key for the distributed log / stream we can easily scale out the number of workflow processors. We only need to add more stream shards and the corresponding new processes.
  • For the fault tolerance of the workflow we can store the events that we already processed and recover in case of failure from this events.

Related references (very recommended):

Sunday, July 30, 2017

Honey Badger Team - Visual History III - CD as driving force

After publishing Honey Badger Team visual history, @artolamola asked me on twitter if I used "continuous delivery" as the driving force for the transformation and why. The short answer is Yes, and in this blog post will try to explain the motivations behind.

For me, agile requires two things (see the-two-pillars-of-agile-software):
  • A healthy knowledge culture focused on people (collaboration, learning, respect, team work, creativity...) 
  • Looking for quality and technical excellence (for example XP practices are a great starting point).
To introducing the healthy knowledge culture, I introduce retrospectives and worked as a facilitator,  change agent and "Developer Whisperer".
In our profession knowledge is the bottleneck

For the technical excellence I think that we should follow this steps:
  • If there is a classic vicious cycle of bad quality or bad practices going on. We should stop and break this cycle immediately. The typical example consists in having technical debt, that generates bugs, that generate even more technical debt that ends in the complete collapse in few months.
  • Assume that we have the capacity and very good intentions.
  • Help to create a virtuous cycle to improve our technical quality day by day.
  • To create this virtuous cycle we need to define a clear goal to align all of our efforts.
Vicious Cycle of technical debt :)

In our case, and having in mind that we are a cloud native product, I thought that we can use Continuous Delivery as our team Goal

Why Continuous Delivery?

The answer is that I thought that we require a good practice that force us to do the thing right and doesn't allow us to hide our unprofessionalism or incompetence. And for this, and in the cloud, I think that in our case Continuous Delivery will put a lot of pressure on doing the things right.

Continuous Delivery implies:
  • Low cost and a smooth process for deployment that doesn't affect customers. This requires (or at least recommend) introducing DevOps, automation, rolling deploys and Continuous Integration.
  • Having good confidence in our product. This requires Continuous Integration, Automatic testing, and Monitoring for production.
  • Automatic testing requires integration tests and unit tests, and for sure, having unit tests put a lot of pressure on doing a good design (for example to have a hexagonal architecture or other architecture that decouple business logic from infrastructure).
So indirectly, trying to implement Continuous Delivery requires or recommend doing the following practices: DevOps, Infrastructure Automation, Zero down time deploys (for example rolling deploys), Continuous Integration (perhaps with trunk based development), Integration testing, Unit testing, Good Design (for example using hexagonal architecture).

Virtuous Cycle created when CD is the goal


Continuous Delivery doesn't allow you to hide bad technical quality and force you to introduce good technical practices and work very hard to have a healthy code base.

With this approach, we, as a team, improve a lot our technical knowledge, our code base and our practices. We have to improve many technical practices but we have already come a long way and I sincerely believe that we are going in the right direction. :)

Extraball:

  • If you use micro services, CD force you to think about independent deployability.
  • CD force you to learn how to make parallel changes (using feature toggles, database refactoring, etc.)
  • Some additional insights from @artolamola:
    • Above CD you can implement any agile method for team organization (scrum, kanban, etc.).
    • CD force you to think how to decompose each feature in vertical slices.

References:

Some of these ideas come from:

Some context about how I understand agile:


The history of the Honey Badger team:

Saturday, July 22, 2017

Queues vs Distributed logs (tech pill)

TL;DR A queue is a good choice when you have one kind of job to do that you can divide in independent smaller jobs that can execute in any order and a distributed log is a good choice when you have several kinds of jobs or functionalities for the same stream of data (logs, events, etc.).

Queues vs Distributed Logs

This blog post tries to explain the typical use for Queues and for Distributed Log, but of course, a system usually uses these solutions in combination or in other ways. But I will try to summarize the usual use because some times I see some confusion about how we can use these interesting solutions.

Queue



Use a Queue when:

  • You want to distribute workload between workers for the same feature. 
  • Each message can represent a job and don't require knowledge about other messages at the queue.

Pros:

  • Easy to implement.
  • Unlimited/Easy horizontal scalability.
  • Fault tolerance is very easy to implement (time out and re-queue of the message).
  • Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons:

  • The order is not guaranteed.
  • Each message can only be consumed by one worker for one feature.
  • Usually, we can have duplicates.

Examples

  • Asynchronous processing of email requests.
  • Processing of independent batch jobs.
  • A supermarket queue :)

Alternatives and related variations

  • Queues with guaranteed order (Ex: SQS FIFO).
  • TTL for messages in the Queue.
  • Queues with a max amount of queued messages.

Implementations:

  • AWS SQS, RabbitMQ

Distributed Log


Use a Distributed Log when:

  • You want to execute several functionalities for the same stream of ordered data (log aggregation, statistics calculation, event sourcing, store streams of information, etc.)
  • You want that the data corresponding to the same partition key is processed in order by the same worker instance.

Pros:

  • Allow several functionalities for the same data. Each functionality knows what is his own offset.
  • If two or more functionalities are at the same offset, they can be sure that have seen the same messages in the same order.
  • Having a guaranteed order,  at least at the partition key level, allow design simple solution for calculating statistics, event sourcing, near real-time calculation, etc. 
  • You can add new functionalities that use/process the same data with any change in the actual system (Open/Closed principle: open for extension but closed for modification).

Cons:

  • Usually, the order of the messages is preserved for each partition key only.
  • Scalability using sharding/partitioning.
  • More difficult than queues.
  • Difficult to have balanced shards/partitions (hot partition problem).

Examples

  • Log aggregation / distribution / statistics calculation from the logs.
  • Event streaming to allow event sourcing.

Implementations:

  • AWS Kinesis, Kafka

Practical scalability example:

In this example we have the following:

  • One logical stream
  • Divided into two shards
  • Each partition key (pk) will always be associated with a shard or partition
  • We have one function that runs in two concurrent processes, so the speed of processing is x2. Each process always read events/messages for the same partition keys.

In this example, the first process reads events/messages from the partition key 1 and from the partition key 2.

If the shard s1 have too many event/messages per second, we should scale out dividing the shard s1 into two sub shards.


Now, our speed is x3, and we have one process for each partition key.
As we can see the scalability is not trivial because we should detect when a shard/partition should be divide and rebalance the shards/partition between the running processes.


References:

Saturday, July 15, 2017

Honey badger Team / The visual history II


Disclaimer: I am experimenting with different formats to create presentations, blog post, and other documents, mixing sketch noting and using index cards... I will appreciate your feedback.

In the previous post about the history of the Honey Badger Team, the focus was to define the context, explain how we evolve our practices, our culture and get our identity as a team. But in parallel, there is a history of tension and lot of work to make this change possible :)

As in any software product development effort, the path was not easy. Perhaps the first blog post doesn't reflect the amount of energy and discussions required to introduce quality and good practices in our team.

During this process (one year and a half), there was tension between different forces: time, scope, learning needs, maintainability, quality, short term value, long term value...

As usual, the main problem is that not all the company have the same experience about technology development so is very difficult to reach a complete alignment for a sustainable strategy...

This was my job or at least part of my job, and for sure, it wasn't easy and requires a lot of energy, patience, and continuous learning...  Learn about how to sell ideas, learn about how to get some margin, learn about how to generate trust...
I also tried to teach about agile concepts, lean, collaboration, etc.

During part of this journey we also had a CTO that helped us to defend the need for change, so even if the change process was already there, is true that he helped me to accelerate the process a little bit (for example he negotiated a percentage of time to improve our test network). He also helped me to translate technical concepts to the C level.


The main problem, as in any complex system that involves humans, was the communication. Seems that we use a completely different vocabulary and even language. I try to solve this miscommunication trying to learn about our business, gain trust and adapting our message to the actual understanding of the technology.


There was an inflection point in the process that could cause the collapse of the team. We started to work as a team, introduce the XP practices and improve day by day, but in parallel, there was some misalignments at the company level that was introducing a lot of structure, a lot of levels and an environment not very aligned with our spirit of collaboration...
This problem drained the motivation of the tech team, that feel very good improvements regarding our internal processes but feel the friction generated by this amount of structure and the misalignment.

This process ended with some deep changes at the company level, but during this period of time, we lost great engineers and great colleagues :(  Personally, this was a rough time for me.
Seems that we have overcome that problem and that we are in the right direction, adding new honey badgers to the team.

In summary, the first blog post explained the internal changes but this doesn't reflect the whole history... in parallel, there was a huge effort to improve the communication between each group in the company, deal with all the typical tensions, and a lot of small changes to introduce a more agile culture in times of huge uncertainty...

And I can assure that this path was not easy... required patience and energy...   and this are a scarce resource :)

Reference: