Honey Badger Team - Visual History III - CD as driving force

After publishing Honey Badger Team visual history, @artolamola asked me on twitter if I used "continuous delivery" as the driving force for the transformation and why. The short answer is Yes, and in this blog post will try to explain the motivations behind.

For me, agile requires two things (see the-two-pillars-of-agile-software):

• A healthy knowledge culture focused on people (collaboration, learning, respect, team work, creativity...) 
• Looking for quality and technical excellence (for example XP practices are a great starting point).

To introducing the healthy knowledge culture, I introduce retrospectives and worked as a facilitator,  change agent and "Developer Whisperer".

In our profession knowledge is the bottleneck

For the technical excellence I think that we should follow this steps:

• If there is a classic vicious cycle of bad quality or bad practices going on. We should stop and break this cycle immediately. The typical example consists in having technical debt, that generates bugs, that generate even more technical debt that ends in the complete collapse in few months.
• Assume that we have the capacity and very good intentions.
• Help to create a virtuous cycle to improve our technical quality day by day.
• To create this virtuous cycle we need to define a clear goal to align all of our efforts.

Vicious Cycle of technical debt :)

In our case, and having in mind that we are a cloud native product, I thought that we can use Continuous Delivery as our team Goal. 

Why Continuous Delivery?

The answer is that I thought that we require a good practice that force us to do the thing right and doesn't allow us to hide our unprofessionalism or incompetence. And for this, and in the cloud, I think that in our case Continuous Delivery will put a lot of pressure on doing the things right.

Continuous Delivery implies:

• Low cost and a smooth process for deployment that doesn't affect customers. This requires (or at least recommend) introducing DevOps, automation, rolling deploys and Continuous Integration.
• Having good confidence in our product. This requires Continuous Integration, Automatic testing, and Monitoring for production.
• Automatic testing requires integration tests and unit tests, and for sure, having unit tests put a lot of pressure on doing a good design (for example to have a hexagonal architecture or other architecture that decouple business logic from infrastructure).

So indirectly, trying to implement Continuous Delivery requires or recommend doing the following practices: DevOps, Infrastructure Automation, Zero down time deploys (for example rolling deploys), Continuous Integration (perhaps with trunk based development), Integration testing, Unit testing, Good Design (for example using hexagonal architecture).

Virtuous Cycle created when CD is the goal

Continuous Delivery doesn't allow you to hide bad technical quality and force you to introduce good technical practices and work very hard to have a healthy code base.

With this approach, we, as a team, improve a lot our technical knowledge, our code base and our practices. We have to improve many technical practices but we have already come a long way and I sincerely believe that we are going in the right direction. :)

Extraball:

• If you use micro services, CD force you to think about independent deployability.
• CD force you to learn how to make parallel changes (using feature toggles, database refactoring, etc.)
• Some additional insights from @artolamola:
• Above CD you can implement any agile method for team organization (scrum, kanban, etc.).
• CD force you to think how to decompose each feature in vertical slices.

References:

Some of these ideas come from:

• Competing On The Basis Of Speed Mary Poppendieck
• The Virtuous Cycle of Velocity: What I Learned About Going Fast at eBay and Google Randy Shoup

Some context about how I understand agile:

• Agile is not a recipe for success...
• The two pillars of agile software
• Remove the excuses for not being a professional

The history of the Honey Badger team:

• honey badger team visual history
• honey badger team visual history II
• honey badger team

Honey badger Team / The visual history II

Disclaimer: I am experimenting with different formats to create presentations, blog post, and other documents, mixing sketch noting and using index cards... I will appreciate your feedback.

In the previous post about the history of the Honey Badger Team, the focus was to define the context, explain how we evolve our practices, our culture and get our identity as a team. But in parallel, there is a history of tension and lot of work to make this change possible :)

As in any software product development effort, the path was not easy. Perhaps the first blog post doesn't reflect the amount of energy and discussions required to introduce quality and good practices in our team.

During this process (one year and a half), there was tension between different forces: time, scope, learning needs, maintainability, quality, short term value, long term value...

As usual, the main problem is that not all the company have the same experience about technology development so is very difficult to reach a complete alignment for a sustainable strategy...

This was my job or at least part of my job, and for sure, it wasn't easy and requires a lot of energy, patience, and continuous learning...  Learn about how to sell ideas, learn about how to get some margin, learn about how to generate trust...
I also tried to teach about agile concepts, lean, collaboration, etc.

During part of this journey we also had a CTO that helped us to defend the need for change, so even if the change process was already there, is true that he helped me to accelerate the process a little bit (for example he negotiated a percentage of time to improve our test network). He also helped me to translate technical concepts to the C level.

The main problem, as in any complex system that involves humans, was the communication. Seems that we use a completely different vocabulary and even language. I try to solve this miscommunication trying to learn about our business, gain trust and adapting our message to the actual understanding of the technology.

There was an inflection point in the process that could cause the collapse of the team. We started to work as a team, introduce the XP practices and improve day by day, but in parallel, there was some misalignments at the company level that was introducing a lot of structure, a lot of levels and an environment not very aligned with our spirit of collaboration...
This problem drained the motivation of the tech team, that feel very good improvements regarding our internal processes but feel the friction generated by this amount of structure and the misalignment.

This process ended with some deep changes at the company level, but during this period of time, we lost great engineers and great colleagues :(  Personally, this was a rough time for me.
Seems that we have overcome that problem and that we are in the right direction, adding new honey badgers to the team.

In summary, the first blog post explained the internal changes but this doesn't reflect the whole history... in parallel, there was a huge effort to improve the communication between each group in the company, deal with all the typical tensions, and a lot of small changes to introduce a more agile culture in times of huge uncertainty...

And I can assure that this path was not easy... required patience and energy...   and this are a scarce resource :)

Reference:

• history of the Honey Badger Team (Part I)

Honey badger Team / The visual history (I)

Disclaimer: I am experimenting with different formats to create presentations, blog post, and other documents, mixing sketch noting and using index cards... I will appreciate your feedback.

I prepared these notes for the past AOS 2017, but finally, the session was not selected... The initial idea was to describe our process for creating an agile software development team and create a discussion to identify next steps to expand our agile culture to the rest of the organization

Honey Badger Team / The visual history

In the beginning, there was a bunch of developers at TheMotion starting to create the product. But there was no process and the "established" tendency was to try to optimize the resource usage. As we already know, optimizing the resource usage have some serious problems:

1. People are not resources :) (so this not work very well outside some simple industrial processes/pipelines that don't involve humans)
2. This process generates busy work but not necessarily good outcomes.
3. To optimize the resource usage, we need:
1. Push until all of the resources are 100% used
2. To create buffers to have everybody busy
3. Generate SILOs / Specialization for each skill (front end, backend, video)

The result of this process was:

• An incredible high bus factor
• Stress and bad quality code and practices
• Tons of WIP/inventory and unneeded features as we develop based on how is available (frontend, backend...)

I arrived at this point to the company... There was a Funny party going on :)

One thing was clear (at least for me), using a Lean and Agile approach can help to create a product in a startup.... It also can help to improve the overall frustration :)

I am completely biased because I like the agile mindset... but I sincerely think that in a startup the agile approach is the right choice:  Fail (agile is not recipe for success) and learn as fast as possible :)

For me, Agile is a set of values and principles, not a methodology and two pillars:

• A healthy culture focused on people (collaboration, respect, teamwork, creativity...) 
• Looking for quality and technical excellence (for example XP practices are a great starting point).

So my first target was introducing quality and technical excellence to be capable of delivering software in small cycles (days instead of weeks)... My idea was to introduce the culture at the same time that we improve our technical capacity. Any software developer is more open to a new ideas if he shows improvements in the code and in his day to day work. :)

But how can we detect if we are in the right path?
I discovered that a good KPI for measure these improvements was how much anxiety was generated in each team member when we change the frequency of a development event (deploy, merge, etc...)
For example, the target was:

• Deploys frequency, from 1 a month, to several per day (the deploy is not an event).
• Integration frequency, from branches that last forever to trunk based development.
• Feedback time:
• For review, from code reviews and pull request to pairing and continuous review.
• Subsecond for unit tests.
• Seconds for integration tests.
• Minutes for production monitoring.

But How can we improve the quality and our technical capacity???

For me, the response is agile development, XP practices and modern agile... In summary being professionals and pragmatic delivering software. And mentoring is the only way I know to introduce this technical practices....

Initialy, I was the only one that started to send the message about XP, Craftmanship, cloud native, etc... At the same time, we were trying to improve the day-to-day work improving the infrastructure and the base platform.

We, as a team, was making improvements, but learning TDD or pairing have an important entry barrier so we need external help with experience teaching this practices (by doing of course).

To improve the "mentoring power", the collaboration of Modesto San Juan and Alfredo Casado from Carlos Ble y Asociados (now codesai) was key. They help us to improve our TDD and to introduce other XP practices (shared ownership, pairing, etc). At this point, our internal alignment started to be evident.

The first day that I detected that the mentoring effort was having a profound effect was when one developer, talking aloud, explained what Modesto or Alfredo would say about his code, and immediately he started to improve the code :-)

Another key point was that from day 0, we assumed that everybody wants to make a good job so the smarter approach to improve is to "Make easy to make the right thing". Instead on focus in preaching how the things should be done and instead of forcing to do the things in a certain way, our focus was on creating an awesome tooling and platform that make very easy to create microservices, include instrumentation, metrics...  We pushed very hard to have an automated infrastructure and a very good tooling.... This includes rolling deploys, standard makefiles, a devbox (now deprecated), containers, etc.a great monitoring.

With this platform and this tooling, it was easy to introduce a DevOps culture, with microservices and improve our deployment cadence day by day.

An important note here.... during this process, the most important factor was to use an agile process and make small improvements, with low risk in parallel to the normal development...

Right now, our practices include, continuous delivery (several times a day), automatic tests (TDD in the way but not the default option yet), simple design, immutable infrastructure, infrastructure as code, collective ownership and we work as a high-performance team with our own identity and a high level of trust….
We have a lot of things to improve, including our quality at the front end (software design and practices), more solid understanding and usage of the XP practices, and so on... but I can say that the agile mindset, the learning culture, and the eager to improve, are already there. :)

We continue our journey, improving continuously...

At a company level, we passed for some periods with lack of alignment, but this is changing very fast and seems that in the next months we will have very clear goals at a company level. This is great and can help us with our next challenge... Expand the agile mindset to the rest of the organization...

A final note:

IMHO now that we are capable of creating/evolving a software product, is the right time to expand this culture to the rest of the organization... I see a lot of times that there is a huge effort trying to introduce the agile mindset at the company level but without having a solid technical capacity, and I think that this is a huge mistake for software/tech based companies. In such cases, the story usually goes as follows:  Huge effort to transmit the benefits of the agile mindset, everybody is very excited, the change begins, but the core of the systems is not capable of generating software at a sustainable pace, the technical debt begins to grow without control,  and the product collapse... The cultural change fails, and everybody thinks that Agile doesn't work...

The explanation is easier than that: You don't know how to make software. So first, learn how to make software and later you can introduce other parts of agile. (See: the two pillars of agile).

A culture of Agile software development with a good quality is a prerequisite for introducing Agile at a company level for software based companies.

References:

• http://www.eferro.net/2017/02/honey-badger-team.html
• http://www.eferro.net/2017/07/honey-badger-team-visual-history-ii.html (Part II)
• http://www.eferro.net/2016/06/why-am-i-at-themotion.html
• http://www.eferro.net/2017/03/the-two-pillars-of-agile-software.html
• http://www.eferro.net/2016/08/agile-is-not-recipe-for-success.html
• http://www.eferro.net/2017/07/aos-agile-open-space-2017-segovia.html

Thanks:

• To Jorge Jardines for the fast feedback and some improvements
• To the Honey Badger team :)

Why am I at TheMotion?

originally published at TheMotion.com blog

In recent months, several colleagues asked me why I started working at TheMotion if I was already employed with one of the best development and product teams in the country: development practices consolidated, sane culture and in continuous improvement, XP, DevOps culture, an interesting product with good quality (see Alea Soluciones Bifer team).

For me, the answer was simple: to learn and improve as a professional.

For some time now, I’ve had an interest in scalability at all levels (system scalability, development, teams, culture), distributed systems, cloud-native apps and SaaS.

These challenges only appear in products with a focus on the global market from day one and with features that make the operational cost to grow sublinear in relation to demand, that is, a web-scale business case.

Coincidentally, as I’m in the midst of thought, a colleague of mine, Cesar Ortiz, contacts me and tells me about a company called TheMotion. Initially, I wasn’t so sure about how TheMotion would fit into my plan to explore scalability but after speaking with CEO Iñigo Vega, I understood the huge market opportunity. It was as if a product primed for my area of interest, fell into my lap.


Some consolidated examples of web-scale companies are Facebook, Google, Netflix and Amazon AWS, but unlike working for these companies, working at TheMotion gave me the opportunity to contribute to the technical design, culture and growth of the team, almost from the beginning.

Needless to say, such opportunities are rare, and I couldn’t pass this one up.

From a workplace cultural point of view, the challenges included creating and establishing the correct mindset for experimentation, and developing an environment that nurtured collaboration and continual improvement through high alignment and high autonomy. For an agile culture to work, everybody needs to feel secure (to experiment, fail, learn, repeat), have high trust (to facilitate real collaboration) and value people above all. This is the type of environment which allows for the innovation, collaboration and scalability we need.

Does that sound difficult to achieve? Great because as Cultural Leader, this is my challenge, and I love a good challenge. Fortunately, although the process is difficult and slow, it is also personally rewarding (what’s better than helping people enjoy their work and feel committed to a great project?).

 

From a product point of view, the challenges consist of following the “Lean” principles (Lean Startup, Lean UX) and the step-by-step creation of a great product that delights our customers.

At a deeper, more technical level, the challenges are impressive: reaching a global scale, elasticity, multi-region high availability and, in parallel, growing the technical team.

How am I planning to do this? It will be all about small cycles, feedback and continuous improvement (lean principles and continuous delivery). As the first steps, we are improving our skills in testing, continuous integration, OO design, hexagonal architecture, DevOps practices and culture, native cloud application design, etc. (always keeping in mind the XP practices and values).

What is my responsibility in the technical side? Almost nothing... just the architecture. But when I say architecture I don’t mean as in the classic “Software/System Architect” that works in the ivory tower, making arbitrary technical decisions without being connected to the day to day reality of things. I think of architecture as a shared technical vision, in a constantly evolving state. In this role, my core function is to facilitate the creation of this kind of system and help team members remember our core values: operability, performance, scalability and high-level design. It reminds me of something similar to urban planning and mentoring.

One of the most interesting aspects of working at TheMotion is that we are using the elasticity and capabilities of the cloud from the very beginning. As you can plainly see, this is a perfect challenge for someone like me, eager to learn and continuously improve. And, I get to do all of this with a team which I think would be the envy of any tech-focused company. Life is good.

Learning and improving as a professional, in the right environment, with the right team, building a web-scale product with interesting technical challenges… that is why I’m at TheMotion.

By the way, we are hiring:

• Video Engineer
• Software developer

If you want to be part of this adventure, contact me :)