Saturday, January 06, 2024

Be humble, no Rockstars allowed

Continuous delivery is based on the idea that when dealing with uncertainty and complex problems (as described in the Cynefin framework), the best approach is to assume we don’t already know the solution. Instead, we must commit to continuously learning about both the problem and the optimal solution. Moreover, continuous delivery encourages us to implement systems that effectively manage errors—because we are bound to make mistakes—while minimizing their impact.

If we assume that we do know the solution, or if the problem is straightforward (or just "complicated" within the Cynefin framework), continuous delivery and iterative learning might not be the most efficient approach. In such cases, a direct solution delivered by a team with prior experience can work better. However, in this scenario, don’t expect to gain new insights or improve the process further. And if the problem has already been solved before, reproducing the solution could simply be wasteful—something we should avoid at all costs.

As Kent Beck aptly put it:

"The only way it's all going to go according to plan is if you don't learn anything."

My experience and context


In my case:

  • I assume that the solutions I come up with are rarely the best ones.
  • I acknowledge that we (myself included) make frequent mistakes—even with something as simple as writing "Hello, World" correctly on the first try.
  • I recognize that we still have a lot to learn—about the domain, the business, tools, needs, and everything else.
  • I have never worked in a simple or purely complicated context where a ready-made solution was available. And if I ever did encounter such a case, I would avoid reinventing the wheel to minimize waste.
Within this context, the discipline of lean product development, coupled with practices like continuous delivery (focusing on continuous flow), continuous learning, and sustainable quality, remains the best strategy I know.

For Self-Sufficient Experts

If you consider yourself an expert in everything, someone who rarely makes mistakes and feels like a 'rockstar' in their field, you might believe that you don’t need continuous delivery. I suspect that humility and the recognition that EVERYONE (including yourself) makes mistakes might be lacking in this case. Perhaps I’m wrong, and you’re some mythological being who never errs and doesn’t need to learn continuously. If that’s the case, my apologies… :)

However, I won’t be able to work with you since I do make mistakes and need to be part of a team that uses techniques enabling us to develop products despite our errors, all without living in constant stress.

In summary

I believe the general trick of this profession is to work as a team in the best way we know how, using the best techniques we have, and still assuming that:

  • We are going to make mistakes.
  • We have to keep learning.
  • We need to improve continuously.
  • We need a level of quality that allows us to work sustainably.

The discipline of continuous delivery (CD, TBD, TDD, etc.) provides all of that for me.

Please...

  • Be a great team player. Collaborate and get involved.
  • Create a safe learning environment for your team.
  • Program as well as you can but assume you will fail, so build an environment where failing is manageable and has minimal impact on customers.
  • Make the best architectural decisions you can, but assume they will need to evolve and improve over time.
  • Seek continuous delivery as a goal—it fosters the right discipline and creates a safe, sustainable work environment.
  • Be humble and accept failure, uncertainty, and your own limitations.


I don’t want to work with Rockstars, nor with x10 developers… I want to work in x10 teams and environments where learning is constant, questioning is welcomed, errors are accepted (because they are low-cost), and improvement is continuous.

Remember:

We need professionals, not heroes.

“I’m not a great programmer; I’m just a good programmer with great habits.” — Kent Beck

Sunday, December 31, 2023

Low friction cli tools using docker

Although I last wrote about technical topics here a while ago, I would like to share some ideas that are helping me reduce friction in creating tools for my colleagues at ClarityAI.
As the Platform Lead, among other things, I am involved in the development experience, which means that my team creates tooling (as part of our platform) for the rest of the engineering and data science teams in the company. This tooling includes command-line tools that help users use our platform with minimal friction and autonomously (without requiring support).
In our case, this means we want command-line tools with the following characteristics:

  • Usable on Linux and Mac
  • Low friction for usage (easy installation and usage)
  • Easy to evolve
  • Easy to distribute to all our users
  • Few external dependencies so that our users don't have to install additional tools

In this context, and considering that we are already heavy users of Docker, it is a good idea to implement these types of tools as Docker images that our users can use.
This would allow us to include any external tools our users may need in the Docker images and eliminate friction by not requiring them to install those tools.

At the same time, using Docker presented its own challenges:

  • Difficulty in distributing a Docker image from a private repository
  • Difficulty in usage for execution requiring many parameters (volumes, environment variables, execution parameters and arguments, etc.)

Below, I will explain how we have solved (or are in the process of solving) each of the mentioned challenges.

Controlled Environment

Any modern development platform is implemented using different technologies and solutions.
This means the user must use different clients for each of these technologies (kubectl, AWS CLI, Git, Teleport, etc.).
To eliminate friction, avoid installation problems, and reduce the number of potential issues due to misuse or using incorrect versions, what has worked very well for us is to include all these tools (with controlled versions and environment) in a Docker image along with code to control their usage and configuration.
This allows the platform team to forget about problems caused by using the wrong version of kubectl or incorrect AWS client configuration.

Installation

If the installation process of a tool is complicated, you can be sure that a significant percentage of users will encounter problems and require support.
It is often assumed that good documentation is sufficient, but no matter how good the documentation is, if there are many steps to follow and many tools to install, there will be problems.
A better solution is to create a specific installer without configuration and with the fewest possible dependencies.
In our case, the tool to be distributed was built with Docker and stored in a private repository (AWS ECR).
As a lean team, we have iteratively improved the solution, reducing friction at each step:

  1. Installation instructions are documented in the engineering handbook. It depends on Docker, properly configured AWS CLI, and knowledge of ECR.
  2. Installer script in the code repository. It improved the experience and generated a helper script for future usage. It depends on Docker and properly configured AWS CLI but does not require knowledge of ECR.
  3. Cross-platform installer implemented with Golang. This is the version currently in development. It only requires Docker, and the user has an AWS Key. (Example: ECR Docker image puller / installer)

Execution

When we create an open-source solution or a product that will be used in different environments, the ability to configure and adapt it to different environments is crucial.
But when we develop a tool for our colleagues and our goal is to minimize friction and reduce cognitive load, we need to provide the minimum necessary flexibility.
We can even avoid the need for any parameters. 
This becomes a challenge when you have a tool built with Docker that requires multiple credentials and the ability to modify files on the user's machine, as it requires running Docker with many parameters and environment variables.
Once again, iteration has allowed us to improve the solution. The steps we have taken are:
  1. Document all the parameters and environment variables for tool execution in the README. This serves us well for development, but we never intended for our users to use the tool following this documentation.
  2. Using an execution script that hides all the parameters and variables. In our case, this execution script was generated by the installer.
  3. Cross-platform wrapper implemented in Golang. This allows us to distribute a binary that only depends on having Docker and hides the tool's complexity. This is the version we will distribute soon. (Example: Simple wrapper to execute (interactive/non interactive) docker image)

Continuous Updating

In addition to facilitating installation, it is essential that the platform team can quickly push new versions. In this case, having Docker already facilitates this because simply uploading new versions to the Docker repository makes them available to our users.
On the other hand, to streamline the update process, what has worked well for us is to include an option in the execution script/wrapper to check and download new versions.
Initially, we made it so that each execution would check for new versions, but we found that it caused too much delay during startup, so now we prefer to let the user decide when to update.


Conclusions:

  • Docker allows packaging applications with specific versions in a controlled environment.
  • Creating a cross-platform wrapper in Go is an excellent solution to eliminate friction in the distribution and execution of tools implemented using Docker.
  • Investing in making installation and updating simple allows tools to evolve rapidly without generating much friction.


Wednesday, November 15, 2023

Updated My Personal Mission. Same Essence, Different Words

 

Eduardo Ferro's Personal Mission

Cultivate Happiness: Foster personal joy and spread happiness to those around me, starting with family and friends, while continuously expanding my circle of influence.

Methods of Achievement

  • Responsibility and Respect: Uphold personal accountability and respect for others in all interactions.
  • Balance Seeking: Strive for a harmonious balance in personal life and contribute towards global equilibrium.
  • Authenticity: Remain true to myself in every aspect of life, including professional pursuits.
  • Value Generation: Utilize my abilities to create both intrinsic and economic value, mindful of social and ethical considerations.
  • Win-Win Scenarios: Continually seek outcomes that are mutually beneficial.
  • Continuous Learning: Embrace lifelong learning, both personally and professionally.
  • Conscious Decision-Making: Choose my reactions and actions thoughtfully.

Ethical Stance

Selective Engagement: Refrain from involvement with businesses that conflict with my values, including:
  • Financial enterprises that prioritize profit over social and environmental responsibility.
  • Weapon manufacturers and the military industry.
  • Casinos and gambling enterprises.
  • Entities involved in the abuse or exploitation of humans, animals, or the environment.
  • Businesses that negatively impact the lives of stakeholders.

Vision for the Future

I believe we are at a pivotal moment where our actions can significantly impact the planet, both positively and negatively. It's crucial to remember this responsibility at all times. We are amidst a revolution that blends collaboration, knowledge, and a federation of power. Networks are becoming the structures of the future, and their growth should be both sustained and sustainable.

Professional Alignment

My focus is on engaging with green and teal organizations, where people are valued as the primary asset. I am drawn to environments that prioritize sustainable and ethical practices, recognizing the importance of human capital in driving positive change.

Personal Commitment

I engage with the world passionately and responsibly, yet always at a measured pace, ensuring a balanced approach in all my endeavors.

Sunday, November 12, 2023

Learning lean/Agile concepts / short videos

 


🆕 Updated on June 24, 2025 — Added a new section on Systems Thinking and Quality with the classic Red Bead Experiment by W. Edwards Deming.

Yesterday, my colleague Cristina Verdi asked for interesting resources to help introduce Lean Software / Product Development concepts.
I passed her a series of short videos that I always use to illustrate certain related concepts. I'm posting them here in case they can help someone.

Resource Efficiency vs Flow Efficiency:
Systems Thinking and Quality:
  • The Red Bead Experiment (14m) by W. Edwards Deming — A powerful demonstration that performance and quality depend on the system, not individual effort. A reminder that systemic issues require systemic fixes.

Wednesday, November 01, 2023

Harnessing Efficiency: The Theory of Constraints in Software Development



This micro post was previously published at linkedin

Navigating the multi-faceted domain of software development often presents a series of bottlenecks that could hinder project momentum and delivery timelines. The Theory of Constraints (TOC) serves as a beacon, guiding teams to identify, address, and overcome these bottlenecks, thereby unlocking a pathway to streamlined processes and enhanced productivity.

Here's a snapshot of how TOC unfolds in software development:
  1.  Identify the most important Constraint: Pinpoint the process, resource, or technology bottleneck obstructing progress (lack of quality, knowledge silos, convoluted deployment process, individual ownership about parts of the code or processes, etc.).
  2. Exploit the Constraint: Maximize the efficiency of the identified constraint without additional resources (reducing the WIP, redirecting people to help with the bottleneck, etc.).
  3. Subordinate Everything Else to the Constraint: Ensure all other processes are aligned to support and work around the constraint.
  4. Elevate the Constraint: Allocate necessary resources to alleviate or eliminate the constraint, promoting better throughput (investing in automation, test automation, pair/ensemble programming to spread knowledge, feature toggles to reduce risk, etc.).
  5. Repeat the Process: Embark on a cycle of continuous identification and resolution of constraints to foster a culture of ongoing improvement.

For a deeper dive into TOC and its application in IT landscapes, 'The Goal' by Eliyahu M. Goldratt and 'The Phoenix Project' by Gene KimKevin Behr, and George. Spafford are essential reads.

Using the Theory of Constraints (TOC) can help software development professionals and teams deal with problems smartly. It helps turn these problems into opportunities for improvement, growth and faster delivery.

Indeed, the Theory of Constraints relates to Lean Software Development and agile software development methodologies.

The lightning talk "Stop Starting and Start Finishing" by Jason Yip is one of the best explanations I know for understanding how to use TOC and other Lean ideas to optimize the sustained delivery flow of a software development team.

#TheoryOfConstraints #SoftwareDevelopment #ContinuousImprovement #TheGoal #LeanSoftwareDevelopment

The virtuous loop of software development



This micro post was previously published at linkedin

The ultimate aim is Continuous Delivery (CD), a goal that enables fast flow with rapid iterations and continuous feedback. At the same time, this goal promotes technical excellence and good design.

Continuous Integration (CI) is required (integrating at least once a day, also known as Trunk Based Development), along with a strong focus on Test-Driven Development (TDD) or other similar practices to ensure high confidence and emphasis on excellent and simple design. This approach is closely linked to Extreme Programming and the DevOps mindset, which emphasizes collaboration and continuous improvement. By following these principles, software development teams can enhance their efficiency and deliver high-quality products to customers.


Here are some related resources that you might find interesting: