Showing posts with label development. Show all posts
Showing posts with label development. Show all posts

Saturday, January 03, 2026

Stop Building Software. Start Cultivating It

There's a pervasive anxiety in the software industry. It's the feeling of never being good enough, of working late with the constant worry that what you just shipped will explode in production. It's the pressure to go faster, even when you know you're sacrificing quality. It's the frustration of feeling unprofessional, of never quite reaching a state of sustainable, high-impact work. Many of us have been there, living with this constant, low-grade stress.

But there is a better way. There is a path to professional tranquility that also delivers greater business impact. It doesn't come from a new framework or the latest methodology, but from a fundamental shift in how we think about our work. After nearly three decades in this industry, I've come to rely on five counter-intuitive but powerful mindset shifts.

1. Stop "Building" Software. Start Cultivating It.

One of the most damaging ideas I've encountered in our industry is the metaphor of software development as construction. We talk about "building" applications like we build houses. This metaphor is flawed, and it is the root cause of immense dysfunction.

It's harmful because it implies a static, finished product. A house, once built, is largely done. This mindset separates the "building" phase from a supposedly smaller "maintenance" phase. It leads to the absurd but common belief that software must be thrown away and completely rebuilt every few years.

We need a new metaphor: software as something that evolves, like a garden or a living system. It must be cultivated and guided. The single greatest advantage of software is its malleability, its ability to change. The construction metaphor negates this core strength. We are not masons laying permanent bricks; we are gardeners tending to a system that must constantly adapt to its environment.

If we saw the nature of software as something more evolutionary, as something that is alive, as a system that we are modifying all the time... that metaphor seems to me to be much more in line with the real nature of software development.

2. Your Most Valuable Contribution Might Be the Code You Didn't Write

Our industry often rewards the wrong things. Résumés are filled with lists of technologies used and massive projects "built." Productivity is mistakenly measured by the quantity of code written. More features, more complexity, and more lines of code are seen as signs of progress.

The truth is that true impact often comes from simplification. The most valuable work an engineer can do is often invisible. It's achieving an 80/20 solution that delivers most of the value with a fraction of the effort. It's proposing a simpler path that avoids a six-month project. It's having the courage to delete a feature that adds more cost than value.

Every feature, every line of code, has a "basal cost", an ongoing tax on the system. It adds to the cognitive load for new developers, increases maintenance, and creates friction that slows down all future innovation. The best engineers are masters of preventing unnecessary complexity. Their biggest wins, a disastrous project averted, a legacy system retired, will never appear on a performance review, but they are immensely valuable.

3. Agility Is a Strict Discipline, Not a Free-for-All

The word "Agile" has been misinterpreted to the point of becoming meaningless in many organizations. Teams use "delivering value" as an excuse for shipping shoddy work at high speed. For others, "being agile" has come to mean "anything goes", no documentation, no planning, no rigor.

This interpretation is a complete departure from the concept's original intent. True agility is a difficult and demanding discipline. It is not a shortcut or an excuse for chaos. It is a rigorous commitment to practices that enable sustainable speed and responsiveness.

In my head, agility is equivalent to a discipline, and a difficult one at that... a discipline of "Hey, I'm going to write the test first, then I'll do the implementation, then I'll refactor, I'll even eliminate functionalities that aren't useful." It's actually a tough discipline.

This discipline is embodied in concrete, systematic practices. On the technical side, it means Test-Driven Development (TDD) as the foundation for design and quality, Continuous Integration (CI) and Continuous Delivery (CD) to enable rapid, safe deployment, and continuous refactoring to keep the codebase simple and maintainable. On the product side, it means applying Lean Product Development principles to validate ideas before committing to full implementation, running experiments to test hypotheses, and ruthlessly prioritizing based on real user feedback.

But as the above highlights, it's not just about additive practices. It's also a discipline of subtraction, of proactively controlling technical debt before it becomes a crisis, of simplifying systems even when they're "working," and of removing features that don't add value. The goal is to maintain agility, both from a technical perspective (the codebase remains easy to change) and from a product perspective (the team can pivot based on what they learn).

These are not optional extras; they are the very foundation of agility. They require an uncompromising commitment to quality, because it is only through high quality that we can earn the ability to move fast, adapt, and innovate sustainably over the long term.

4. Individual Performance Metrics Are a Trap

There is a growing and dangerous trend of trying to measure individual developer productivity with simplistic metrics like the number of commits, pull requests, or story points completed. This approach is devastating. In 2023, McKinsey published an article proposing a framework to measure individual developer productivity using metrics like "contribution analysis" and "inner/outer loop time spent." The response from the software engineering community was swift and unequivocal. Kent Beck, creator of Extreme Programming, called it "absurdly naive," while Kent Beck and Gergely Orosz wrote a detailed rebuttal explaining why such frameworks do far more harm than good.

The problem isn't just with McKinsey's specific approach. It's with the entire premise of measuring individuals in a collaborative discipline. This is a direct application of Taylorism, the management philosophy developed by Frederick Winslow Taylor in the early 1900s for factory work. Taylorism treats people as interchangeable resources to be optimized locally, decomposing work into specialized tasks and measuring each person's individual output. It took manufacturing 50 years to move past this thinking. Yet in software development, a creative, knowledge-based discipline where these ideas are least effective, we continue to apply them universally.

When you incentivize local optimization through individual metrics, the negative results are entirely predictable. As W. Edwards Deming taught, over 90% of an organization's performance is a result of the system, not the individuals within it. But individual metrics create perverse incentives that optimize for the wrong things. We reward behaviors that are easy to measure but destructive to the whole: a high number of commits (encouraging smaller, more frequent check-ins regardless of value), celebrating people for "being 100% busy" (even if they're blocking others), or lionizing "heroes" who constantly put out fires (often of their own making). These incentives inevitably lead to chronic bottlenecks, constant rework, and a toxic hero culture where knowledge is hoarded and collaboration is discouraged.

It is devastating because it promotes individualism in a profession that is fundamentally about collaborative problem-solving. It discourages essential practices like pair programming because it makes it harder to assign "credit." It optimizes for busy-ness (output) instead of actual business results (impact). As Beck and Orosz point out, measuring effort and output is easy, but it changes developer behavior in ways that work against the outcomes that actually matter.

Software development is a team sport centered on learning. The true measure of performance is the impact and health of the team. The most valuable person on a team would often score terribly on these individual metrics. They might be the "glue" that holds everyone together, the mentor who elevates the skills of others, the person who prevents bad code from ever being written, or the engineer who just deleted 10,000 lines of obsolete code, making the system simpler for everyone. Their contribution is profound, yet these metrics would render them invisible or, worse, label them a poor performer.

5. Don't Ask for Permission to Be a Professional

Too many engineers wait for permission to do their job properly. They see quality practices like writing automated tests as something they need to negotiate or justify to management. This is a fundamental mistake.

You don't ask your manager for permission to use a for loop or a recursive function; you use the right tool for the job because you are a professional. Writing tests is the same. It is a non-negotiable, foundational part of professional software development, not an optional feature you need to bargain for.

This responsibility extends beyond just testing. Your job is not merely to execute instructions. It is to solve problems. That means taking the initiative to understand the "why" behind a feature, respectfully challenging requirements that don't make sense, and proposing simpler, better solutions. This isn't overstepping; it is the very core of engineering. By taking this professional responsibility, you build trust, earn autonomy, and position yourself to make a real, lasting impact.

Conclusion: A More Sustainable Future

The key to a more sustainable, impactful, and professionally satisfying career is to abandon the "construction" mindset. When we stop thinking of ourselves as builders of static artifacts and start seeing ourselves as cultivators of living, evolving systems, everything changes.

This single shift in perspective is the thread that connects all five of these principles. It leads us to value subtraction over addition, to embrace discipline over chaos, to measure team impact over individual output, and to take ownership of our professional standards. It is the path away from anxiety and toward durable, meaningful work.

The arrival of AI does not invalidate these principles; it reinforces them. If anything is going to change, it's that these mindset shifts will become more critical, not less. When we can generate code faster than ever, distinguishing between building and cultivating becomes more important than ever. When AI can produce features at unprecedented speed, knowing what NOT to build becomes the differentiating skill. The tools evolve, the disciplines adapt, but the goal remains constant: sustainable impact over time, real value delivered, and complexity kept under control. Now we have new tools and evolving disciplines, but they're still pursuing the same fundamental objective.

What is one "construction" habit you can challenge in your team this week to start cultivating your software instead?

References

  • Stop Building Waste: 6 Principles for High-Impact Engineering Teams Eduardo Ferro (2025)
    eferro.net/stop-building-waste
    A complementary piece exploring how to maximize outcomes while minimizing software complexity.
  • Perverse Incentives, Predictable Results: When Your System Sabotages Your Teams Eduardo Ferro (2025)
    eferro.net/perverse-incentives
    Explores how Taylorist thinking creates perverse incentives in software development and offers systemic solutions.
  • Measuring developer productivity? A response to McKinsey Kent Beck and Gergely Orosz (2023)
    Kent Beck's version | Gergely Orosz's version
    A detailed rebuttal to McKinsey's framework, explaining why measuring effort and output damages engineering culture.
  • Yes, you can measure software developer productivity McKinsey (2023)
    mckinsey.com
    The original McKinsey article proposing individual developer productivity metrics.
  • Basal Cost of Software Eduardo Ferro (2021)
    eferro.net/basal-cost-of-software
    Introduces the concept of ongoing cognitive and maintenance costs each feature adds to a system.
  • Extreme Programming Explained: Embrace Change Kent Beck (1999, 2nd Ed. 2004), Addison-Wesley
  • Software has diseconomies of scale – not economies of scale Allan Kelly (2015, revised 2024)
    allankelly.net/archives/472
  • The Most Important Thing Marty Cagan (2020), Silicon Valley Product Group
    svpg.com/the-most-important-thing

Friday, June 20, 2025

YAGNI and the Value of Learning: An Additional Premise

For years, I’ve been applying—almost without realizing it—an extension of the YAGNI principle that I want to share. It’s become part of how we work as a team, a “gut feeling” we’ve refined through experience, and I believe it’s worth making explicit.

Beyond Traditional YAGNI

YAGNI (You Aren't Gonna Need It) is a fundamental principle reminding us not to implement features just because we think we might need them in the future. It's a powerful defense against overengineering and unnecessary complexity.

But there are situations where the premise shifts. Sometimes we know we’re going to need something. It’s not speculation—it’s a reasonable certainty based on product context, business needs, or the natural evolution of the system.

In those cases, our response is not to implement the full solution just because “we know we’ll need it.” Instead, we ask ourselves:

Is there a smaller version of this that lets us learn earlier?

The Value of Learning as a Decision Criterion

The key is to evaluate the learning value of each intermediate step. Not every small step is worth taking—only those that provide meaningful insight into:

  • Actual user behavior
  • The technical feasibility of our approach
  • The validity of our assumptions about the problem
  • The real impact on the metrics we care about

When the cost of that small step is lower than the value of the learning it brings, it’s almost always worth it. This is a practical application of Lean Startup principles to technical development.

Nonlinear Risk: Why Small Steps Matter

There’s another factor reinforcing this approach: risk doesn’t grow linearly with the size of the change. A change that’s twice as big doesn’t carry twice the risk—it carries exponentially more risk.

Small steps allow us to:

  • Catch issues while they’re still manageable and easy to fix
  • Validate assumptions before investing more resources
  • Maintain the ability to pivot without major cost (optionality)
  • Generate more frequent and higher-quality feedback

How We Apply This in Practice

We’re quite radical about this approach. We aim to get product changes to users within 1–1.5 days, and within that cycle, we ship even smaller technical changes to production. These micro-changes give us valuable information about the “how” while we continue refining the “what.”

Our mental process is almost instinctive: whenever a need arises, we consider multiple options—some that others might call “hacky”—and always choose the smallest possible step, no matter how strange it may seem.

We use techniques like Gojko Adzic’s hamburger method to slice functionality, but we go even further. We constantly ask ourselves:

  • “Can we start with a hardcoded version to validate the UX?”
  • “What if we begin with a manually uploaded CSV before building an automated integration?”
  • “Can we simulate this feature with manual config while we learn the real flow?”
  • “What if we do it just for one user or a specific case first?”

This isn’t about being naive about future needs. It’s about being smart about how we get there. Each micro-step gives us signals about whether we’re going in the right direction, both technically and functionally. And when something doesn’t work as expected, the cost to pivot is minimal.

This obsession with the smallest possible step not only reduces risk, it also accelerates real learning about the problem we’re solving and the behavior of the solution we’re implementing.

Connection with Other Premises

This way of working naturally aligns with other guiding principles in our approach:

  • Postpone decisions: Small steps allow us to delay irreversible choices until we have more information
  • Small safe steps: We work incrementally to reduce risk and increase learning
  • Software as a means: We focus on impact, not on building the most complete solution upfront
  • Optimize for feedback: We prioritize fast learning over perfect implementation, because we know we don’t have all the answers—we need to discover them

A Premise in Evolution

Like all the premises we use, this isn’t universal or applicable in every context. But in software product development, where uncertainty is high and the cost of mistakes can be significant, it has proven extremely valuable.

It’s part of our default way of working: we always look for the smallest step that lets us learn something useful before committing to the full step. And when that learning has value, it’s almost always worth the detour.

Have you experienced something similar in your work? How do you evaluate the trade-off between implementing something fully and taking intermediate steps to learn?

References

Friday, December 06, 2024

Focus on Verbs, Not Names: A Strategy for Better System Design

In my experience, the key to deeply understanding a system or product lies in focusing on behaviors—the actions, flows, and events that drive its operation. Prioritize identifying verbs over names. Here’s why this approach works and how it can transform your design process.

Start with Behaviors

Shift your analysis from “What is this thing?” to “What does it do?”. Focus your research and conversations on:

  • Identifying actions and business flows.
  • Understanding dependencies and concurrency.
    • What depends on what?
    • Which actions can happen in parallel?
    • What triggers or informs each behavior?

When you analyze behaviors, you uncover the dynamic interactions users have with your system. This focus naturally aligns with designing systems as collections of small, independent pieces that encapsulate state and communicate through messages—perfect for paradigms like OOP, Actors, and Microservices.

The Pitfall of Focusing on Names

Many teams fall into the trap of identifying entities (names) first, which often results in anemic, static models disconnected from real-world dynamics. This approach, while intuitive, neglects the rich context of flows, dependencies, and rules. Misunderstanding Object-Oriented Analysis (OOA) in this way often leads to systems that lack expressiveness and scalability.

Behaviors as the Foundation of Value

Remember, customers don’t value software for its own sake. Software is a liability, not an asset. The true asset lies in the actions and outcomes your system enables. Identifying behaviors first ensures your design delivers meaningful value to the customer by focusing on what they actually need: actions, not abstractions.

Scalability Through Behaviors

Focusing on behaviors reveals the concurrent nature of the real world. Systems that prioritize names struggle to address concurrency, parallelism, and scalability. By contrast, analyzing actions and flows allows you to design systems that are naturally reactive and distributed. At higher levels, this approach helps define bounded contexts, domain events, and microservices. At lower levels, it aids in designing concurrent and scalable services.

Event Storming: A Behavioral Lens

Techniques like Event Storming are powerful tools for identifying domain events, dependencies, and key behaviors. They bring the behavioral focus to life, helping teams collaboratively uncover what drives their system and how it should respond.



Conclusion

Identifying entities (names) has its place, but behaviors (verbs) are more critical. A system’s essence lies in what it does, not what it’s called. Adopting a behavior-first approach ensures you design systems that are adaptive, scalable, and valuable to customers.

In the end, this mindset reflects a simple truth: development is always iterative. By continuously refining our understanding of behaviors, we build systems that evolve gracefully with the needs of their users.

Notes:

Tuesday, July 16, 2024

El coste basal del software

Traduccion del articulo original Basal Cost of Software
Traduccion realizada por https://x.com/simonvlc y publicada originalmente en su genial newsletter Estrategia de Producto: El Coste Basal del Software
---

En mi opinión, el coste de desarrollo de cada funcionalidad en un producto puede dividirse de la siguiente manera:

  1. El coste directo de desarrollo o coste inicial.
  2. Un coste semanal o mensual únicamente asociado con su existencia en el sistema. Haciendo una comparación con la Tasa Metabólica Basal del cuerpo humano, podríamos llamar a este segundo coste la Tasa Metabólica Basal o Coste Basal.

El Coste Basal se compone de dos partes diferenciadas:

  • El impacto directo en la capacidad del equipo debido a la complejidad añadida (nuevas dependencias, más código para entender, más posibilidades de bugs ocultos, etc.).
  • El impacto en el coste de desarrollo o evolución de otras funcionalidades debido a posibles incompatibilidades, acoplamiento, etc.

El coste inicial

Es el coste incurrido por el equipo durante el desarrollo inicial de la funcionalidad. Incluye el coste desde que el equipo comienza a trabajar en la funcionalidad hasta que el cliente la tiene disponible y comienza a usarla. Por supuesto, este proceso debería consistir en múltiples despliegues y lanzamientos parciales para obtener feedback y hacer los ajustes necesarios...

El Coste Basal

Tras el coste inicial de desarrollo, afrontamos también un coste continuo que reduce la capacidad del equipo para el desarrollo de nuevas funcionalidades (innovación).

Este coste, que continúa con el tiempo y solo termina con la eliminación de la funcionalidad o el fin de vida del producto, es el coste que paga el equipo por la existencia de ese código en el producto.

Es importante tener en cuenta que este no se refiere al coste de hacer modificaciones o corregir errores; se refiere al coste de simplemente tener el código allí...

¿Por qué se da este coste? ¿Por qué una funcionalidad que no evoluciona supone un coste? 

  • El equipo tiene que conocer ese código (dónde está, qué dependencias tiene, qué interactúa con él, cómo está diseñado...).
  • El conocimiento y las técnicas del equipo están en constante evolución. Cuando mejora en cualquiera de estos aspectos, deberá actualizar el código.
  • Cuando el equipo diseña una nueva funcionalidad, el código debe ser diseñado de manera que sea compatible con todas las anteriores y no genere problemas o regresiones. Y, por supuesto, este coste es proporcional a todas las que ya teníamos en el sistema.
  • Algo similar ocurre cuando un nuevo miembro se une al equipo. Este nuevo integrante debe aprender sobre todas las funcionalidades, por lo que el coste es proporcional a su número..

Y lo peor de todo, este sobrecoste es continuo hasta la "muerte" de la funcionalidad. Por ejemplo, hasta el fin de la vida del producto, hasta que ningún usuario la use, o hasta el fin del mundo (lo que ocurra primero).

Evolución del coste de una funcionalidad

Como hemos visto, el Coste Basal de una funcionalidad es más o menos constante durante su vida. Pero cada lenguaje, dependencia o tecnología que hayamos usado en su desarrollo, puede alcanzar un punto en el que ya no es utilizable por cualquier razón (dependencias obsoletas, problemas de seguridad, evolución normal que deprecia la versión del lenguaje que usamos, etc.). Desde este momento, el coste puede dispararse porque estamos obligados a actualizarlo, incluso aunque no queramos evolucionar la funcionalidad.

En resumen, el coste real puede ser representado por el siguiente gráfico:

El problema

Un error común es negar el Coste Basal y considerar que si no se hacen cambios, el coste de mantenimiento de una funcionalidad es cero. Supongo que esto proviene de utilizar metáforas de otras profesiones como la construcción, que no son aplicables al desarrollo de software.


La capacidad no es infinita

A pesar de que la capacidad de un equipo cambia (positiva o negativamente) con el paso del tiempo debido a distintos factores (conocimiento del negocio, técnicas, estrés…), esta es finita.

El Coste Basal acumulado de todas las funcionalidades de las que el equipo es responsable reduce la capacidad disponible para desarrollar nuevas funcionalidades (innovar).

Con el tiempo, podemos observar como la capacidad del equipo para innovar decae rápidamente:

Hasta el punto de que cuando la capacidad se agota, el equipo se encuentra en una situación en la que le es imposible innovar porque pasa todo su tiempo “manteniendo” las funcionalidades de las que es responsable.


O bien reduce su velocidad al desarrollar nuevas funcionalidades, acumulando coste basal, pero a menor ritmo.

Conclusiones

Las secciones anteriores destacan varios principios que debemos considerar para mejorar la eficiencia de nuestros equipos de producto.

  • Debemos minimizar el Coste Basal tanto como sea posible, logrando el impacto deseado con la menor cantidad de código posible.
    • Cuando sea posible, incluso sin añadirlo.
    • Iterando la funcionalidad, partiendo de una solución mínima, para adaptarla lo más posible a las necesidades del usuario.
    • Haciendo el software/diseño lo más simple posible. Debe ser fácil de evolucionar sin sobreingeniería (YAGNI).
  • Debemos eliminar cualquier funcionalidad del producto que no tenga el impacto deseado, eliminando así su Coste Basal.
  • Debemos monitorizar continuamente el código del equipo para detectar la obsolescencia de dependencias, lenguajes y tecnologías para evitar que el Coste Basal se dispare.

Veamos un ejemplo teórico sencillo para ayudar a transmitir el mensaje. Supongamos que un equipo acumula un 1% de Coste Basal por semana, lo que significa que, por cada 100 unidades de producto (código, características, mejoras), añade 1 unidad de Coste Basal. A medida que este se acumula, en 52 semanas (aproximadamente 1 año), el equipo solo podrá dedicar el 60% de su capacidad a nuevas funcionalidades/mejoras (innovación). En dos años, será apenas el 35%. Por supuesto, este ejemplo es una simplificación, pero destaca que negar este coste no es una opción viable.


Recordad: La Simplicidad — el arte de maximizar el trabajo no hecho — es esencial.

Relacionado y referencias

Otros artículos interesantes


Saturday, January 06, 2024

Be humble, no Rockstars allowed

Continuous delivery is based on the idea that when dealing with uncertainty and complex problems (as described in the Cynefin framework), the best approach is to assume we don’t already know the solution. Instead, we must commit to continuously learning about both the problem and the optimal solution. Moreover, continuous delivery encourages us to implement systems that effectively manage errors—because we are bound to make mistakes—while minimizing their impact.

If we assume that we do know the solution, or if the problem is straightforward (or just "complicated" within the Cynefin framework), continuous delivery and iterative learning might not be the most efficient approach. In such cases, a direct solution delivered by a team with prior experience can work better. However, in this scenario, don’t expect to gain new insights or improve the process further. And if the problem has already been solved before, reproducing the solution could simply be wasteful—something we should avoid at all costs.

As Kent Beck aptly put it:

"The only way it's all going to go according to plan is if you don't learn anything."

My experience and context


In my case:

  • I assume that the solutions I come up with are rarely the best ones.
  • I acknowledge that we (myself included) make frequent mistakes—even with something as simple as writing "Hello, World" correctly on the first try.
  • I recognize that we still have a lot to learn—about the domain, the business, tools, needs, and everything else.
  • I have never worked in a simple or purely complicated context where a ready-made solution was available. And if I ever did encounter such a case, I would avoid reinventing the wheel to minimize waste.
Within this context, the discipline of lean product development, coupled with practices like continuous delivery (focusing on continuous flow), continuous learning, and sustainable quality, remains the best strategy I know.

For Self-Sufficient Experts

If you consider yourself an expert in everything, someone who rarely makes mistakes and feels like a 'rockstar' in their field, you might believe that you don’t need continuous delivery. I suspect that humility and the recognition that EVERYONE (including yourself) makes mistakes might be lacking in this case. Perhaps I’m wrong, and you’re some mythological being who never errs and doesn’t need to learn continuously. If that’s the case, my apologies… :)

However, I won’t be able to work with you since I do make mistakes and need to be part of a team that uses techniques enabling us to develop products despite our errors, all without living in constant stress.

In summary

I believe the general trick of this profession is to work as a team in the best way we know how, using the best techniques we have, and still assuming that:

  • We are going to make mistakes.
  • We have to keep learning.
  • We need to improve continuously.
  • We need a level of quality that allows us to work sustainably.

The discipline of continuous delivery (CD, TBD, TDD, etc.) provides all of that for me.

Please...

  • Be a great team player. Collaborate and get involved.
  • Create a safe learning environment for your team.
  • Program as well as you can but assume you will fail, so build an environment where failing is manageable and has minimal impact on customers.
  • Make the best architectural decisions you can, but assume they will need to evolve and improve over time.
  • Seek continuous delivery as a goal—it fosters the right discipline and creates a safe, sustainable work environment.
  • Be humble and accept failure, uncertainty, and your own limitations.


I don’t want to work with Rockstars, nor with x10 developers… I want to work in x10 teams and environments where learning is constant, questioning is welcomed, errors are accepted (because they are low-cost), and improvement is continuous.

Remember:

We need professionals, not heroes.

“I’m not a great programmer; I’m just a good programmer with great habits.” — Kent Beck

Sunday, December 31, 2023

Low friction cli tools using docker

Although I last wrote about technical topics here a while ago, I would like to share some ideas that are helping me reduce friction in creating tools for my colleagues at ClarityAI.
As the Platform Lead, among other things, I am involved in the development experience, which means that my team creates tooling (as part of our platform) for the rest of the engineering and data science teams in the company. This tooling includes command-line tools that help users use our platform with minimal friction and autonomously (without requiring support).
In our case, this means we want command-line tools with the following characteristics:

  • Usable on Linux and Mac
  • Low friction for usage (easy installation and usage)
  • Easy to evolve
  • Easy to distribute to all our users
  • Few external dependencies so that our users don't have to install additional tools

In this context, and considering that we are already heavy users of Docker, it is a good idea to implement these types of tools as Docker images that our users can use.
This would allow us to include any external tools our users may need in the Docker images and eliminate friction by not requiring them to install those tools.

At the same time, using Docker presented its own challenges:

  • Difficulty in distributing a Docker image from a private repository
  • Difficulty in usage for execution requiring many parameters (volumes, environment variables, execution parameters and arguments, etc.)

Below, I will explain how we have solved (or are in the process of solving) each of the mentioned challenges.

Controlled Environment

Any modern development platform is implemented using different technologies and solutions.
This means the user must use different clients for each of these technologies (kubectl, AWS CLI, Git, Teleport, etc.).
To eliminate friction, avoid installation problems, and reduce the number of potential issues due to misuse or using incorrect versions, what has worked very well for us is to include all these tools (with controlled versions and environment) in a Docker image along with code to control their usage and configuration.
This allows the platform team to forget about problems caused by using the wrong version of kubectl or incorrect AWS client configuration.

Installation

If the installation process of a tool is complicated, you can be sure that a significant percentage of users will encounter problems and require support.
It is often assumed that good documentation is sufficient, but no matter how good the documentation is, if there are many steps to follow and many tools to install, there will be problems.
A better solution is to create a specific installer without configuration and with the fewest possible dependencies.
In our case, the tool to be distributed was built with Docker and stored in a private repository (AWS ECR).
As a lean team, we have iteratively improved the solution, reducing friction at each step:

  1. Installation instructions are documented in the engineering handbook. It depends on Docker, properly configured AWS CLI, and knowledge of ECR.
  2. Installer script in the code repository. It improved the experience and generated a helper script for future usage. It depends on Docker and properly configured AWS CLI but does not require knowledge of ECR.
  3. Cross-platform installer implemented with Golang. This is the version currently in development. It only requires Docker, and the user has an AWS Key. (Example: ECR Docker image puller / installer)

Execution

When we create an open-source solution or a product that will be used in different environments, the ability to configure and adapt it to different environments is crucial.
But when we develop a tool for our colleagues and our goal is to minimize friction and reduce cognitive load, we need to provide the minimum necessary flexibility.
We can even avoid the need for any parameters. 
This becomes a challenge when you have a tool built with Docker that requires multiple credentials and the ability to modify files on the user's machine, as it requires running Docker with many parameters and environment variables.
Once again, iteration has allowed us to improve the solution. The steps we have taken are:
  1. Document all the parameters and environment variables for tool execution in the README. This serves us well for development, but we never intended for our users to use the tool following this documentation.
  2. Using an execution script that hides all the parameters and variables. In our case, this execution script was generated by the installer.
  3. Cross-platform wrapper implemented in Golang. This allows us to distribute a binary that only depends on having Docker and hides the tool's complexity. This is the version we will distribute soon. (Example: Simple wrapper to execute (interactive/non interactive) docker image)

Continuous Updating

In addition to facilitating installation, it is essential that the platform team can quickly push new versions. In this case, having Docker already facilitates this because simply uploading new versions to the Docker repository makes them available to our users.
On the other hand, to streamline the update process, what has worked well for us is to include an option in the execution script/wrapper to check and download new versions.
Initially, we made it so that each execution would check for new versions, but we found that it caused too much delay during startup, so now we prefer to let the user decide when to update.


Conclusions:

  • Docker allows packaging applications with specific versions in a controlled environment.
  • Creating a cross-platform wrapper in Go is an excellent solution to eliminate friction in the distribution and execution of tools implemented using Docker.
  • Investing in making installation and updating simple allows tools to evolve rapidly without generating much friction.


Wednesday, November 01, 2023

Harnessing Efficiency: The Theory of Constraints in Software Development



This micro post was previously published at linkedin

Navigating the multi-faceted domain of software development often presents a series of bottlenecks that could hinder project momentum and delivery timelines. The Theory of Constraints (TOC) serves as a beacon, guiding teams to identify, address, and overcome these bottlenecks, thereby unlocking a pathway to streamlined processes and enhanced productivity.

Here's a snapshot of how TOC unfolds in software development:
  1.  Identify the most important Constraint: Pinpoint the process, resource, or technology bottleneck obstructing progress (lack of quality, knowledge silos, convoluted deployment process, individual ownership about parts of the code or processes, etc.).
  2. Exploit the Constraint: Maximize the efficiency of the identified constraint without additional resources (reducing the WIP, redirecting people to help with the bottleneck, etc.).
  3. Subordinate Everything Else to the Constraint: Ensure all other processes are aligned to support and work around the constraint.
  4. Elevate the Constraint: Allocate necessary resources to alleviate or eliminate the constraint, promoting better throughput (investing in automation, test automation, pair/ensemble programming to spread knowledge, feature toggles to reduce risk, etc.).
  5. Repeat the Process: Embark on a cycle of continuous identification and resolution of constraints to foster a culture of ongoing improvement.

For a deeper dive into TOC and its application in IT landscapes, 'The Goal' by Eliyahu M. Goldratt and 'The Phoenix Project' by Gene KimKevin Behr, and George. Spafford are essential reads.

Using the Theory of Constraints (TOC) can help software development professionals and teams deal with problems smartly. It helps turn these problems into opportunities for improvement, growth and faster delivery.

Indeed, the Theory of Constraints relates to Lean Software Development and agile software development methodologies.

The lightning talk "Stop Starting and Start Finishing" by Jason Yip is one of the best explanations I know for understanding how to use TOC and other Lean ideas to optimize the sustained delivery flow of a software development team.

#TheoryOfConstraints #SoftwareDevelopment #ContinuousImprovement #TheGoal #LeanSoftwareDevelopment

The virtuous loop of software development



This micro post was previously published at linkedin

The ultimate aim is Continuous Delivery (CD), a goal that enables fast flow with rapid iterations and continuous feedback. At the same time, this goal promotes technical excellence and good design.

Continuous Integration (CI) is required (integrating at least once a day, also known as Trunk Based Development), along with a strong focus on Test-Driven Development (TDD) or other similar practices to ensure high confidence and emphasis on excellent and simple design. This approach is closely linked to Extreme Programming and the DevOps mindset, which emphasizes collaboration and continuous improvement. By following these principles, software development teams can enhance their efficiency and deliver high-quality products to customers.


Here are some related resources that you might find interesting:


Sunday, October 15, 2023

The Art of Small Steps in Software Development: A Lean Vision

Have you ever played Tetris? If so, you know that every decision, every piece, and every move counts. The same applies to software development. Every choice, every line of code, and every interaction influences the final outcome. This post will explore how Lean software development and the "safe small steps" approach can simplify complexity and create efficient systems.

Small Steps: The Simplicity of Tetris

Just like in Tetris, where we choose where to place small pieces to complete lines, in software development, we continuously make decisions about how to build our product. These decisions, when made in small increments, allow us to:

  • Iterate quickly: Fixes and adjustments are simpler.
  • Reduce risks: We limit potential errors at each stage.
  • Understand better: Each step is understood and analyzed more clearly.

Principles of Lean Software Development

Based on Lean philosophy, Lean software development focuses on maximizing customer value and minimizing waste. Some of the key principles include:

  • Reduce waste: Eliminate the unnecessary and focus on the essentials. Every step must add value or increase the internal or external quality of the product.
  • Decide as late as possible: Be flexible and adapt to changes. See Lean Software development: The art of postponing decisions.
  • Deliver quickly: Allows for feedback and adaptation to customer needs.
  • Build with integrity and quality: Ensures a reliable and long-lasting product. This is key to maintaining speed sustainably over time.
  • Proactively simplify: Avoid unnecessary complexity and keep the system manageable. Simplification also adds value, protecting the current value and avoiding future costs.

Small Steps: The Simplicity and Continuity of Tetris

Tetris, one of the most iconic video games in history, presents us with a continuous and infinite challenge. In it, pieces of different shapes fall from the top of the screen at an increasing speed. Our task is to accommodate these pieces efficiently at the bottom, trying to fit them so that they complete horizontal lines. Completing these lines eliminates them, granting us points and, more importantly, space to continue playing.

In Tetris, value accumulates by adding points, piece by piece, line by line. However, the real challenge and essence of the game lie in balancing the accumulation of that value (points) while managing increasing complexity (the accumulated pieces). If we allow the pieces to pile up and reach the top of the screen, we lose the game.

Similarly, in software development, we continuously decide how to build our product. Like in Tetris, we must accumulate value (features and characteristics) while managing and keeping complexity at bay. These decisions, when made in small increments or "steps," allow us to:

  • Iterate quickly: Fixes and adjustments are simpler.
  • Reduce risks: We limit potential errors at each stage.
  • Understand better: Each step is understood and analyzed more clearly.

Tetris / Larger vs Smaller pieces

Just like playing with large pieces in Tetris, working with large steps in software development makes the game/work much more difficult and stressful.

Conclusion

Software development is not just about writing code; it's about making informed and continuous decisions to create valuable products. Just like in Tetris, every piece, every step, counts. Embrace the Lean philosophy and focus on safe small steps to simplify, adapt, and deliver with quality.

If this approach has intrigued you, I invite you to delve deeper into Lean software development and experience how it can transform your development process.

Related: