Perverse Incentives, Predictable Results: When Your System Sabotages Your Teams

It's a strange paradox. It took the manufacturing industry 50 years to move past the ideas of Frederick Winslow Taylor, known as Taylorism or Scientific Management. Yet in software development, a creative, knowledge-based discipline where these ideas are least effective, we continue to apply them universally.

Taylorism in a software context is the practice of maximizing the local utilization of each individual. We treat people like resources in a factory, keeping them 100% busy with their specialized tasks. This creates a cascade of perverse incentives that actively undermine collaboration, destroy flow, and cripple the efficiency of the entire system.

This article explores the three most common traps this thinking creates and offers a practical, systemic path toward improvement.

1. The Trap of Resource Efficiency

The Core Fallacy

Most organizations are obsessed with individual performance and 100% resource utilization. The goal is to keep every person busy all the time, believing this equates to maximum productivity. However, basic queueing theory teaches us that running any system at full capacity only generates blockages, chronic bottlenecks, and massive wait times. As a wise colleague once said, "If we optimize locally, in most cases we make everything worse."

The Connection to Taylorism

This is a direct application of Taylorism: maximizing the utilization of each "human resource" locally. We decompose work, assign it to specialists, and measure them on their individual output, ignoring the devastating impact on the system as a whole.

The Consequences

When you incentivize local optimization, the negative results are entirely predictable.

• Perverse local incentives: We reward metrics that are easy to measure but destructive to the whole. This includes rewarding a high number of commits, celebrating people for "being 100% busy," or lionizing "heroes" who constantly put out fires.
• Predictable negative results: These incentives inevitably lead to chronic bottlenecks, constant rework (also known as "failure demand" because you're fixing things that failed the first time), and a toxic hero culture where knowledge is hoarded and collaboration is discouraged.

The Alternative: Flow Efficiency

The antidote to this trap is shifting our focus from resource efficiency to flow efficiency. The book This is Lean explains the difference well.

• Resource Efficiency: This is a process where work is handed off between specialists in silos. To keep each specialist busy, a queue of work forms before each handoff. From the customer's perspective, this is incredibly inefficient, as their request spends most of its time waiting in a queue.
• Flow Efficiency: This is a team-based approach where everyone collaborates to finish a single piece of work. There are no "my parts." The team owns the work from start to finish. This delivers value to the customer much faster and eliminates the immense waste and rework inherent in siloed handoffs.

2. The Trap of Disconnecting from "Why"

The "Feature Factory" Anti-Pattern

A common anti-pattern is the "feature factory," where teams operate like an assembly line. Someone defines requirements, others implement them, and no one on the team connects with the customer or the actual impact of the work. Success is measured by output, not outcome.

The Connection to Taylorism

This is another core tenet of Taylorism: the rigid separation of "thinkers" from "doers." It promotes the unquestioning execution of specifications, regardless of whether those specifications actually solve a real customer problem.

The Predictable Results

Building without a clear purpose has disastrous and wasteful consequences.

• Irrelevant products: Features are built that nobody uses.
• Wasted talent: People work hard on things that simply don't matter.
• Unnecessary complexity: Technology is over-engineered without a connection to a real-world need, creating a massive maintenance burden.

Advocate for Empowered Teams

The alternative is an empowered product team that deeply understands the purpose of its work: what problem they are solving, for whom, and why. These teams don't just blindly follow specs; they adapt solutions to meet real user needs, achieving far greater impact and velocity with less effort.

The Evolution to a Product Team

The journey from a siloed, handoff-driven process to a true product team is an evolution. An effective model shows the team's boundary of collaboration expanding across the entire value stream. In a traditional Waterfall model, each step (Opportunity Selection, Design, Build, Test, Release, Run) is a separate silo. In an ideal Product Team, the entire team collaborates across the whole process, from identifying the opportunity to operating the solution and learning from its impact.

3. The Trap of Deferred Quality

The Core Myth

There's a false dilemma that pits quality against velocity. We're taught to believe there's a tradeoff: you can be "fast without quality" or "slow with quality." This is a foundational myth in our industry.

The Truth

The data is clear. As shown in years of research from the Accelerate/DORA reports, the highest-performing teams have both high quality and high speed. The truth is: Quality does not compete with velocity; it enables it. Of course, context matters. A one-shot script, an MVP, or a proof of concept have different quality constraints than a core product. But for any sustainable product development, building quality in from the start through practices like testing and fast feedback is the only path to sustainable speed. A low-quality approach leads to a progressive slowdown as technical debt and failures accumulate.

The Connection to Taylorism

This myth is a direct descendant of a Taylorist mindset. It treats productivity as a measure of immediate outputs, ignoring sustainability. Quality is seen as a separate, final inspection phase, rather than something built into the process from the beginning.

The Negative Cycle

This mindset creates a system that actively discourages quality.

• Perverse Incentives: The system rewards behaviors that produce low quality. We celebrate heroes who fix crises (often of their own making), equate long hours on weekends with commitment, and reward improvised hacks that create future problems.
• Predictable Results: The consequences are severe and unavoidable. Teams are trapped in a cycle of chronic firefighting. Technical debt grows relentlessly, and the organization suffers from a high rate of talent drain as good people flee the constant chaos.

4. The Path Forward: Systemic Improvement

As W. Edwards Deming taught, over 90% of an organization's performance is a result of the system, not the individuals within it. The responsibility of management is not to blame people but to change the system so that work can flow and people can grow.

Here are actionable strategies to start changing your system.

Lead by Example

As a manager, you must model the behaviors you want to see. If you want teams to write tests, you should write tests. If you want them to refactor, you refactor. Make the desired behaviors visible and celebrate them publicly. Your job is to improve the system, not to find culprits.

Embrace Blameless Postmortems

Analyze incidents and errors transparently with the sole purpose of finding and fixing systemic weaknesses. The goal is to learn and improve, not to assign blame. When the system allows a mistake to happen, the focus must be on strengthening the system.

Show Vulnerability to Build Safety

To create an environment of psychological safety where people feel comfortable highlighting problems, leaders must be the first to show vulnerability. I often share my own past mistakes like the time I took down the phone system for an entire town or made a critical DNS error. Sharing these stories openly shows that errors are a part of work and that the important thing is what we learn from them.

Hack Your Performance Reviews

Traditional performance reviews are often toxic, reinforcing the individualistic, Taylorist mindset. If you are required to use them, hack them. Sometimes you must reinterpret corporate-level mandates to create the right local incentives. Subordinate individual accomplishments to the person's contribution to the team's overall impact. Focus on how they helped the team succeed, not just what they did on their own.

Build End-to-End Ownership

Create cross-functional teams that own a problem from end to end. This often means fighting against the silo mentality. A strategy I've used is to have my team actively take on more work (like operations or QA tasks) to eliminate external dependencies. This gives the team full control, deepens their knowledge, and allows them to optimize for the entire value stream.

Redefine 'Done'

A task is not "done" when the code is committed. It's not "done" when it's deployed. Work is only truly "done" when its impact on the customer has been validated with real feedback. This reinforces a product mindset and focuses the team on outcomes, not just output.

Hire for Mindset

Prioritize mindset and attitude over a checklist of specific technical skills. You can teach someone a new programming language; it's much harder to teach them that quality is their responsibility.

• Product Thinking: Ask product management questions in interviews, like "What products do you admire and why?" or "How would you measure the impact of this feature?" Invite product managers into engineering interviews to help assess this.
• Collaboration: Assess real collaboration, not just talk. When possible, have candidates work with the team for a day (in a paid, respectful way) on a real problem.
• Flexibility: Your career path must reward generalists who provide flexibility, not just specialists.

Master Your Workflow with Kanban

Many teams use Kanban boards, but few use them correctly as a pull system. The three most important rules are:

1. Use WIP (Work In Progress) limits. This is the simplest and most powerful way to optimize flow.
2. Read the board from right to left. During stand-ups, review the work, not the people. Start with what's closest to being "done" (validated impact) and work backward. The focus must always be on finishing work, not starting new work.
3. Use slack to unblock. When WIP limits leave someone without an immediate task, they should not pull new work. Their job is to help colleagues with open tasks, unblocking work further down the chain.

Reducing Resistance by Turning Change Into Experiments

The status quo naturally resists change. To move forward, we need to build the capability to generate change continuously, not just through occasional big initiatives. Approaches inspired by Lean Change Management can help: make small, frequent adjustments, learn quickly, and evolve based on evidence rather than assumptions.

One powerful tactic is to present changes as experiments. Framing them this way reduces resistance, creates psychological safety, and encourages people to engage with curiosity rather than fear. Even when the goal is meaningful transformation, breaking it into small, testable steps makes progress far more sustainable.

Define clear success criteria and a limited timeframe. This lowers resistance and allows you to validate ideas with data. As Grace Hopper famously said, “It’s easier to ask for forgiveness than it is to get permission.”

Conclusion: A Better Way to Work

A bad system will always defeat a good person. The Taylorist principles so common in our industry create systems that are destined to fail, leading to wasted effort, frustrated teams, and mediocre results.

We are discovering better ways to work. Through this experience, we have learned to value:

• Collaboration and shared responsibility over heroes and a culture of blame.
• Global optimization over local efficiency and specialized silos.
• Significant impact and results over occupation and "being busy."
• Autonomy with purpose over rigid control and supervision.
• Continuous learning and adaptation over predefined and static processes.

By focusing on changing the system instead of blaming the people, we can create environments where teams can truly thrive and deliver exceptional value.

Management frameworks and their impact on software development.

Further Reading and Resources

• The Efficiency Paradox by Niklas Modig: A key resource for understanding the difference between Resource Efficiency and Flow Efficiency.
• The resource utilization trap by Henrik Kniberg: A short video that clearly explains why aiming for 100% utilization is a trap.
• The Red Bead Experiment by W. Edwards Deming: A powerful demonstration that performance and quality depend on the system, not on individual effort. It's a vital reminder that systemic issues require systemic fixes.
• For more on these topics:
  • Creating Value and Flow in Product Development (7m) by John Cutler
  • Stop starting and start finishing (5m) by Jason Yip
  • Making Work Visible: How to Unmask Capacity Killing WIP (29m) by Dominica DeGrandis

Mutation Testing: When "Good Enough" Tests Weren't

For weeks, I had been carrying this nagging doubt. The kind of doubt that's easy to ignore when everything is working. My inventory application had 93% test coverage, all tests green, type checking passing. The code had been built with TDD from day one, using AI-assisted development with Claude, Cursor (with Sonnet 4.5, GPT-4o, and Claude Composer), what I like to call "vibecoding". Everything looked solid.

It's not a big application. About 650 lines of production code. 203 tests. A small internal tool for tracking teams and employees. The kind of project where you might think "good enough" is actually good enough.

But something was bothering me.

I had heard about mutation testing years ago. I even tried it once or twice. But let's be honest: it always felt like overkill. The setup was annoying, the output was overwhelming, and the juice rarely seemed worth the squeeze. You had to be really committed to quality (or really paranoid) to go through with it.

This time, though, with AI doing the heavy lifting, I decided to give it another shot.

The First Run: 726 Mutants

I added mutmut to the project and configured it with AI's help. Literally minutes of work. Then I ran it:

$ make test-mutation
Running mutation testing
726/726  🎉 711  ⏰ 0  🤔 0  🙁 0  🔇 15  🔴 0
33.50 mutations/second

Not bad. 711 mutants killed out of 726. That's 97.9% mutation score. I felt pretty good about it.

Until I looked at those 15 survivors.

The 15 Survivors

I ran the summary command to see what had survived:

$ make test-mutation-summary
Total mutants checked: 15
Killed (tests caught them): 0
Survived (gaps in coverage): 15

=== Files with most coverage gaps ===
    5 inventory.services.role_config_service
    4 inventory.services.orgportal_sync_service
    2 inventory.infrastructure.repositories.initiative
    1 main.x create_application__mutmut_6: survived
    1 inventory.services.orgportal_sync_service.x poll_for_updates__mutmut_6: survived
    1 inventory.db.gateway
    1 inventory.app_setup.x include_application_routes__mutmut_33: survived

There they were. Fifteen little gaps in my test coverage. Fifteen cases where my tests weren't as good as I thought.

And remember: this is a 650-line application with 203 tests. If I found 15 significant gaps here, what would I find in a 10,000-line system? Or 100,000?

The thing is, a few months ago, this would have been the end of the story. I would have looked at those 15 surviving mutants, felt slightly guilty, and moved on. The effort to manually analyze each mutation, understand what it meant, and write the specific tests to kill it would have taken days. Maybe a week.

Not worth it for a small internal tool.

But this time was different.

What the Mutants Revealed

Before jumping into fixes, I wanted to understand what these surviving mutants were actually telling me. With AI's help, I analyzed them systematically.

Here's what we found:

In role_config_service (5 survivors):
The service loaded YAML configuration for styling team roles. My tests verified that the service loaded the config and returned the right structure. But they never checked what happened when:

• The YAML file was missing
• The YAML was malformed
• Required fields were absent

The code had error handling for all these cases. My tests didn't verify any of it.

In orgportal_sync_service (4 survivors):
This service synced data from S3. Tests covered the happy path: download file, process it, done. But mutants survived when we:

• Changed log messages (I wasn't verifying logs)
• Skipped metadata checks (last_modified, content_length)
• Removed directory existence checks

The code was defensive. My tests assumed everything would go right.

In database and infrastructure layers (6 survivors):
Similar story. Error paths that existed in production but were never exercised in tests:

• SQLite connection failures
• Invalid data in from_db_row factories
• 404 responses in API endpoints

Classic case of "it works, so I'm not testing the error cases."

The pattern was clear: I had good coverage of normal flows, but my tests were optimistic. They assumed the happy path and left the defensive code untested.

This is what deferred quality looks like at the micro level. Like Deming's red bead experiment (where defects came from the system, not the workers), these weren't random failures. They were systematic gaps in how I verified the system. Every surviving mutant is a potential bug waiting in production, interrupting flow when it surfaces weeks later. The resource efficiency trap: "we already have 93% coverage" feels cheaper than spending 2-3 hours... until you spend days debugging a production issue that a proper test would have caught.

The AI-Powered Cleanup

But this time I had AI. So I did something different.

I asked Claude to analyze the surviving mutants one by one, understand what edge cases they represented, and create or modify tests to cover them. I just provided some guidance on priorities and made sure the new tests followed the existing style.

(The app itself had been built using a mix of tools: Claude for planning and architecture, Cursor with different models for implementation. But for this systematic mutation analysis, Claude's reasoning capabilities were particularly useful.)

In about two or three hours, we had addressed all the key gaps:

• SQLite error handling: I thought I was testing error paths, but I was only testing the happy path. Added proper error injection tests.
• Factory method validation: My from_db_row factories had validation that was never triggered in tests. Added tests with invalid data.
• Edge cases in services: Empty results, missing metadata, nonexistent directories. All cases my code handled but my tests never verified.
• 404 handling in APIs: The code worked, but no test actually verified the 404 response.

The result after several iterations:

$ make test-mutation
Running mutation testing
726/726  🎉 724  ⏰ 0  🤔 0  🙁 2  🔇 0
30.02 mutations/second

$ make test-mutation-summary
Total mutants checked: 2
Killed (tests caught them): 0
Survived (gaps in coverage): 2

=== Files with most coverage gaps ===
    1 inventory.services.role_config_service
    1 inventory.db.gateway

From 15 surviving mutants down to 2. From 97.9% to 99.7% mutation score.

The coverage numbers told a similar story:

Coverage improvements:
- database_gateway.py: 92% → 100%
- teams_api.py: 85% → 100%
- role_config_service.py: 86% → 100%
- employees_api.py: 95% → 100%
- Overall: 93% → 99%
- Total tests: 203 passing

The Shift in Economics

Here's what struck me about this experience: the effort-to-value ratio had completely flipped.

Before AI, mutation testing was something you did if:

• You had a critical system where bugs were expensive
• You had a mature team with time to invest
• You were willing to spend days or weeks on it
• The application was large enough to justify the investment

For a 650-line internal tool? Forget about it. The math never worked out.

Now? The math is different. The AI did all the analysis work. I just had to review and approve. What used to take days took hours. And most of that time was me deciding priorities, not grinding through mutations.

The barrier to rigorous testing has dropped dramatically. And it doesn't matter if your codebase is 650 lines or 650,000. The cost per mutant is the same.

The Question That Remains

I've worked in teams that maintained sustainable codebases for years. I know what that forest looks like (to use Kent Beck's metaphor). I also know how much discipline, effort, and investment it took to stay there.

Now I'm seeing that same level of quality becoming accessible at a fraction of the cost. Tests that used to require days of manual work can be generated in hours. Mutation testing that was prohibitively expensive is now just another quick pass.

The technical barrier is gone.

So here's the question I'm left with: now that mutation testing costs almost nothing, will we actually use it? Will teams that never had the resources to invest in this level of testing quality start doing it?

Or will we find new excuses?

Because the old excuse ("we don't have time for that level of rigor") doesn't really work anymore. The time cost has collapsed. The tooling is there. The AI can do the heavy lifting.

What's left is just deciding to do it. And knowing that it's worth it.

What I Learned

Three concrete takeaways from this experience:

1. Line coverage lies, even in small codebases: 93% coverage looked great until mutation testing showed me the gaps. Those 15 surviving mutants were in critical error handling paths. After fixing them, I still had 99% line coverage. But now the tests actually verified what they claimed to test. If a 650-line application had 15 significant gaps, imagine larger systems.

2. AI makes rigor accessible for any project size: What used to be prohibitively expensive (manual mutation analysis) is now quick and almost frictionless. The economics have changed. From 15 survivors to 2 in just a few hours of work, most of it done by AI. This level of rigor is no longer reserved for critical systems. It's accessible for small internal tools too.

3. 99.7% is good enough: After the cleanup, I'm left with 2 surviving mutants out of 726. Could I hunt them down? Sure. Is it worth it? Probably not. They're edge cases in utility code that's already well-tested. The point isn't perfection. It's knowing where your gaps are and making informed decisions about them.

The real win isn't the numbers. It's the confidence. I now know exactly which 2 mutants survive and why. That's very different from having 93% coverage and hoping it's good enough.

This was a small project. If it had been bigger, I probably would have skipped mutation testing entirely (too expensive, too time-consuming). But now? Now I can't think of a good reason not to do it. Not when it costs almost nothing and reveals so much.

I used to think mutation testing was for perfectionists and critical systems only. Now I think it should be standard practice for any codebase you plan to maintain for more than a few months.

Not because it's perfect. But because it's no longer expensive.

And when the cost drops to almost zero, the excuses should too.

The AI Prompt That Worked

When facing surviving mutants, this single prompt did most of the heavy lifting:

"Run mutation testing with make test-mutation. For each surviving mutant, use make test-mutation-show MUTANT=name to see the details. Analyze what test case is missing and create tests to kill these mutants, following the existing test style. After adding tests, run make test-mutation again to verify they're killed. Focus on the top 5-10 most critical gaps first: business logic, error handling, and edge cases in services and repositories."

The key: let the AI drive the mutation analysis loop while you focus on reviewing and prioritizing.

Getting Started

If you want to try this:

1. Add mutmut to your project (5 minutes with AI help)
2. Create simple Makefile targets to make it accessible for everyone:
   • make test-mutation - Run the full suite
   • make test-mutation-summary - Get the overview
   • make test-mutation-report - See which mutants survived
   • make test-mutation-show MUTANT=name - Investigate specific cases
   • make test-mutation-clean - Reset when needed
3. Run it weekly, not on every commit (mutation testing is slow)
4. Use AI to triage survivors (ask it to analyze and prioritize)
5. Review the top 5-10 gaps as a pair, decide which matter
6. Start with one critical module, not the whole codebase

Making it easy to run is as important as setting it up. The barrier is gone. What's stopping you?

When NOT to chase 100%: Those final 2 surviving mutants? They're in logging and configuration defaults that are battle-tested in production. Perfect mutation score isn't the goal. Knowing your gaps is. Focus on business logic and error handling first. Skip trivial code.

---

About This Project

This application was developed using TDD and AI-assisted development with Claude code and Cursor (using Sonnet 4.5, GPT-5 codex, and Composer1). The mutation testing setup and gap analysis were done with Claude's help using mutmut.

Timeline: The entire mutation testing setup and gap analysis took about 2-3 hours with AI assistance.

Final stats: 649 statements, 208 tests, 99% line coverage, 726 mutants tested, 724 killed (99.7% mutation score).

Related Reading

• When AI Makes Good Practices Almost Free - How AI is changing the economics of software quality practices
• Deming's Red Bead Experiment (video) - Understanding systematic vs random variation in processes
• My Base Setup for Augmented Coding with AI
• Pseudo TDD with AI
• When AI Makes Good Practices Almost Free

When AI Makes Good Practices Almost Free

Since I started working with AI agents, I've had a feeling that was hard to explain. It wasn't so much that AI made work faster or easier, but something harder to pin down: the impression that good practices were much easier to apply and that most of the friction to introduce them had disappeared. That many things that used to require effort, planning, and discipline now happened almost frictionlessly.

That intuition had been haunting me for weeks, until this week, in just three or four days, two very concrete examples put it right in front of me.

The Small Go Application

This week, a colleague reached out to tell me that one of the applications I had implemented in Go didn't follow the team's architecture and testing conventions. They were absolutely right: I hadn't touched Go in years and, honestly, I didn't know the libraries we were using. So I did what I could, leaning heavily on AI to get a quick first version as a proof of concept to validate an idea.
The thing is, my colleague sent me a link to a Confluence page with documentation about architecture and testing, and also a link to another Go application I could use as a reference.

A few months ago, changing the entire architecture and testing libraries would have been at least a week of work. Probably more. But in this case, with AI, I had it completely solved in just two or three hours. Almost without realizing it.

I downloaded the reference application and asked the AI to read the Confluence documentation, analyze the reference application, and generate a transformation plan for my application. Then I just asked it to apply the plan, no adjustments needed, just small interactions to decide when to make commits or approve some operations. In just over two hours, and barely paying attention, I had the entire architecture changed to hexagonal and all the tests updated to use other libraries. It felt almost effortless.

It was a small app, maybe 2000 to 3000 lines of code and around 50 tests, but still, without AI, laziness would have won and I would have only done it if it had been absolutely essential.

The cost of keeping technical coherence across applications has dropped dramatically. What used to take serious effort now happens almost by itself.

The Testing That Stopped Hurting

A few days later, I encountered another similar situation, this time in Python. Something was nagging at me: some edge cases weren't well covered by the tests. I decided to use mutmut, a mutation testing library I'd tried years ago but usually skipped because the juice rarely seemed worth the squeeze.

This time I threw in the library, got it configured in minutes with AI's help, and then I basically went on autopilot: I simply generated the mutations and told the AI to go, one by one, analyzing the mutations and creating or modifying the necessary tests to cover those cases. This process required almost no effort from me. The AI was doing all the heavy lifting. I just prioritized a few cases and gave the tests a quick once-over, simply to check that they followed the style of the others.

In a couple of hours, the change in feeling was complete. Night and day. My confidence in the project's tests had shot up and the effort? Practically nothing.

The Intuition That Became Visible

These two examples, almost back-to-back, confirmed the intuition I had been carrying since I started working with AI agents: the economy of effort is changing. Radically.

Refactoring, keeping things coherent, writing solid tests, documenting decisions... None of that matters less now. What has changed is its cost. And when the cost drops to nearly zero, the excuses should vanish too.

If time and effort aren't the issue anymore, why do we keep falling into the same traps? Why do we keep piling on debt and complexity we don't need?

Perhaps the problem isn't technical. Perhaps the problem is that many teams have never really seen what sustainable code looks like, have never experienced it firsthand. They've lived in the desert so long they've forgotten what a forest looks like. Or maybe they never knew in the first place.

Beth Andres-Beck and Kent Beck use the forest and desert metaphor to talk about development practice. The forest has life, diversity, balance. The desert? Just survival and scarcity.

For years I've worked in the forest. I've lived it. I know it's possible, I know it works, and I know it's the right way to develop software. But I also know that building and maintaining that forest was an expensive discipline. Very expensive. It took mature teams, time, constant investment, and a company culture that actually supported it.

Now, with AI and modern agents, building that forest costs almost the same as staying in the desert. The barrier has dropped dramatically. The barrier isn't effort or time anymore. It's just deciding to do it and knowing how.

The question I'm left with is no longer whether it's possible to build sustainable software. I've known that for years. The question is: now that the cost has disappeared, will we actually seize this opportunity? Will we see more teams moving into that forest that used to be out of reach?

Related Content

• My Base Setup for Augmented Coding with AI
• Pseudo TDD with AI
• Mutation Testing: When "Good Enough" Tests Weren't

Systems: Control its evolution / or be its slave

 When we develop software (digital products), we usually deal with complex systems. We are not prepared (by definition) for emergent behaviors of these types of systems. As a result, it is hard to predict its future evolution and very easy to lose control over it over time.

If we don't control complexity, it will become increasingly difficult to adapt the system to what we need, and we will move from developing and evolving the system to spending more time reacting to it as it evolves. We will become slaves to the system rather than controlling its evolution. We will be spending our time working on production incidents, scalability improvements under pressure, and being in a firefighting mode instead of adding value to the product in a sustainable and continuous manner. Needless to say, the business impact is terrible (poor efficiency, unhappy customers, difficulty in defining strategy, etc).

In my experience, we can keep the system under control by using appropriate tactics.

Tactics to control system evolution

I have used the following tactics to control the evolution of systems I have worked on:

• Good basic hygiene: automated tests, minimum observability, product instrumentation, etc.
• Simple design based on current needs: Using incremental design and lean product development we can develop simple solutions that fulfill our current needs but allow us to evolve the system in the future.
• Evolutionary architectures: Invest in an architecture that can evolve and that allows us to postpone decisions (reversible decisions, fitness functions, observability, etc.).
• Acceptance criteria: Make the acceptance criteria explicit, so that we all know what we consider sufficient hygiene (testing, observability, performance, etc.).
• Define product limits: Define product limits that allow us to convert problems commonly considered "technical" to business/product problems. For example, instead of talking about scalability as an abstract concept, concrete it at the product level with clear limits (number of concurrent users/requests, maximum response times, etc).
• Look for boundaries and use them to assign clear ownership for each identified part. The concepts of bounded contexts of DDD and value stream can assist in identifying such limits within the system. Having clear ownership of each part of the system is fundamental since we must remember that we work on complex socio-technological systems.
• Work in small safe steps: Controlling the evolution of the system is complex and requires us to constantly rethink and adapt. We can only accomplish this if we work in small safe increments of functionality that allow us to change direction without leaving features half-integrated.

The use of these tactics is usually adequate, but as with everything in our profession, it depends, there are times when a little upfront planning and attention to future needs would be beneficial. See "It depends" / The development Mix (Product, Engineering, Hygiene.

Below I include several examples of how I have utilized some of these tactics to control the evolution of the system and not be at its mercy.

Examples

System Load / Scalability evolution

Having performance or stability/availability problems when a system is under high load is very common. The most common approaches for this evolution dimension are:

• Doing nothing and reacting when we discover the first problems (potentially damaging the relationship with our users due to production incidents).
• Solving “imaginary” scalability problems by preparing/designing our systems for a load that is not real and generating a high waste. Sometimes this is the default option chosen in engineering simply for the pleasure of solving an interesting problem.

I think that there is a more sensible approach that allows us to control evolution without generating waste (See Lean software development). In this case, we need to have some metrics about the current load and a rough idea of how much maximum load our system can support (without degrading).

With this information, we can define a soft limit (Product Limit) for the load, as the maximum supported load minus a threshold that gives us time to evolve the system to support more load.

As we can see in the example diagram, we receive a notification at t1 informing us that we reach the moment to improve the scalability of the system. Between t1 and t2, we improve the scalability to support a new load maximum and redefine the new soft limit and repeat the process when the limit is reached (t3).

Storage volume growth

When we store information it is very common that the associated growth of the information volume reaches a point when the technology selected is no longer valid or that we can start to have some problems (high cost, backup problems, etc). 

As in the previous example, we can do nothing and react after we start to have problems or use a tactic to control the evolution/growth.

In this case, if we need to maintain all the historical information, we can follow a similar approach as in the previous example about scalability. But if we don’t need to store the historical information we can create a soft limit to detect when we should start to develop an automated data purge process that periodically removes obsolete information. This approach is very lean because it allows us to postpone the implementation of the data purge process until the last responsible moment.

Controlled errors and quotas

One of the easiest ways to lose control of a system is to have no controlled way of handling errors. When we don’t control errors, we are in risk of having cascade errors and affecting more users. This is one of the most common examples of losing control of the system.

One of the first tactics that help us to maintain control of a system is to detect unexpected runtime errors, avoid crashing and at least send a message to the user and avoid as much sa possible that this error impacts other users.

Another good tactic is to define quotas per user (Product Limits) and use these quotas to throttle requests and show a warning message to the user. In the message, we can inform the user to contact support and use this opportunity to get more information about how the user is using the system or even offer a better SLA to the user.

Committed capacity

When we work in a multitenant SaaS environment is very common that the resources used by the sum of all of our customers increase very fast. One interesting tactic to maintain the control of the evolution of the system in this context is to define how much capacity (of a resource) we commit to each type of user (Product limits). Even knowing that customers will not fully utilize their full capacity, having these definitions allows us to identify the worst-case load scenarios we could have.

In the example diagram, we define the different quotas for each type of user (basic, professional, enterprise). With these definitions and the number of users of each type, we can calculate the maximum committed load at any moment and use this information to make decisions.

This tactic is used frequently by AWS. For each type of resource, they define default limits, and to extend these limits you need to contact support. AWS can use the information from support requests to make very accurate capacity plans and make better product decisions.

Modular monolith

Building a system from scratch is not an easy task, and it is very easy to lose control of the system as it grows (more developers / more functionality). The two most common failures are:

• Growing a monolith organically without committing effort to modularization and ending up with a big ball of mud (http://www.laputan.org/mud/).
• Incorporating a microservices architecture when it is not yet time to scale greatly increases complexity without obtaining any of the benefits of this architecture.

In general, better results can be achieved by developing a monolith, but organizing the code and data into internal modules (modular monolith). As we become familiar with the system and domain, we organize it internally into modules (in alignment with bounded contexts), so we can maintain control of the architecture throughout its evolution. 

When we see the need, we can extract some of these modules into independent services.

It is essential to create mechanisms that help us maintain this modularity during evolution (to perform architecture tests to avoid not allowed dependencies between modules, to use different schemas in the DB, to analyze dependencies periodically).

In addition, we can use these modules to assign ownership of each to a team, allowing each team to work independently.

Conclusions

When we lose control over the evolution of the system:

• we become inefficient and fail to create value.
• we lose customer trust (incidents, errors, etc).
• we react to problems instead of being able to follow our product strategy.

The business impact is huge. Therefore, it is essential to manage complexity and keep evolution under our control.

The tactics I discuss in this article are quite simple and focus on detecting needs early enough so that we can react in a planned way (rather than reacting under pressure). Moreover, they do not require large investments, in fact, they allow us to have a lean approach and avoid over-engineering. But they do require good hygiene practices (testing, observability, reasonable quality, etc). 

References and related content:

• The Last Responsible Moment  
• The development Mix (Product, Engineering, Hygiene) 
• Monolith First 
• It depends (blog series)  
• An introduction to Sociotechnical Architecture Patterns  
• Lean Software Development  
• The 7 wastes of Lean (Software Development)  

Thanks

The post has been improved based on feedback from:

• @islomar@mastodon.social @islomar 
• @georgina_not
• @cristina_verdi  
• @pachilo  
• @saski@mastodon.xyz @saski    
• @gardenunez 
• @MartaMans0  

Thank you very much to all of you

"It depends" / The development Mix (Product, Engineering, Hygiene)

We already know that is everything about context. I read a lot of blog posts talking about how much time a team should use for decoupling components, introducing a new cache system, or improving the scalability of their systems. When reading this type of content, I always think that they are completely right and completely wrong. Everything in our profession depends a lot on the context (the moment of the company, the business strategy, the market traction, etc.).

I use a mental model that helps me classify the work we do, which allows me to communicate and make decisions. I call this mental model "The Mix".

In "The Mix", I classify the work we do as product engineers in:

• Normal product development.
• Implementing the Engineering Roadmap.
• Basic hygiene work.

Normal product development

Normal product development should be the most common type of work for a Stream Aligned team. It should help to fulfill the mission of the team. It can be composed of new feature development, discovery experiments, feature evolution, etc. I prefer a very lean approach for this work, following agile development methods such as XP or Lean Software Development. It is essential to generate the expected outcomes with the minimal amount of code possible and a good internal quality that minimizes the maintenance cost. Following YAGNI, KISS, Simple design is the perfect approach for this kind of work. We don't know the future. The most efficient way to work is to have the most simple solution that covers our customer's needs without making any "speculative" design that generates tons of accidental complexity in 99% of the cases.

Summary:

• Focus on outcomes for the customer within the business constraints.
• Evolutionary design uses Simple design and avoids creating anything for future "expected/invented" needs.
• Use a Lean approach (working in small safe steps).
• Avoid solving problems that we don't have.
• High-speed feedback loop.
• Aligned with the Product Roadmap.

Implementing the Engineering Roadmap

In parallel to the product work, it is very common to identify engineering needs derived from the company's engineering strategy. This strategy should prepare and maintain the current and future engineering capability. Examples of this type of work are:

• Designing the system for fast expected growth (the number of customers, engineering team size, etc.).
• A technology stack change.
• A change in the delivery strategy (from On-Prem to SaaS, from Web to mobile, etc.).
• Prepare the architecture to enable work in autonomous teams.
• This kind of work usually affects several Stream Aligned teams simultaneously and requires coordination at the engineering organization level.
• These initiatives require a lot of investment and should be coordinated with the product roadmap and aligned with the company's general strategy.

Summary:

• Focus on outcomes for the internal architecture and engineering processes.
• Require more upfront effort to design the solution.
• It can be implemented with an agile approach but based on the initial design.
• Low-speed feedback loop.
• By definition, try to solve problems that we don't have (yet).
• It is aligned with the Engineering Roadmap (coordinated with the Product Roadmap).

Basic hygiene work

To develop any nontrivial product, we need to have some practices and development infrastructure that I consider basic hygiene. I'm talking about having a reasonable test strategy, zero-downtime releases, good internal code quality, basic security practices, etc.
In the middle of 2021, not considering these points above seems simply a lack of professionalism. 

So the Basic hygiene work includes any effort we make to implement or improve these minimal practices.

Of course, I am a big fan of product discovery with prototypes, and these, for example, do not have to have the same test strategy. But remember, a prototype that ends up in production, staying in front of our customers for months, is not a prototype. It is a trap.

 

Using The Mix

Thinking about these three types of work and separating them helps me be more explicit about the context and the appropriate trade-offs in each situation. For example, suppose we are in a Normal Product Development initiative. In that case, we cannot expect big architecture change decisions to emerge, and it is better to focus on small safe steps that add value. At the same time, we take notes to consider some initiatives to introduce in the engineering roadmap.

A mature product organization will introduce performance, scalability, and availability initiatives into the product roadmap. In a less mature organization, those needs are likely to be missing from the product roadmap, and it is up to engineering to fight to get them into the engineering roadmap.

We can summarize the different dimensions in this table:

 

	Product development	Engineering Roadmap	Hygiene
Source	Product	Engineering	Team (+Engineering)
Development	Small Safe Steps	Upfront Planning + Small Safe Steps	Small Safe Steps
Practices	YAGNI, KISS, TDD...	Evolutionary Architecture, Load Testing, Testing in Production, migration planning...	Clean Code, CI, CD, Zero downtime, Observability...
Type of needs	Current needs	Future needs	Prerequisite
Value Delivery	Very Fast	Slow	Very Fast
Coordination Needs	The team should be autonomous	Coordination with other teams	Coordination with other teams

If we analyze the current trends in technology using this mental model, some questions arise:

• How do technologies like PaaS or Serverless influence this Mix?
• How does working in Cloud vs. working on-prem affect the engineering roadmap?
• Does it make sense to consider ourselves good professionals if we don't have strong knowledge about hygiene factors?
• How does the mix change in the different phases of the company (startup pre-product-market fit, scale-up, big tech)? And with the life cycle of the product?

The interesting thing about mental models is that they help us think. I hope this model is as valuable for you as it is to me.

Related / Other mental models

• "It depends" / context on creating software products (I)
• "It depends" / Jocelyn Goldfein model for software classification