Cloud Mindset: A Different Way of Thinking (tech Pill)

When building software systems in the cloud, we must adopt a different perspective compared to architecting for on-premises data centers or legacy VPS environments. In traditional setups, capacity is fixed, requiring months of lead time to scale, which makes resources both expensive and scarce. The cloud, however, flips these limitations on their head by offering abundant, rapidly provisioned resources—reshaping how we think about infrastructure and application design.

Traditional Infrastructure: Limitations of the Past

• Fixed capacity: Scaling up in on-premises environments or VPS setups can take months because it involves purchasing and installing new hardware.
• Scarce resources: Businesses often invest in the bare minimum of hardware to minimize costs, leaving little room for flexibility.
• High costs: Upfront hardware purchases and ongoing maintenance are expensive, with costs that must be amortized over time.

The Cloud Paradigm: A New Frontier

In the cloud, servers, storage, and databases feel virtually limitless. These resources can be spun up or down in minutes, allowing teams to adapt quickly to changing needs. This flexibility is both cost-effective and efficient. However, to fully leverage these benefits, we need to shift both our mindset and engineering practices.

Key Principles of the Cloud Mindset

1. Treat Resources as Disposable (Owning vs. Renting a Fleet of Cars)

In traditional IT environments, servers are treated like personally owned cars—carefully maintained, upgraded over time, and expected to last for years. In the cloud, the mindset shifts: servers resemble a fleet of rental cars—standardized, easy to replace, and requiring minimal upkeep. This approach highlights the importance of automation and uniform configurations. When a server or infrastructure component fails, it shouldn’t be manually repaired. Instead, it should be automatically replaced.

Recommended Reading: The History of Pets vs. Cattle and How to Use the Analogy Properly (In cloud architecture, servers are treated as commodities, often explained with the “cattle, not pets” analogy.)

2. Design for Failure

Failures are inevitable in cloud platforms, which run on commodity hardware distributed across multiple data centers and regions. Instead of trying to prevent every failure, embrace them by designing for resilience. Use redundancies, fault tolerance, and graceful degradation to ensure your application continues to operate when something breaks.

Key takeaway: Assume failure will happen and architect your system to recover quickly and minimize impact.

3. Define Infrastructure as Code (IaC)

Infrastructure as Code (IaC) tools, like Terraform or AWS CloudFormation, let you define and version-control your infrastructure. This approach makes provisioning fast, consistent, and repeatable. With IaC, you can test, review, and iterate on infrastructure changes the same way you do with application code.

Learn More: Immutable Infrastructure (Tech Pill)

4. Take Advantage of Cloud Elasticity

Elastic scalability is one of the cloud’s biggest advantages. Instead of over-provisioning for occasional traffic spikes, you can scale up during peak loads and scale down when demand decreases. To do this effectively, design your applications for horizontal scaling—adding more instances rather than making existing ones bigger.

5. Pay Per Use: Rent, Don’t Buy

The cloud’s on-demand pricing model means you only pay for what you use. This flexibility allows you to scale resources up or down based on demand, helping you adapt quickly to changing usage patterns. By spinning up resources during heavy loads and deprovisioning them when idle, you keep costs under control without compromising capacity.

The Bigger Picture

Understanding cloud services, APIs, and GUIs is just the tip of the iceberg when it comes to cloud adoption. The true transformation lies in embracing a fundamental shift in engineering culture and design philosophy. It’s about accepting new assumptions that define how we build and operate in the cloud:

• Resources are limitless: Stop hoarding and start focusing on how to use resources effectively.
• Failure is inevitable: Design for resilience from the outset instead of trying to avoid every possible failure.
• Speed matters: Leverage automation, scripting, and repeatable processes to enable rapid experimentation and iteration.

A New Engineering Challenge

“Construct a highly agile and highly available service from ephemeral and assumed broken components.” Adrian Cockcroft

This challenge captures the essence of building in the cloud. Unlike traditional data centers, the cloud requires us to design for an environment where components are temporary and failure is expected. Adopting a true “cloud mindset” means rethinking old habits and fully leveraging the cloud’s unique characteristics to deliver robust, scalable, and cost-effective solutions.

Key Takeaways

In summary, building for the cloud means embracing four key principles:

• Embrace disposability: Treat infrastructure as temporary and replaceable.
• Design for failure: Build resilience into your system instead of trying to prevent every failure.
• Automate everything: Use tools and processes that allow for speed and consistency.
• Pay only for what you use: Take advantage of the cloud’s cost-efficiency by scaling dynamically.

By adopting these principles, you’ll create services that are highly available, scalable, and agile—perfectly aligned with the demands of modern business.

Related:

• Immutable infrastructure
• Infrastructure as code 
• The History of Pets vs Cattle and How to Use the Analogy Properly
• Technology pills

Thin infrastructure wrappers

Normally, when using architectures that separate infrastructure code from the rest of the application (hexagonal architecture, clean architecture), it is advantageous to create Thin Infrastructure Wrappers. These wrappers allow us to control (and limit) how we use that infrastructure and decouple us from it.

I've been using this approach for over a decade, and it has always worked very well for me. Also, the small cost of creating the wrapper has always been worth it.

Moreover, these wrappers allow us to use them as boundaries with the infrastructure and avoid mocking third-party libraries, which, although sometimes might seem like a good idea, I can say from experience that is a bad decision that couples us with the concrete implementation and makes it harder to apply TDD with the rest of the code.

When we decouple the infrastructure services using a thin wrapper, we can define some Integration tests to validate the implementation and mock/fake the wrapper to implement the rest of the application (instead of mocking the 3ºParty SDK/driver).

Advantages of this approach:

• Minimal API surface (minimal features of the infrastructure used)
• Minimal number of slow integration tests
• Minimized technology-based code spreading through the code base
• It allows us to create code that is more tailored to the business language, separating us from purely technical concepts
• Easy to validate new SDK/infrastructure changes
• Easy to upgrade to new SDK/infrastructure version (including fast security patching)
• Enable TDD flow even for code that uses/integrate infrastructure services
• Less tendency to vendor lock-in (as we reduce the coupling with the infrastructure to the minimum)

Sometimes if we need to use most of the concrete infrastructure functionalities and characteristics (performance, scalability, etc), the thin infrastructure wrapper is not so thin, and we need a different approach. But in most cases, this approach has more advantages than disadvantages.

Example / Redis Cache

Imagine that we need to implement a Cache in our system. We analyze different options and decided that a multi-node Redis cluster can do a good job. As we only need a cache and we use lean software development, we try to satisfy only the current needs (Yagni, defer commitment, simplicity, etc). This means that despite Redis having many use cases and functionalities, we limit its use to the minimum number of functionalities necessary to implement the cache. In this case, it would suffice to have: set_key_with_ttl, get_key, remove_key.

So the RedisWrapper (Thin Infrastructure Wrapper) can be defined as:

RedisWrapper

• set_key_with_ttl(key, value, ttl)
• get_key(key): value
• remove_key(key)

This small piece can be validated easily with a few small integration tests:

• it_should_return_the_previously_stored_value
• a _key_should_expire_when_its_ttl_is_over
• it_does_not_return_a_key_not_previously_stored
• it_does_not_return_a_removed_key
• …

Having this Thin Wrapper, we can develop with TDD a Cache class using only fast unit tests that validate with test doubles the usage of the RedisWrapper. The rest of our system doesn’t require interacting with Redis directly and will only use the Cache class. 

The important thing, in any case, is that although Redis provides us with many types of data (hashmaps, sets, counters, etc...) and allows us to implement many patterns (distributed locks, leader boards, bloom filters, object persistence, cache), we focus only on what we need now, thus minimizing the surface area to cover. This allows us to only worry about whether the use case we are using still works when a new version exists, which we can quickly validate with our wrapper integration tests.

In some cases, if the wrapper interface makes it easy to implement the use cases we need, we can include the implementation in the wrapper itself and create only one abstraction instead of two. In our example, this would mean that if implementing the cache using our RedisWrapper is straightforward, we could just use one abstraction and implement a RedisCache class instead of two (RedisWrapper and Cache). Of course, this only makes sense when there is little logic to implement.

Example / RabbitMQ

If we have to implement queue-based communications, we can actually divide the use case into two parts: the publisher and the subscriber.

If the RabbitMQ SDK makes it very easy to implement both the publisher and the subscriber use cases, a good option is to create only one class for each use case that includes the wrapper and the minimum logic to create the publisher and the subscriber.

In that case, the interface of the classes could look like this:

QueuePublisher

• publish_message(topic, message)

QueueSubscriber

• subscribe(topic, message_processing_function)

Example / AWS boto

Upon entering TheMotion, the initial team had created the foundations of what would later become the product. It consisted of a pipeline of different applications that generated videos from templates, photos, and other dynamic elements. The system ran on AWS, and the different applications communicated using queues (AWS SQS) and stored images and rendered video parts in AWS S3.

Unfortunately, the initial team did not have the habit of wrapping third-party SDKs, so all applications were filled with calls to boto, the AWS SDK for Python, scattered throughout the code. The result of this approach was:

• Integration tests that only covered some cases (since many similar cases were scattered throughout the code).
• Unit tests that were very difficult to understand. They used Moto (https://docs.getmoto.org/en/latest/) as a library to mock the AWS SDK (boto). The Setup phase of  those tests was infernal since it required many low-level calls.
• They only covered some cases due to the cost of developing these tests. The same problem that with the integration tests.
• We had many errors and debugging problems because it was so difficult to control how each developer used the AWS services (different flags, configurations, etc.) was so difficult. Using boto directly gives too many options for inexperienced AWS developers.
• Updating boto versions was a titanic task, so the library's version was outdated (and was likely insecure).

In summary, it was a technical debt hell that prevented us from progressing with good testing practices and generated a lot of frustration in our day-to-day work.

With this starting point and identifying the lack of infrastructure wrappers as one of the problems of the initial codebase, we carried out a systematic and conscious process to solve the problem.

1. We identified the uses of boto throughout the application and defined basic use cases (which covered 80% of the cases).
2. We implemented several wrappers that practically maintained the AWS boto API as it was used in the applications. That is, we created wrappers for publishing messages to SQS and for saving and retrieving objects/files with AWS S3. These initial wrappers were very similar to the boto API but encapsulated sequences of calls that we always repeated (connecting, getting the client, etc.).
3. With these wrappers and by extending the wrapper API as needed, we gradually replaced direct use of boto with the wrappers. At the same time, we modified and simplified the tests.
4. During the substitution and adaptation process, we identified every subtle difference in the use of the AWS API, analyzing whether the difference was necessary or simply the result of chance. If it was necessary, we adapted the API of our wrapper. Otherwise, we simply removed and simplified it.
5. We removed all the old code when all AWS uses pointed to the new wrappers.
6. As a final step, we simplified the API of the wrappers by adapting them to their use, rather than to how the AWS API is designed.

Certainly, this process, although costly, was fundamental to accelerate the subsequent development of the product and to introduce the practices that, as an XP team, allowed us to maintain sustained speed (Continuous Integration, TDD, TBD, etc.).

Conclusions

SDKs for infrastructure services usually have the following characteristics:

• They cover all functionalities and use cases of the service/infrastructure.
• They are designed for general use.
• Due to the previous reasons, they have an extensive API (a huge potential coupling surface).

On the other hand, in our applications, we need to define what we want from the service and infrastructure very well, minimizing the API to use as much as possible. This way of working allows us to:

• Minimize coupling
• Make more reversible the decision of using this concrete infrastructure
• Facilitate maintenance  (version updates, configuration changes, etc.).

Additionally, as professionals (because we are professionals, aren’t we?), we need to facilitate application testing. :)

In this context, a Thin Wrapper that covers only what we use, is small, and can be easily mocked is a fundamental piece to help us have a decoupled infrastructure.

In summary, this way of working:

• Allows us to have tight control over which functionalities and API of an infrastructure service we use.
• Allows for easy and secure updating and changing of that service or driver/SDK version (with the support of integration tests).
• Allows for development using fast unit tests that mock the wrapper we have developed.

If we don't create these kinds of wrappers, we may encounter an issue where the use of the infrastructure service API spreads across the application. This can result in a loss of control over the specific functionalities that we are using, and eventually make it difficult to implement any changes to the infrastructure or SDK/driver, including version upgrades, which can become an extremely challenging process.

Another problem that arises when we don't develop these wrappers is that we create many complicated unit tests that mock or, worse, do monkey patching of many SDK/driver methods.

In summary, my preferred option is to create a Thin Wrapper to narrow down the parts of the SDK we need and abstract the basic interactions. Then, we would have another class oriented towards business with the necessary abstractions (publishing a message, storing an object, etc.).

As an alternative option, and only if the business need can be implemented with very little logic, we could potentially combine the Thin Wrapper with the business abstraction. However, this should only be done in those cases and assuming there are no other abstractions on that part of the infrastructure.

To ensure adaptability to changes, it is essential to minimize the surface area of the SDK used, keep it under control, and decouple it as much as possible. It's important to note that we have no control over changes to the SDK, as these are determined by third-party entities such as vendors, cloud providers, and communities.

Antipatterns

When using this approach, one must be very careful not to fall into some of the common pitfalls:

• Overcomplicating the abstraction
• Allowing complex types or other concepts from the SDK to escape the created abstraction (Leaky abstraction)
• Ending up exposing the entire SDK API "just in case"

References and notes:

In the article we talk about mocks as a generic term, but to clarify, depending on the needs of the test, we should use different types of test doubles (stubs, spies, fakes, mocks, etc.). More info https://martinfowler.com/bliki/TestDouble.html and http://xunitpatterns.com/Test%20Double.html

References:

• Don't Mock 3rd Party Code 
• Integrated Tests Are A Scam
• Simplificators (Nat Pryce) (Via @trikitrok)

Thanks

The post has been improved based on feedback from:

• @georgina_not
• @islomar
• @pmareke
• @msanjuan   

Thank you very much to all of you

API contract & Expand. (Small Safe Steps)

In the latest Toughtworks tech radar (Vol 24 April 2021), API Contract & Expand appears as a technique to be adopted. I couldn't agree more.
We often use complex versioning mechanisms even if a more straightforward API expand and contract strategy works perfectly for most cases.

In general, API expand and contract should be enough for internal APIs. For external APIs, expand and contract should work for small changes, and API versioning is a better option for more significant changes.

If you want to practice this and other strategies to reduce risk and slice technical changes, please check out this Small Safe Steps workshop.

References:

• https://www.thoughtworks.com/radar/techniques/api-expand-contract
• https://www.eferro.net/p/small-safe-steps-3s-workshop.html
• https://www.martinfowler.com/bliki/ParallelChange.html
  

Cloud mindset (tech pill)

To create a significant software system using the cloud we need to think differently than when we are building a system to run in our data center or using VPS from a classic hosting provider.

Outside the Cloud (data centers and VPS),  the underlying computing resources are:

• Effectively fixed because obtaining new resources requires months.
• They are scarce because we try to buy the minimum resources needed.
• Expensive since we have to pay them entirely and amortize them.

But in the Cloud, the underlying computing resources are abundant and fast to provision and this change the rules of the game and deserve a mindset change.

In the Cloud we should:

• Consider the servers/disks/databases as disposable resources, treated as "just one more". This concept is called Pets vs Cattle and even if I don't like the analogy too much, it is true that it is interesting to read about it: The History of Pets vs Cattle and How to Use the Analogy Properly 
• Always assume that anything will fail and design for it. The Cloud is based on commodity hardware that is not under our control and allows us to create massive scale systems. Due to the scale, we experience partial failures all the time (machines go down, disks broke, network fail...). As we can not predict at this scale, the best approach is to design for failure and improve the resilience of our system continuously. 
• Define the infrastructure as code to allow rapid iteration. You can learn more about this concept in this previous blog post  Immutable infrastructure (tech pill).
• Design to take advance of the elasticity of the Cloud. This includes architect for horizontal scalability so you can scale during peak hours without paying for this capacity during low activity hours.
• Pay per use and rent not buy. When we design for the Cloud, we should try to use the resources as much as possible and deal with the variability in the load using the elasticity. The mindset should be, rent the most suited resources for the task and release them as soon as possible.

In summary, understanding the Cloud providers APIs and services is only a small part of your journey to the Cloud. The most critical step is to change your mindset and accept new underlying assumptions.

A new engineering challenge 

Construct a highly agile and highly available service from ephemeral and assumed broken components. (Adrian Cockcroft)

Related:

• Immutable infrastructure
• Infrastructure as code 
• The History of Pets vs Cattle and How to Use the Analogy Properly
• Technology pills

Immutable infrastructure (tech pill)

Immutable infrastructure is a pattern to change/evolve processes without modifying the running code, actual configuration, and the base components (library, auxiliary processes, SO configuration, SO packages, etc).  In summary, avoid any manual or automatic configuration change to the running systems.
But if we don't allow changes, how can we evolve our application and service?  Easy, change the whole piece (function, container, machine) in one shot...

So to make any kind of change (code, configuration, SO) we don't connect to the target box to execute commands, we "cook" a new artifact (container image or machine image) and use it to create a new instance (container or machine) with the changes. In the past, the cost/time to create a new instance was huge, so in order to optimize the process, we tend to execute the minimal change needed to run the new version. I mean, use a manual or automatic process to ssh to the box, install the needed packages, change the configuration, update the code, etc.

But...
¿What happens when the ssh dies in the middle of an update?
¿And if we have a problem installing a package?
¿How can we be sure about the actual state of a machine?
¿How can we calculate the changes to execute if we are not sure about the actual state of a machine?

Making changes in a machine using ssh, is not a transactional operation so we can have a failure in the middle of the process.

The solution, Immutable Infrastructure...
Create a new artifact and run it. Without intermediate states. Only "Not Ready" or "Ready". Simple.  And if something is wrong, destroy the artifact and try again.

This pattern is at the core of the principals container-orchestration systems and PaaS (kubernetes, swarm, mesos, heroku, open shift, Deis, etc).

Why is a good idea

• Simplicity. Is an order of magnitude easier to destroy a resource and create a new one from scratch than to calculate the deltas to apply and execute them. 
• If we need scalability we need to support this patter anyway.
• Right now is easy to implement with the help of the different clouds and PaaS providers.
• Very easy to return to the previous version.
• Very easy to troubleshoot a problem, because there are no intermediate states. You can know the exact content of a running version (the SO state, the concrete conf, the concrete code, etc).
• With this approach, there is no difference between languages and running environment (python, jruby, java, all the same...).  Is a general solution for all the technology your systems requires.

What are the downsides

• In some cases, the bootstrapping/setup time to use a new machine is larger than modifying an existing one. But the time is improving continuously and if we have a scalable architecture we already are dealing with it. Another solution is to use the patter at a different level, for example, functions or containers instead of at the machine level.
• This pattern requires more and longer steps that the classic approach so it is not practical to do it without automation. But not automatize this kind of task is shooting in your foot anyway.

Implementation Samples

As a general implementation strategy we need to be capable to make the following steps:

1. Start a new running process (without processing workload).
2. Detect when this running process is ready to accept real workload.
3. Connect the running process to start processing workload.
4. Disconnect the old running process to stop receiving new workload.
5. The old running process completes the workload that already has.
6. Detect that all the pending workload of the old process is completed.
7. Destroy the old process.

If we talk about web servers the steps can be:

1. Start a new machine/container running the web server in an internal endpoint.
2. Detect when the web server is ready using the health check endpoint.
3. Connect the new web server internal endpoint to the load balancer with the external endpoint.
4. Inform the old web server to stop processing new traffic and disconnect from the load balancer.
5. Detect when the old web server finished.
6. Destroy the old web server.

If we talk about a background queue job processor the workflow can be:

1. Start a new machine/container running the background queue job processor.
2. Detect that the new machine is processing jobs from the queue.
3. Inform the old background queue job processor to not get new jobs.
4. Detect when the old background queue job processor has completed its pending work.
5. Destroy the old background queue job processor.

We can think similar steps for other kinds of processes.

Design Notes

As we can see, this pattern requires collaboration from our code and support from the platform, but I assume that this is part of designing scalable systems that make great use of the capabilities of the cloud. For a good reference about designing applications for the cloud, please read The 12 Factor Apps

We can apply the same pattern at different levels. Virtual machines, Containers, Functions... The ideas are the same but the granularity is different.

Using Infrastructure as Code is recommended prerequisite to implement this pattern.

Conclusions

• The platforms are going in this direction so they have a lot of help to implement it.
• This is the fundamental pattern for:
• Scale up and down in the cloud.
• Advance patterns for deploy with zero downtime (blue-green deploy, rolling deploy, canary releasing, etc).

Other tech pills:

• Infrastructure as code IaC
• Pub-Sub the swiss army knife
• Scalable design pattern sample
• Queues vs Distributed logs

References:

• Trash Your Servers and Burn Your Code: Immutable Infrastructure and Disposable Components Chad Fowler.

Notes:

Thanks to @Fortiz2305 and @cesarob for the feedback.

Infrastructure as code IaC (tech pill)

Infrastructure as code (IaC) is the practice of defining/declaring the infrastructure we need for a system using some kind of machine-readable source files. These source files are used by a tool to provision, create or maintain in a defined state our infrastructure.

These definitions help to provision/create a different kind of resources, compute, storage, communication services, network

For cloud-based infrastructures, we can use these definitions to create "virtual resources" and to configure them to be in a certain state. For example, we can create a virtual machine with an initial  OS image and later install some software and configure it.

In a bare-metal environment, we can use the definition to configure a fixed number of machines and devices already defined in an inventory.

The goals of this practice are:

• Avoid server configuration drifting
• Avoid proliferation of Snow flake Servers.
• Reduce drastically the maintenance cost and the total cost of ownership.
• Allow easy and infrastructure evolution.

As a collateral effect, this practice also allows:

• Use development practices for the infrastructure (version control, testing, audit, collaboration, live documentation...).
• Create on-demand systems for development, QA, testing, and experimentation.
• Developers collaboration.

Cons:  really, it's 2018... no, seriously I don't find a good reason to not to create infrastructure as code. And when the problem is that these resources are difficult or impossible to define using code or some kind of definition, we should avoid them as much as possible.

General Approaches and styles

• Push. We execute a tool that parses the definitions, calculate the changes to do and execute them.
• Pull. Each node/device have its definition and execute all the time in a loop executing the needed changes.
• Push + Pull. A combination of the previous ones, so we can push a change (when we need to be sure or force some changes) or wait until each node/device update its configuration 

Sometimes we can restrict the IaC to provision the low-level infrastructure. For example, we can use this practice to create a PaaS (using Kubernetes, or similar), so the rest of elements are dynamically provision inside the PaaS we have created.

Related Tools:

• Terraform
• Puppet
• Chef
• Ansible
• SaltStack
• Cloudformation
• https://cloud.google.com/deployment-manager/

In summary, IaC is a core DevOps practice and is the base for a lot of the innovations and evolution that the cloud brings us. It is a must for modern development in the cloud and also very recommendable for on-premise deployments.

Other tech pills:

• http://www.eferro.net/2017/08/scalable-design-pattern-sample-tech-pill.html
• http://www.eferro.net/2017/09/pub-sub-swiss-army-knife-tech-pill.html
• http://www.eferro.net/2017/07/queues-vs-distributed-logs-tech-pill.html

"It depends" / Jocelyn Goldfein model for software classification

Continuing with the idea of knowing the context of our software as the first step to making better decisions (see it depends blogpost). I will explain in this post a software classification that I found very useful.

This software classification was created/defined by Jocelyn Goldfein in the article http://firstround.com/review/the-right-way-to-ship-software/ and explained at The "right" way to ship software Jocelyn Goldfein - hack.summit 2016.

The presentation's core message is that there is no single "right" way to ship software, as every release process optimizes for different goals and involves trade-offs. The optimal approach is situational, determined by a company's technology stack/deployment model and business model/mission, requiring a flexible and adaptable mindset rather than adherence to a "religious" belief about one methodology.

The model classifies an application in two axes:

• Horizontal axis: Technology stack and deployment model. From very costly to deploy (on-premise, operating systems, embedded software, etc.) to easy to deploy (web application in a cloud PaaS).

• Vertical axis: Business model and customer type. From costly enterprise-critical software to free consumer apps.

This classification helps define the cost of making a mistake, the optimal release process, how to gather feedback, and more.

For instance:

• Web/cloud deployment allows rapid iteration and testing. Bugs are easier to fix, enabling a more risk-seeking approach ("move fast and break things").
• On-prem or device deployments are slower and riskier. Bugs are expensive to fix and may severely disrupt customer operations, thus requiring more conservative, predictable processes.
• Enterprise-grade software often requires predictability over speed. Customers need to plan around releases.
• Consumer-facing software benefits from great UX and fast experimentation cycles. New features can be introduced continuously without prior scheduling.

Beta programs are an especially effective tool across all quadrants: they allow for frequent shipping and direct feedback from users who are willing to tolerate early-stage issues in exchange for access and influence.

Changing release processes is hard, because it usually demands culture change. Engineers tend to become attached to the processes that worked for them in the past, even if the current context requires a different approach. This can lead to internal conflict, especially when companies pivot from one quadrant (e.g., web) to another (e.g., mobile). To navigate this, teams should align on shared values and mission. Processes are negotiable; purpose is not.

When hiring, prioritize learning mindset over specific methodologies. New hires should understand the current context to avoid blindly applying processes that made sense elsewhere but may not fit now.

I found this classification very useful for my day to day work. But remember, the context can be different for each part of a large system and also evolve with time.

According to this classification, these are the systems in which I have been involved:

Thank you, Jocelyn Goldfein, for this useful classification model.

References:

• http://firstround.com/review/the-right-way-to-ship-software/
• The "right" way to ship software Jocelyn Goldfein - hack.summit 2016
• http://www.eferro.net/2017/12/it-depends-context-on-creating-software.html

Pub-Sub the swiss army knife (tech pill)

Pub-Sub / Publish-Subscribe

"In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are." wikipedia Publish-Subscribe pattern

As the Wikipedia explains, the pub-sub patterns allow decoupling the publishers (P) from the possible subscribers or consumers (C). So the publisher doesn't need to have any knowledge about the components interested in receiving the message.
To implement this pattern we need a central service in charge to receive the messages from the publisher and resend the message for each one of the interested subscribers. Usually, this central service is called broker (B).

Some common characteristics for a broker are:

• Allow broadcasting message to several consumers.
• Allow the consumers to configure in which messages are interested using some kind of consumer defined rule.
• By topic
• By regular expression over a topic 
• By a query over some attributes of the message
• Allow the publisher to send messages and add some meta info to it
• Attributes
• A destination topic
• Other meta info as priority, TTL, etc
• Allow several consumers to receive the same message

Another typical characteristic, and in fact, the most important one is that for each consumer the broker usually has a queue to maintain the pending messages.

And this queue has all the characteristics that we explain in the Queues vs Distributed log post... For example, the broker can load balance the messages between a group of consumers.

In summary, the main characteristics of a pub-sub system are:

• Publishers:
• They don't know the final destination of the messages (decoupling).
• They send the messages to a topic/exchange/abstract destination.
• They can add attributes to the messages.
• Consumers/Subscribers:
• Each one informs the broker the in which messages are interested (using the name of the topic, a regular expression over the topic or a combination of attributes of the message).
• Each message can be consumed by several consumers (broadcasting).

Pros:

• Great decoupling between publishers/consumers.
• Allow easy creation of flexible communication topologies.
• All the pros of Queues.
• Easy to implement.
• Unlimited/Easy horizontal scalability.
• Fault tolerance is very easy to implement (time out and re-queue of the message).
• Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons (same as queues):

• The order is not guaranteed.
• Usually, we can have duplicates.
• Requires more resources from the broker some in some scenarios it has worst scalability than other solutions (not important for the majority of the cases).

Use a Pub-Sub system when:

For any use case that requires flexibility, broadcasting of messages and doesn't require that order of the messages is guaranteed. In these scenarios, a pub-sub system is a great solution because it includes all the use cases of a queue and all the flexibility of sending the same message to several queues/consumers.

The real potential of this kind of solution is when you combine several brokers (using federation or replication)  to create flexible topologies that can communicate several systems and services.

Use cases / Examples:

• Any good scenario for queues.
• Async processing of requests.
• When we can allow losing some data (or having some delays), Pub-Sub systems are great for:
• Log / Monitoring info distribution and processing.
• Asynchronous processing of email requests.
• Processing of independent batch jobs.
• Using Federation
• Info replication and distribution between data centers.
• Global distribution of periodic information.
• Global distribution of reference info.

Implementations:

• RabbitMQ
• Google cloud Pub/Sub
• Pulsar
• SNS+SQS (not very flexible, but very scalable solution)
• Azure Service bus
• ZeroMQ (Low level solution)

Related content:

• Queues vs Distributed logs (tech pill)
• Scalable design pattern sample (tech pill)
• http://www.enterpriseintegrationpatterns.com/patterns/messaging/ Book and repository of integration patterns (using messages) (thanks @bberrycarmen)

Scalable design pattern sample (tech pill)

As a complement to the previous blog post Queues vs Distributed logs (tech pill) this blog post describes how to solve a concrete business problem using a distributed log and a queue.... I used this design several times in production with good results.

Any feedback or improvements of the design will be more than welcomed :)

The problem:

We have to implement a business process with the following characteristics:

• The business process (Job1) can be divided into three different sequential phases (S1, S2, S3). For example, we can think about the generation and mailing of all the invoices for all the customer of one account manager.
• The second phase (S2) can be divided in several (hundreds or thousands) individual and independent sub-jobs (S2.1, S2.2, ...). This sub-jobs can be executed/processed in any order. For example, generate and email one invoice. Each sub-job require one minute of process time.
• The complete process should complete in less than 90 minutes without being affected by the number of sub-jobs of the second step (up to 10k sub-jobs).

We also need to:

• Notify when each step starts and when each step finished.
• Generate some statistics of each the process.
• Access to the detailed status of the process at any moment.

The solution should have the following characteristics:

• We can have several numbers of concurrent business processes of this type for each customer.
• For the sub-jobs at step 2:
• We have retries for the sub-job execution.
• We can balance the time of the process and the cost.
• We have horizontal scalability.

The Design:

After analyzing the requirements and make a fast web storming session we consider the following events:

If we need to include more information about the detailed state of the process we can also signal the starting point of each step using a StepXStarted event. But these Starting Events are not required because we already know when a step started (just when the previous step finished).

Implementation

As we want to implement several functionalities for the same events and we want to have each one completely separated, we can use a distributed log / stream that allows us to design a simple solution to manage the workflow, calculate statistics, compute a detailed status.
The distributed log / stream and the corresponding services can scale using as partition/sharding key the job id or the customer id (assuming that each customer can generate several jobs at the same time).

The Step2 require additional design.
It can be subdivided in several individual jobs (S2.2, S2.2, ...), this jobs can be processed in any order and in parallel so we can have a queue for dispatching this subjobs to a pool workers that can be scaled out if needed.
We can balance the number of workers (and the corresponding cost) with the duration of the Step2 that we want. Each worker get a job description from the queue, execute the job, and include the result in an event (Step2SubjobCompleted) published in the distributed log / stream.

The Workflow Manager should implement a response for each event and generate corresponding events to make the job progress.

The responses for each event are:

Job1Requested:

• Execute Step1
• Publish Step1Completed

Step1Completed:

• Split the Step2 in several subjobs (S2.1, S2.2, ...)
• Store the identifiers of the jobs created (S2.1, S2.2, ...)
• Send the subjobs to the queue of S2 subjobs

Step2SubjobCompleted:

• Mark the id of the job as no longer pending
• Validate if we have jobs pending
• if there is no more jobs pending, publish Step2Completed

Step2Completed:

• Execute Step3
• Publish Step3Completed

Step3Completed:

• Do any garbage recollection needed
• Publish Job1Completed

Job1Completed:

• Nothing, Everything is already done  :)

For the same events, other functionalities as Statistics, reports or notifications, will define other reactions/implementation to process each event... Compute and store statistics, generate emails or push notification, create a view with detailed information on the status of the job, etc...

Notes and complementary designs:

• The queue allows duplication so the workflow should be prepared to receive several events Step2SubjobCompleted for the same subjob.
• We can have retries for the queue jobs, so we should include a mechanism to detect when we should stop waiting for Step2SubjobCompleted. For example, we can use a periodic tick event and use this event to decide if we should continue with the next step (for example marking a subjob as an error).
• Is also possible to continue receiving Step2SubjobCompleted even if we are at Step3, the easy solution is to ignore this messages.
• If we select the Job id as the sharding/partition key for the distributed log / stream we can easily scale out the number of workflow processors. We only need to add more stream shards and the corresponding new processes.
• For the fault tolerance of the workflow we can store the events that we already processed and recover in case of failure from this events.

Related references (very recommended):

• Distributed Sagas: A Protocol for Coordinating  Microservices Caitie McCaffrey
• Confusion about Saga pattern Roman Liutikov
• Applying the Saga Pattern Caitie McCaffrey

Honey Badger Team - Visual History III - CD as driving force

After publishing Honey Badger Team visual history, @artolamola asked me on twitter if I used "continuous delivery" as the driving force for the transformation and why. The short answer is Yes, and in this blog post will try to explain the motivations behind.

For me, agile requires two things (see the-two-pillars-of-agile-software):

• A healthy knowledge culture focused on people (collaboration, learning, respect, team work, creativity...) 
• Looking for quality and technical excellence (for example XP practices are a great starting point).

To introducing the healthy knowledge culture, I introduce retrospectives and worked as a facilitator,  change agent and "Developer Whisperer".

In our profession knowledge is the bottleneck

For the technical excellence I think that we should follow this steps:

• If there is a classic vicious cycle of bad quality or bad practices going on. We should stop and break this cycle immediately. The typical example consists in having technical debt, that generates bugs, that generate even more technical debt that ends in the complete collapse in few months.
• Assume that we have the capacity and very good intentions.
• Help to create a virtuous cycle to improve our technical quality day by day.
• To create this virtuous cycle we need to define a clear goal to align all of our efforts.

Vicious Cycle of technical debt :)

In our case, and having in mind that we are a cloud native product, I thought that we can use Continuous Delivery as our team Goal. 

Why Continuous Delivery?

The answer is that I thought that we require a good practice that force us to do the thing right and doesn't allow us to hide our unprofessionalism or incompetence. And for this, and in the cloud, I think that in our case Continuous Delivery will put a lot of pressure on doing the things right.

Continuous Delivery implies:

• Low cost and a smooth process for deployment that doesn't affect customers. This requires (or at least recommend) introducing DevOps, automation, rolling deploys and Continuous Integration.
• Having good confidence in our product. This requires Continuous Integration, Automatic testing, and Monitoring for production.
• Automatic testing requires integration tests and unit tests, and for sure, having unit tests put a lot of pressure on doing a good design (for example to have a hexagonal architecture or other architecture that decouple business logic from infrastructure).

So indirectly, trying to implement Continuous Delivery requires or recommend doing the following practices: DevOps, Infrastructure Automation, Zero down time deploys (for example rolling deploys), Continuous Integration (perhaps with trunk based development), Integration testing, Unit testing, Good Design (for example using hexagonal architecture).

Virtuous Cycle created when CD is the goal

Continuous Delivery doesn't allow you to hide bad technical quality and force you to introduce good technical practices and work very hard to have a healthy code base.

With this approach, we, as a team, improve a lot our technical knowledge, our code base and our practices. We have to improve many technical practices but we have already come a long way and I sincerely believe that we are going in the right direction. :)

Extraball:

• If you use micro services, CD force you to think about independent deployability.
• CD force you to learn how to make parallel changes (using feature toggles, database refactoring, etc.)
• Some additional insights from @artolamola:
• Above CD you can implement any agile method for team organization (scrum, kanban, etc.).
• CD force you to think how to decompose each feature in vertical slices.

References:

Some of these ideas come from:

• Competing On The Basis Of Speed Mary Poppendieck
• The Virtuous Cycle of Velocity: What I Learned About Going Fast at eBay and Google Randy Shoup

Some context about how I understand agile:

• Agile is not a recipe for success...
• The two pillars of agile software
• Remove the excuses for not being a professional

The history of the Honey Badger team:

• honey badger team visual history
• honey badger team visual history II
• honey badger team

Queues vs Distributed logs (tech pill)

TL;DR A queue is a good choice when you have one kind of job to do that you can divide in independent smaller jobs that can execute in any order and a distributed log is a good choice when you have several kinds of jobs or functionalities for the same stream of data (logs, events, etc.).

Queues vs Distributed Logs

This blog post tries to explain the typical use for Queues and for Distributed Log, but of course, a system usually uses these solutions in combination or in other ways. But I will try to summarize the usual use because some times I see some confusion about how we can use these interesting solutions.

Queue

Use a Queue when:

• You want to distribute workload between workers for the same feature. 
• Each message can represent a job and don't require knowledge about other messages at the queue.

Pros:

• Easy to implement.
• Unlimited/Easy horizontal scalability.
• Fault tolerance is very easy to implement (time out and re-queue of the message).
• Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons:

• The order is not guaranteed.
• Each message can only be consumed by one worker for one feature.
• Usually, we can have duplicates.

Examples

• Asynchronous processing of email requests.
• Processing of independent batch jobs.
• A supermarket queue :)

Alternatives and related variations

• Queues with guaranteed order (Ex: SQS FIFO).
• TTL for messages in the Queue.
• Queues with a max amount of queued messages.

Implementations:

• AWS SQS, RabbitMQ

Distributed Log

Use a Distributed Log when:

• You want to execute several functionalities for the same stream of ordered data (log aggregation, statistics calculation, event sourcing, store streams of information, etc.)
• You want that the data corresponding to the same partition key is processed in order by the same worker instance.

Pros:

• Allow several functionalities for the same data. Each functionality knows what is his own offset.
• If two or more functionalities are at the same offset, they can be sure that have seen the same messages in the same order.
• Having a guaranteed order,  at least at the partition key level, allow design simple solution for calculating statistics, event sourcing, near real-time calculation, etc. 
• You can add new functionalities that use/process the same data with any change in the actual system (Open/Closed principle: open for extension but closed for modification).

Cons:

• Usually, the order of the messages is preserved for each partition key only.
• Scalability using sharding/partitioning.
• More difficult than queues.
• Difficult to have balanced shards/partitions (hot partition problem).

Examples

• Log aggregation / distribution / statistics calculation from the logs.
• Event streaming to allow event sourcing.

Implementations:

• AWS Kinesis, Kafka

Practical scalability example:

In this example we have the following:

• One logical stream
• Divided into two shards
• Each partition key (pk) will always be associated with a shard or partition
• We have one function that runs in two concurrent processes, so the speed of processing is x2. Each process always read events/messages for the same partition keys.

In this example, the first process reads events/messages from the partition key 1 and from the partition key 2.

If the shard s1 have too many event/messages per second, we should scale out dividing the shard s1 into two sub shards.

Now, our speed is x3, and we have one process for each partition key.
As we can see the scalability is not trivial because we should detect when a shard/partition should be divide and rebalance the shards/partition between the running processes.

References:

• The Log: What every software engineer should know about real-time data's unifying abstraction
• Using Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures

Quotes about software architecture

Interesting quotes that can inspire you as software developer or at least make you think :)

"In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’ This understanding includes how the system is divided into components and how the components interact through interfaces. These components are usually composed of smaller components, but the architecture only includes the components and interfaces that are understood by all the developers."  Ralph Johnson

"A complex system that works is invariably found to have evolved from a simple system that worked." John Gall

 "Architecture is the decisions that you wish you could get right early in a project." Ralph Johnson

"So you might end up defining architecture as things that people perceive as hard to change." Martin Fowler

"I think that one of an architect's most important tasks is to remove architecture by finding ways to eliminate irreversibility in software designs." Martin Fowler

"Complexity is what makes software hard to change. That, and duplication." Ralph Johnson

"irreversibility was one of the prime drivers of complexity" Enrico Zaninotto

"If you don’t actively fight for simplicity in software, complexity will win…and it will suck." - @HenrikJoreteg

References about Evolutionary Architectures

After my talk El arte del patadon pa'lante / Libro de recetas (Spanish) at the Pamplona Software Craftmanship 2017, some colleagues ask me about references for evolutionary architectures. This post tries to give the best references I know about this concept...

To have context for the other references, I find the following resources essential:

• Who needs an Architect? (Martin Fowler)
• Competing On The Basis Of Speed  (Mary Poppendieck)  Optimizing for speed, flow, and waste reduction. Lean Software Development

As general resources:

• Agile Architecture Keynote Molly Dishman & Martin Fowler
• Evolutionary Architecture & Micro-Services Rebecca Parsons
• Evolutionary Software Architectures Neal Ford
• Clean Architecture Robert C. Martin
• Embracing Uncertainty Dan North
• The art of destroying software Greg Young
• Lean Software Development
• Lean Principle #4 – Defer Commitment

You also need techniques to make changes to things that traditionally are "hard to change"... For this Martin Fowler bliki is a gold mine:

• https://martinfowler.com/bliki/SacrificialArchitecture.html
• https://martinfowler.com/bliki/Yagni.html
• https://martinfowler.com/bliki/ParallelChange.html
• https://martinfowler.com/bliki/BranchByAbstraction.html
• https://martinfowler.com/bliki/FeatureToggle.html
• https://martinfowler.com/books/refactoringDatabases.html
• https://martinfowler.com/articles/evodb.html

To document all the decisions you make to gain the sharing understanding of the design:

• lightweight architecture decision records

We also need a lot of tricks for our bag of tricks to have different approaches for different problems... Related with this, I think is important to have knowledge about:

• DDD mainly the strategic patterns and the bounded context concept
• Microservices (once you are a master of the monolith of course)
• CQRS
• Continuous delivery (and all the things that require)
• Reactive systems
• Enterprise integration patterns and Patterns of Enterprise Application Architecture
• ...