Pub-Sub the swiss army knife (tech pill)

Pub-Sub / Publish-Subscribe

"In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are." wikipedia Publish-Subscribe pattern

As the Wikipedia explains, the pub-sub patterns allow decoupling the publishers (P) from the possible subscribers or consumers (C). So the publisher doesn't need to have any knowledge about the components interested in receiving the message.
To implement this pattern we need a central service in charge to receive the messages from the publisher and resend the message for each one of the interested subscribers. Usually, this central service is called broker (B).

Some common characteristics for a broker are:

• Allow broadcasting message to several consumers.
• Allow the consumers to configure in which messages are interested using some kind of consumer defined rule.
• By topic
• By regular expression over a topic 
• By a query over some attributes of the message
• Allow the publisher to send messages and add some meta info to it
• Attributes
• A destination topic
• Other meta info as priority, TTL, etc
• Allow several consumers to receive the same message

Another typical characteristic, and in fact, the most important one is that for each consumer the broker usually has a queue to maintain the pending messages.

And this queue has all the characteristics that we explain in the Queues vs Distributed log post... For example, the broker can load balance the messages between a group of consumers.

In summary, the main characteristics of a pub-sub system are:

• Publishers:
• They don't know the final destination of the messages (decoupling).
• They send the messages to a topic/exchange/abstract destination.
• They can add attributes to the messages.
• Consumers/Subscribers:
• Each one informs the broker the in which messages are interested (using the name of the topic, a regular expression over the topic or a combination of attributes of the message).
• Each message can be consumed by several consumers (broadcasting).

Pros:

• Great decoupling between publishers/consumers.
• Allow easy creation of flexible communication topologies.
• All the pros of Queues.
• Easy to implement.
• Unlimited/Easy horizontal scalability.
• Fault tolerance is very easy to implement (time out and re-queue of the message).
• Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons (same as queues):

• The order is not guaranteed.
• Usually, we can have duplicates.
• Requires more resources from the broker some in some scenarios it has worst scalability than other solutions (not important for the majority of the cases).

Use a Pub-Sub system when:

For any use case that requires flexibility, broadcasting of messages and doesn't require that order of the messages is guaranteed. In these scenarios, a pub-sub system is a great solution because it includes all the use cases of a queue and all the flexibility of sending the same message to several queues/consumers.

The real potential of this kind of solution is when you combine several brokers (using federation or replication)  to create flexible topologies that can communicate several systems and services.

Use cases / Examples:

• Any good scenario for queues.
• Async processing of requests.
• When we can allow losing some data (or having some delays), Pub-Sub systems are great for:
• Log / Monitoring info distribution and processing.
• Asynchronous processing of email requests.
• Processing of independent batch jobs.
• Using Federation
• Info replication and distribution between data centers.
• Global distribution of periodic information.
• Global distribution of reference info.

Implementations:

• RabbitMQ
• Google cloud Pub/Sub
• Pulsar
• SNS+SQS (not very flexible, but very scalable solution)
• Azure Service bus
• ZeroMQ (Low level solution)

Related content:

• Queues vs Distributed logs (tech pill)
• Scalable design pattern sample (tech pill)
• http://www.enterpriseintegrationpatterns.com/patterns/messaging/ Book and repository of integration patterns (using messages) (thanks @bberrycarmen)

Scalable design pattern sample (tech pill)

As a complement to the previous blog post Queues vs Distributed logs (tech pill) this blog post describes how to solve a concrete business problem using a distributed log and a queue.... I used this design several times in production with good results.

Any feedback or improvements of the design will be more than welcomed :)

The problem:

We have to implement a business process with the following characteristics:

• The business process (Job1) can be divided into three different sequential phases (S1, S2, S3). For example, we can think about the generation and mailing of all the invoices for all the customer of one account manager.
• The second phase (S2) can be divided in several (hundreds or thousands) individual and independent sub-jobs (S2.1, S2.2, ...). This sub-jobs can be executed/processed in any order. For example, generate and email one invoice. Each sub-job require one minute of process time.
• The complete process should complete in less than 90 minutes without being affected by the number of sub-jobs of the second step (up to 10k sub-jobs).

We also need to:

• Notify when each step starts and when each step finished.
• Generate some statistics of each the process.
• Access to the detailed status of the process at any moment.

The solution should have the following characteristics:

• We can have several numbers of concurrent business processes of this type for each customer.
• For the sub-jobs at step 2:
• We have retries for the sub-job execution.
• We can balance the time of the process and the cost.
• We have horizontal scalability.

The Design:

After analyzing the requirements and make a fast web storming session we consider the following events:

If we need to include more information about the detailed state of the process we can also signal the starting point of each step using a StepXStarted event. But these Starting Events are not required because we already know when a step started (just when the previous step finished).

Implementation

As we want to implement several functionalities for the same events and we want to have each one completely separated, we can use a distributed log / stream that allows us to design a simple solution to manage the workflow, calculate statistics, compute a detailed status.
The distributed log / stream and the corresponding services can scale using as partition/sharding key the job id or the customer id (assuming that each customer can generate several jobs at the same time).

The Step2 require additional design.
It can be subdivided in several individual jobs (S2.2, S2.2, ...), this jobs can be processed in any order and in parallel so we can have a queue for dispatching this subjobs to a pool workers that can be scaled out if needed.
We can balance the number of workers (and the corresponding cost) with the duration of the Step2 that we want. Each worker get a job description from the queue, execute the job, and include the result in an event (Step2SubjobCompleted) published in the distributed log / stream.

The Workflow Manager should implement a response for each event and generate corresponding events to make the job progress.

The responses for each event are:

Job1Requested:

• Execute Step1
• Publish Step1Completed

Step1Completed:

• Split the Step2 in several subjobs (S2.1, S2.2, ...)
• Store the identifiers of the jobs created (S2.1, S2.2, ...)
• Send the subjobs to the queue of S2 subjobs

Step2SubjobCompleted:

• Mark the id of the job as no longer pending
• Validate if we have jobs pending
• if there is no more jobs pending, publish Step2Completed

Step2Completed:

• Execute Step3
• Publish Step3Completed

Step3Completed:

• Do any garbage recollection needed
• Publish Job1Completed

Job1Completed:

• Nothing, Everything is already done  :)

For the same events, other functionalities as Statistics, reports or notifications, will define other reactions/implementation to process each event... Compute and store statistics, generate emails or push notification, create a view with detailed information on the status of the job, etc...

Notes and complementary designs:

• The queue allows duplication so the workflow should be prepared to receive several events Step2SubjobCompleted for the same subjob.
• We can have retries for the queue jobs, so we should include a mechanism to detect when we should stop waiting for Step2SubjobCompleted. For example, we can use a periodic tick event and use this event to decide if we should continue with the next step (for example marking a subjob as an error).
• Is also possible to continue receiving Step2SubjobCompleted even if we are at Step3, the easy solution is to ignore this messages.
• If we select the Job id as the sharding/partition key for the distributed log / stream we can easily scale out the number of workflow processors. We only need to add more stream shards and the corresponding new processes.
• For the fault tolerance of the workflow we can store the events that we already processed and recover in case of failure from this events.

Related references (very recommended):

• Distributed Sagas: A Protocol for Coordinating  Microservices Caitie McCaffrey
• Confusion about Saga pattern Roman Liutikov
• Applying the Saga Pattern Caitie McCaffrey

Queues vs Distributed logs (tech pill)

TL;DR A queue is a good choice when you have one kind of job to do that you can divide in independent smaller jobs that can execute in any order and a distributed log is a good choice when you have several kinds of jobs or functionalities for the same stream of data (logs, events, etc.).

Queues vs Distributed Logs

This blog post tries to explain the typical use for Queues and for Distributed Log, but of course, a system usually uses these solutions in combination or in other ways. But I will try to summarize the usual use because some times I see some confusion about how we can use these interesting solutions.

Queue

Use a Queue when:

• You want to distribute workload between workers for the same feature. 
• Each message can represent a job and don't require knowledge about other messages at the queue.

Pros:

• Easy to implement.
• Unlimited/Easy horizontal scalability.
• Fault tolerance is very easy to implement (time out and re-queue of the message).
• Allow easy balance between latency (time at the queue + processing time) and the cost of the concurrent workers.

Cons:

• The order is not guaranteed.
• Each message can only be consumed by one worker for one feature.
• Usually, we can have duplicates.

Examples

• Asynchronous processing of email requests.
• Processing of independent batch jobs.
• A supermarket queue :)

Alternatives and related variations

• Queues with guaranteed order (Ex: SQS FIFO).
• TTL for messages in the Queue.
• Queues with a max amount of queued messages.

Implementations:

• AWS SQS, RabbitMQ

Distributed Log

Use a Distributed Log when:

• You want to execute several functionalities for the same stream of ordered data (log aggregation, statistics calculation, event sourcing, store streams of information, etc.)
• You want that the data corresponding to the same partition key is processed in order by the same worker instance.

Pros:

• Allow several functionalities for the same data. Each functionality knows what is his own offset.
• If two or more functionalities are at the same offset, they can be sure that have seen the same messages in the same order.
• Having a guaranteed order,  at least at the partition key level, allow design simple solution for calculating statistics, event sourcing, near real-time calculation, etc. 
• You can add new functionalities that use/process the same data with any change in the actual system (Open/Closed principle: open for extension but closed for modification).

Cons:

• Usually, the order of the messages is preserved for each partition key only.
• Scalability using sharding/partitioning.
• More difficult than queues.
• Difficult to have balanced shards/partitions (hot partition problem).

Examples

• Log aggregation / distribution / statistics calculation from the logs.
• Event streaming to allow event sourcing.

Implementations:

• AWS Kinesis, Kafka

Practical scalability example:

In this example we have the following:

• One logical stream
• Divided into two shards
• Each partition key (pk) will always be associated with a shard or partition
• We have one function that runs in two concurrent processes, so the speed of processing is x2. Each process always read events/messages for the same partition keys.

In this example, the first process reads events/messages from the partition key 1 and from the partition key 2.

If the shard s1 have too many event/messages per second, we should scale out dividing the shard s1 into two sub shards.

Now, our speed is x3, and we have one process for each partition key.
As we can see the scalability is not trivial because we should detect when a shard/partition should be divide and rebalance the shards/partition between the running processes.

References:

• The Log: What every software engineer should know about real-time data's unifying abstraction
• Using Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures