Comparing Azure Messaging Systems

Comparing Azure Messaging Systems

At the moment there are a number of messaging service available.

  • Storage Queue
  • Service Bus Queue
  • Service Bus Topic
  • Event Hubs
  • Event Grid
  • IoT Hub

This article gives you a general overview.

Storage Queue Azure-Storage---Queue

Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account. The maximum time that a message can remain in the queue is 7 days.

Common uses of Queue storage include:

  • Creating a backlog of work to process asynchronously
  • Passing messages from an Azure web role to an Azure worker role

Service Bus Queue Azure-Service-Bus-Queue

Messages are sent to and received from queues. Queues enable you to store messages until the receiving application is available to receive and process them.

Messages in queues are ordered and timestamped on arrival. Once accepted, the message is held safely in redundant storage. Messages are delivered in pull mode, which delivers messages on request.

Service Bus Topic Azure-Service-Bus-Topic

In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. Useful for scaling to large numbers of recipients, each published message is made available to each subscription registered with the topic. Messages are sent to a topic and delivered to one or more associated subscriptions, depending on filter rules that can be set on a per-subscription basis. The subscriptions can use additional filters to restrict the messages that they want to receive. Messages are sent to a topic in the same way they are sent to a queue, but messages are not received from the topic directly. Instead, they are received from subscriptions. A topic subscription resembles a virtual queue that receives copies of the messages that are sent to the topic. Messages are received from a subscription identically to the way they are received from a queue.

By way of comparison, the message-sending functionality of a queue maps directly to a topic and its message-receiving functionality maps to a subscription. Among other things, this feature means that subscriptions support the same patterns described earlier in this section with regard to queues: competing consumer, temporal decoupling, load leveling, and load balancing.

Event Hubs Azure-Event-Hubs

Azure Event Hubs is a Big Data streaming platform and event ingestion service, capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.

Event Hubs is used in some of the following common scenarios:

  • Anomaly detection (fraud/outliers)
  • Application logging
  • Analytics pipelines, such as clickstreams
  • Live dashboarding
  • Archiving data
  • Transaction processing
  • User telemetry processing
  • Device telemetry streaming

Why use Event Hubs?

Data is valuable only when there is an easy way to process and get timely insights from data sources. Event Hubs provides a distributed stream processing platform with low latency and seamless integration, with data and analytics services inside and outside Azure to build a complete Big Data pipeline.

Event Hubs represents the "front door" for an event pipeline, often called an event ingestor in solution architectures. An event ingestor is a component or service that sits between event publishers and event consumers to decouple the production of an event stream from the consumption of those events. Event Hubs provides a unified streaming platform with time retention buffer, decoupling the event producers from event consumers.

Key features

Event Hubs provides message stream handling capability but has characteristics that are different from traditional enterprise messaging. Event Hubs capabilities are built around high throughput and event processing scenarios. Event Hubs contains the following key components:

  • Event producers: Any entity that sends data to an event hub. Event publishers can publish events using HTTPS or AMQP 1.0 or Apache Kafka (1.0 and above)
  • Partitions: Each consumer only reads a specific subset, or partition, of the message stream.
  • Consumer groups: A view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets.
  • Throughput units: Pre-purchased units of capacity that control the throughput capacity of Event Hubs.
  • Event receivers: Any entity that reads event data from an event hub. All Event Hubs consumers connect via the AMQP 1.0 session, and events are delivered through the session as they become available.

Fully managed PaaS

Event Hubs is a managed service with little configuration or management overhead, so you focus on your business solutions. Event Hubs for Apache Kafka ecosystems gives you the PaaS Kafka experience without having to manage, configure, or run your clusters.

Real-time and batching

Ingest, buffer, store, and process your stream in real time to get actionable insights. Event Hubs uses a partitioned consumer model, enabling multiple applications to process the stream concurrently and letting you control the velocity of processing.

Capture your data in near-real time in an Azure Blob storage or Azure Data Lake Store for long-term retention or micro-batch processing. You can achieve this on the same stream you use for deriving real-time analytics. Setting up Capture is fast, there are no administrative costs to run it, and it scales automatically with Event Hubs throughput units. Event Hubs Capture enables you to focus on data processing rather than on data capture.

Azure Event Hubs also integrates with Azure Functions for a serverless architecture.

Scalable

With Event Hubs, you can start with data streams in megabytes, and grow to gigabytes or terabytes. Auto-inflate feature is one of the many options available to scale the number of throughput units to meet your usage needs.

Rich ecosystem

Event Hubs for Apache Kafka ecosystems enables Apache Kafka (1.0 and above) clients and applications to talk to Event Hubs without having to manage any clusters.

With a broad ecosystem available in various languages (.NET, Java, Python, Go, Node.js), you can easily start processing your streams from Event Hubs. All supported client languages provide low-level integration.

Event Grid Azure-Event-Grid3

Azure Event Grid allows you to easily build applications with event-based architectures. You select the Azure resource you would like to subscribe to, and give the event handler or WebHook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. Event Grid also has custom support for application and third-party events, using custom topics and custom webhooks.

You can use filters to route specific events to different endpoints, multicast to multiple endpoints, and make sure your events are reliably delivered. Event Grid also has built in support for custom and third-party events.

Event sources

Currently, the following Azure services support sending events to Event Grid:

  • Azure Subscriptions (management operations)
  • Custom Topics
  • Event Hubs
  • IoT Hub
  • Media Services
  • Resource Groups (management operations)
  • Service Bus
  • Storage Blob
  • Storage General-purpose v2 (GPv2)

Event handlers

Currently, the following Azure services support handling events from Event Grid:

  • Azure Automation
  • Azure Functions
  • Event Hubs
  • Hybrid Connections
  • Logic Apps
  • Microsoft Flow
  • Queue Storage
  • WebHooks

Concepts

There are five concepts in Azure Event Grid that let you get going:

  • Events - What happened.
  • Event sources - Where the event took place.
  • Topics - The endpoint where publishers send events.
  • Event subscriptions - The endpoint or built-in mechanism to route events, sometimes to multiple handlers. Subscriptions are also used by handlers to intelligently filter incoming events.
  • Event handlers - The app or service reacting to the event.

Capabilities

Here are some of the key features of Azure Event Grid:

  • Simplicity - Point and click to aim events from your Azure resource to any event handler or endpoint.
  • Advanced filtering - Filter on event type or event publish path to ensure event handlers only receive relevant events.
  • Fan-out - Subscribe multiple endpoints to the same event to send copies of the event to as many places as needed.
  • Reliability - Utilize 24-hour retry with exponential backoff to ensure events are delivered.
  • Pay-per-event - Pay only for the amount you use Event Grid.
  • High throughput - Build high-volume workloads on Event Grid with support for millions of events per second.
  • Built-in Events - Get up and running quickly with resource-defined built-in events.
  • Custom Events - use Event Grid route, filter, and reliably deliver custom events in your app.

Iot Hub Azure-Iot-Hub

IoT Hub is a managed service, hosted in the cloud, that acts as a central message hub for bi-directional communication between your IoT application and the devices it manages. You can use Azure IoT Hub to build IoT solutions with reliable and secure communications between millions of IoT devices and a cloud-hosted solution backend. You can connect virtually any device to IoT Hub.

IoT Hub supports communications both from the device to the cloud and from the cloud to the device. IoT Hub supports multiple messaging patterns such as device-to-cloud telemetry, file upload from devices, and request-reply methods to control your devices from the cloud. IoT Hub monitoring helps you maintain the health of your solution by tracking events such as device creation, device failures, and device connections.

IoT Hub's capabilities help you build scalable, full-featured IoT solutions such as managing industrial equipment used in manufacturing, tracking valuable assets in healthcare, and monitoring office building usage.

What to use when?

Azure provides myriad options to perform messaging and decouple applications. Which one should you use, and when?

Azure-Event-Grid3 Azure-Event-Hubs Azure-Iot-Hub Azure-Service-Bus-Topic Azure-Service-Bus-Queue Azure-Storage---Queue
Event Ingestion X X X
Device management X
Messaging X X X X X X
Multiple consumers X X X X
Multiple senders X X X X X X
Use for decoupling X X X X X
Use for publish/subscribe X X
Max message size 64K 64K 256KB 1MB 256 KB|1 MB 64KB

Extra Information

Compare Storage Queues and Service Bus Queues

Compare Event Grid, Event Hubs and Service Bus

Related Article