Routing Azure Event Grid Events to a Service Bus Queue

Very Brief Overview of Azure Event Grid

What makes Event Grid one of the coolest (and most innovative) services on Azure is it’s unique integration between event sources and event handlers.

Event sources can emerge from a continually growing list of Azure services. A source can also be a custom event that originates from any line of business application or service, running anywhere.

Event handlers can be services on Azure, such as Azure Functions, Logic Apps, an automation run book and several more. In addition, webhooks (or callbacks) expand Event Grid’s reach by supporting any reachable HTTPS endpoint as a handler.

Delivery of events are supported by a configurable retry schedule, filters and dead-letter channel options.  If you’re interested in learning more about Event Grid delivery, please see this MSDN article: Managing Event Delivery with Azure Event Grid.

I really like this updated diagram of Event Grid because it relates handlers into categories (workflows, serverless, etc.) that are typically associated with other, commonly related requirements.

Event Grid is essentially a message broker that serves as a conduit between the event sources and handlers. Events are delivered to handlers using a push-push approach, which eliminates the need for any polling from the consumer or handler of the event. This attractive feature, in addition to it’s publish-subscribe model, makes Event Grid truly unique.

For a deeper dive into the service and how it fits into an event-driven architecture, see: Event-Driven Architecture in the Cloud with Azure Event Grid.

Service Bus Queue as a Destination

Announced this week, Event Grid now has the ability to register a Service Bus queue as a handler for events.

This long-requested feature opens up a new set of integration stories for solutions that are already using Service Bus Queues.

Why route to a queue?

Subscribing to events with a queue offers several benefits:

1. Queue-based load leveling.  Leveraging a queue can not only decouple tasks but also offer a buffer for back-end services that otherwise might be overwhelmed with an influx of messages. With a queue in place, those services can now dequeue and process those messages at a responsible rate.
2. Long running tasks. Event Grid expects to receive an acknowledgment from it’s handler fairly quickly (a 200 OK). If it does not receive a response in a timely manner, it will resend the event based on the configured retry schedule. In some cases, this could result in multiple events being sent to a handler, and sometimes even ending up in a dead-letter channel if it never completes on time. Employing a queue as a handler will allow another process to come along and take the time it needs to complete the next steps.
3. Wire tap. I sometimes use this pattern (with other pub/sub services such as Service Bus topics) to inspect messages when troubleshooting and monitoring a solution. It also comes in handy in testing scenarios when you simply want to review the events as they are coming in.

Configure a Service Bus Queue Handler

I’m going to use the Azure Cloud Shell to configure everything – it’s easy, cross-platform and just works.

To make the commands easier to read (and reusable), let’s initialize some variables. Nothing is created yet, these will be names of the resources that will be provisioned:

rgname=<your-resource-group-name>
topicname=<event-grid-topic-name>
sbnamespace=<unique-service-bus-namespace-name>
queuename=<queue-name>

Create a resource group:

az group create -n $rgname -l westus2

Create a Service Bus namespace and queue

az servicebus namespace create -n $sbnamespace -g $rgname -l westus2 --sku Standard
az servicebus queue create -n $queuename -g $rgname --namespace-name $sbnamespace

Create a Event Grid custom topic

az eventgrid topic create -g $rgname --name $topicname -l westus2

At this point all the resources needed have been provisioned. What’s left is creating an event subscription for the queue.

Event Grid is expecting the resource ID of the service bus queue, which will be in the following format (line-breaks added for clarity):

/subscriptions/{subscription-id}/
    resourceGroups/{resource-group-name}/
    providers/Microsoft.ServiceBus/
    namespaces/{service-bus-namespace-name}/
    queues/{queue-name}

Create Event Grid subscription to a Service Bus Queue

# get the event grid topic resource id
topicid=$(az eventgrid topic show --name $topicname -g $rgname --query id --output tsv)

# get the service bus queue resource id
endpoint=$(az servicebus queue show --namespace-name $sbnamespace --name $queuename --resource-group $rgname --query id --output tsv)

#create the event subscription
az eventgrid event-subscription create --source-resource-id $topicid --name test-sub --endpoint-type servicebusqueue --endpoint $endpoint

That was fairly easy and almost exactly how you create a subscription for a Storage Queue.

Testing

The following steps will send an event to the Event Grid topic that was created earlier:

# retrieve the event grid topic endpoint
topicendpoint=$(az eventgrid topic show -n $topicname -g $rgname --query "endpoint" --output tsv)

# retrieve the access key to send events to the topic
topickey=$(az eventgrid topic key list --name $topicname -g $rgname --query "key1" --output tsv)

# copy a sample event payload
body=$(eval echo "'$(curl https://raw.githubusercontent.com/dbarkol/azure-event-grid-patterns/master/badsongrequest.json)'")

# off you go...
curl -X POST -H "aeg-sas-key: $topickey" -d "$body" $topicendpoint

That’s a wrap.

References

• Azure Event Grid documentation
• Service Bus Queue event handler