Azure Event Hubs, Kafka and Dependency Injection in Azure Functions

functions-kafka-hubs

This post will demonstrate a solution that leverages the following technologies and Azure features:

GitHub repository: https://github.com/dbarkol/functions-eventhubs-kafka

Event Hubs and Kafka

Azure Event Hubs now supports Apache Kafka 1.0 and higher. What this means is that Event Hubs provides wire protocol compatibility for Kafka. This is a similar pattern used by other cloud services. For example, Azure Cosmos DB takes the same approach for the APIs that it supports (Cassandra, Gremlin, Table, MongoDB and SQL).

20

Events Hubs doesn’t actually run Apache Kafka. Similarly, Cosmos does not run a version of MongoDB underneath. Instead, compatibility is established at the wire level.

Compatibility at this level is exciting because, in addition to providing a managed solution to a distributed streaming platform, it also introduces Event Hubs to the rich ecosystem of Kafka applications and tools in the open source community.

For more information about Event Hubs and Kafka support, see Use Azure Event Hubs from Apache applications.

Enabling Kafka in Azure Event Hubs

From the Portal

This post provides a walkthrough of how to create an Apache Kafka-enabled Event Hubs namespace from the Azure portal: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create-kafka-enabled.

An Event Hubs namespace maps to a Kafka cluster, which is a collection of Brokers.

From the CLI

I prefer to use the Azure CLI over the portal as much as possible. If you are interested in a step-by-step guide using the CLI, see the README in the GitHub repository.

Producing Events from an Azure Function

I’m going to dive right into the implementation of the Azure Function, starting with some of the required packages:

Using Code, I just executed the following commands to add the packages:

dotnet add package Confluent.Kafka
dotnet add package Microsoft.Azure.Functions.Extensions

CA Certificate file

Included in the project is a file called cacert.pem. This is required by the Confluent library to verify that the endpoint comes from a verified source. The messaging services on Azure only expose endpoints over HTTPS (no HTTP endpoints, ever). Leveraging this certificate will ensure client-broker encryption with Event Hubs.

When deploying the Function to Azure, we want to include this file in the output directory. The csproj file in the solution accomplishes this with this entry in the ItemGroup section:

<None Update="cacert.pem">
   <CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>

Application Settings

For local development, the local.settings.json file should look similar to:

{
    "IsEncrypted": false,
    "Values": {
        "AzureWebJobsStorage": "",
        "FUNCTIONS_WORKER_RUNTIME": "dotnet",
        "IsLocal": "1",
        "EventHubConnectionString": "{your-event-hub-connection-string}",
        "EventHubFqdn": "{event-hub-namespace}.servicebus.windows.net:9093",
        "EventHubName": "{event-hub-name}"
    }
}

All the Event Hub settings (connection string, FQDN and name) will need to be added to the Application Settings in the Function App once it’s deployed in Azure.  The IsLocal property is used to help determine the path of the certificate between environments (local vs Azure).

Kafka Producer Interface

The solution includes an interface for a Kafka Producer with a single method for sending an event:

Kafka Producer Implementation

The implementation for the Kafka Producer is responsible for instantiating an instance of the Producer class from the Confluent library. This code has no knowledge of Event Hubs and will work with a traditional Kafka cluster as well.

Registering the Kafka Producer Service

In order to provide an instance of this class within the implementation of the function, we will have to register the service. The assembly attribute registers the configuration method.

Putting it all together

It all comes together in the implementation of the Azure Function, which is triggered by an HTTP request.

Notice how an instance of the Kafka Producer class is passed into the constructor. It is then used within the invocation of the function to send an event to a Kafka endpoint:

Consuming Events with the kafkacat Utility

kafkacat is a really cool utility that can produce and consume events with a Kafka topic. I wanted to continue the story of using the Kafka protocol by using this utility to read from Event Hubs.

If you are on a Windows machine, you can install kafkacat with the Windows Subsystem for Linux.

To consume events, fire up a terminal and initialize some variables:

connectionString="{event-hub-connection-string}"
kafkaEndpoint="{event-hub-namespace}.servicebus.windows.net:9093"
topicname="{topic-name}"

Start consuming events (line-breaks added for clarity):

kafkacat -X security.protocol=SASL_SSL 
         -X sasl.mechanisms=PLAIN 
         -X sasl.username="\$ConnectionString" 
         -X sasl.password=$connectionString 
         -b $kafkaEndpoint 
         -t $topicname 
         -o end

An example of a consumed test message can be seen in the screenshot below:

kafkacat-sample

Summary

There are a lot of moving parts here with Event Hubs, Apache Kafka and some shiny, new dependency injection support within Azure Functions.

Hopefully, there are some helpful, reusable components in this solution that you find useful. Thank you for reading!

Resources

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s