John Downs

Building Human-Focused Software

Filtering by Tag: Azure Storage

Creating Azure Storage SAS Tokens with ARM Templates

This post was originally published on the Kloud blog.

Shared access signatures, sometimes also called SAS tokens, allow for delegating access to a designated part of an Azure resource with a defined set of permissions. They can be used to allow various types of access to your Azure services while keeping your access keys secret.

In a recent update to Azure Resource Manager, Microsoft has added the ability to create SAS tokens from ARM templates. While this is a general-purpose feature that will hopefully work across a multitude of Azure services, for now it only seems to work with Azure Storage (at least of the services I've checked). In this post I'll explain why this is useful, and give some example ARM templates that illustrate creating both account and service SAS tokens.

Use Cases

There are a few situations where it's helpful to be able to create SAS tokens for an Azure Storage account from an ARM template. One example is when using the Run From Package feature - an ARM template can deploy an App Service, deploy a storage account, and create a SAS token for a package blob - even if it doesn't exist at deployment time.

Another example might be for an Azure Functions app. A very common scenario is for a function to receive a file as input, transform it in some way, and save it to blob storage. Rather than using the root access keys for the storage account, we could create a SAS token and add it to the Azure Functions app's config settings, like in this example ARM template.

How It Works

ARM templates now support a new set of functions for generating SAS tokens. For Azure Storage, there are two types of SAS tokens - account and service - and the listAccountSas and listServiceSas functions correspond to these, respectively.

Behind the scenes, these functions invoke the corresponding functions on the Azure Storage ARM provider API. Somewhat confusingly, even though the functions' names start with list, they are actually creating SAS tokens and not listing or working with previously created tokens. (In fact, SAS tokens are not resources that Azure can track, so there's nothing to list.)

Example

Int his post I'll show ARM templates that use a variety of types of SAS for different types of situations. For simplicity I've used the outputs section to emit the SASs, but these could easily be passed through to other parts of the template such as App Service app settings (as in this example). Also, in this post I've only dealt with blob storage SAS tokens, but the same process can easily be used for queues and tables, and even let us restrict token holders to only access to table partitions or sets of rows.

Creating a Service SAS

By using a service SAS, we can grant permissions to a specific blob in blob storage, or to all blobs within a container. Similarly we can grant permissions to an individual queue, or to a subset of entities within a table. More documentation on constructing service SASs is available hereand the details of the listServiceSas function is here.

Service SASs require us to provide a canonicalizedResource, which is just a way of describing the scope of the token. We can use the path to an individual blob by using the form /blob/, such as /blob/mystorageaccountname/images/cat.jpeg or /blob/mystorageaccountname/images/cats/fluffy.jpeg. Or, we can use the path to a container by using the form /blob/mystorageaccountname/images. The examples below show different types of canonicalizedResource property values.

Read Any Blob Within a Container

Our first SAS token will let us read all of the blobs within a given container. For this, we'll need to provide four properties.

First, we'll provide a canonicalizedResource. This will be the path to the container we want to allow the holder of the token to read from. We'll construct this field dynamically based on the ARM template input.

Second, we need to provide a signedResource. This is the type of resource that the token is scoped to. In this case, we're creating a token that works across a whole blob container and so we'll use the value c.

Third, we provide the signedPermission. This is the permission, or set of permissions, that the token will allow. There are different permissions for reading, creating, and deleting blobs and other storage entities. Importantly, listing is considered to be separate to reading, so bear this in mind too. Because we just want to allow reading blobs, we'll use the value r.

Finally, we have to provide an expiry date for the token. I've set this to January 1, 2050.

When we execute the ARM template with the listServiceSas function, we need to provide these values as an object. Here's what our object looks like:

"serviceSasFunctionValues": {
    "canonicalizedResource": "[concat('/blob/', parameters('storageAccountName'), '/', parameters('containerName'))]",
    "signedResource": "c",
    "signedPermission": "r",
    "signedExpiry": "2050-01-01T00:00:00Z"
}

And here's the ARM template - check out line 62, where the listServiceSas function is actually invoked.

Read A Single Blob

In many cases we will want to issues SAS tokens to only read a single blob, and not all blobs within a container. In this case we make a couple of changes from the first example.

First, our canonicalizedResource now will have the path to the individual blob, not to a container. Again, we'll pull these from the ARM template parameters.

Second, the signedResource is now a blob rather than a container, so we use the value b instead of c.

Here's our properties object for this SAS token:

"serviceSasFunctionValues": {
    "canonicalizedResource": "[concat('/blob/', parameters('storageAccountName'), '/', parameters('containerName'), parameters('blobName'))]",
    "signedResource": "b",
    "signedPermission": "r",
    "signedExpiry": "2050-01-01T00:00:00Z"
}

And here's the full ARM template:

Write A New Blob

SAS tokens aren't just for reading, of course. We can also create a token that will allow creating or writing to blobs. One common use case is to allow the holder of a SAS to create a new blob, but not to overwrite anything that already exists. To create a SAS for this scenario, we work back at the container level again - so the canonicalizedResource property is set to the path to the container, and the signedResource is set to c. This time, we set signedPermission to c to allow for blobs to be created. (If we wanted to also allow overwriting blobs, we could do this by setting signedPermission to cw.)

Here's our properties object:

"serviceSasFunctionValues": {
  "canonicalizedResource": "[concat('/blob/', parameters('storageAccountName'), '/', parameters('containerName'))]",
  "signedResource": "c",
  "signedPermission": "c",
  "signedExpiry": "2050-01-01T00:00:00Z"
}

And here's an ARM template:

Creating an Account SAS

An account SAS works at the level of a storage account, rather than at the item-level like a service SAS. For the majority of situations you will probably want a service SAS, but account SASs can be used for situations where you want to allow access to all blobs within a storage account, or if you want to allow for the management of blob containers, tables, and queues. More detail on what an account SAS can do is available here.

More documentation on constructing account SASs is available hereand the details of the listAccountSas function is here.

Read All Blobs in Account

We can use an account SAS to let us read all of the blobs within a storage account, regardless of the container they're in. For this token we need to use signedServices = b to indicate that we're granting permissions within blob storage, and we'll use signedPermission = r to let us read blobs.

The signedResourceTypes parameter is available on account SAS tokens but not on service SAS tokens, and it lets us specify the set of APIs that can be used. We'll use o here since we want to read all blobs, and blobs are considered to be objects, and reading blobs would involve object-level API calls. There are also two other values for this parameter - s indicates service-level APIs (such as creating new blob containers), and c indicates container-level APIs (such as deleting a blob container that already exists). You can see the APIs available within each category in the Azure Storage documentation.

So our SAS token will be generated with the following properties:

"accountSasFunctionValues": {
    "signedServices": "b",
    "signedPermission": "r",
    "signedResourceTypes": "o",
    "signedExpiry": "2050-01-01T00:00:00Z"
}

Here's the full ARM template that creates this SAS:

List Blob Containers in Account

Finally, let's create a SAS token that will allow us to list the blob containers within our account. For this token we need to use signedServices = b again, but this time we'll use signedPermission = l (since l indicates permission to list).

Somewhat non-intuitively, the API that lists containers is considered a service-level API, so we need to use signedResourceTypes = s.

This means the parameters we're using to generate a SAS token are as follows:

"accountSasFunctionValues": {
    "signedServices": "b",
    "signedPermission": "l",
    "signedResourceTypes": "s",
    "signedExpiry": "2050-01-01T00:00:00Z"
}

Here's the full ARM template that creates this SAS:

You can test this by executing a GET request against this URL: https://.

Other Services

Currently it doesn't appear that other services that use SAS tokens support this feature. For example, Service Bus namespacesEvent Hubs namespaces, and Cosmos DB database accounts don't support any list* operations on their resource provider APIs that would allow for creating SAS tokens. Hopefully these will come soon, since this feature is very powerful and will allow for ARM templates to more self-contained.

Also, I have noticed that some services don't like the listAccountSas function being embedded in their resource definitions. For example, Azure Scheduler (admittedly soon to be retired) seems to have a bug where it doesn't like SAS tokens generated in this way for scheduler jobs. However, I have used this feature in other situations without any such issues.

Deploying App Services with 'Run From Package', Azure Storage, and Azure Pipelines

This post was originally published on the Kloud blog.

Azure App Service recently introduced a feature called Run From Package. Rather than uploading our application binaries and other files to an App Service directly, we can instead package them into a zip file and provide App Services with the URL. This is a useful feature because it eliminates issues with file locking during deployments, it allows for atomic updates of application code, and it reduces the time required to boot an application. It also means that the 'release' of an application simply involves the deployment of a configuration setting. And because Azure Functions runs on top of App Services, the same technique can be used for Azure Functions too.

While we can store the packages anywhere on the internet, and then provide a URL to App Services to find them, a common approach is to use Azure Storage blobs to host the packages. Azure Storage is a relatively cheap and easy way to host files and get URLs to them, making it perfect for this type of deployment. Additionally, by permanently storing our packages in an Azure Storage account we can keep a permanent record of all deployments - and we can even use feature of Azure Storage like immutable blobs to ensure that the blobs can't be tampered with or deleted.

However there are a few different considerations that it's important to think through when using storage accounts in conjunction with App Services. In this post I'll describe one way to use this feature with Azure DevOps build and release pipelines, and some of the pros and cons of this approach.

Storage Account Considerations

When provisioning an Azure Storage account to contain your application packages, there are a few things you should consider.

First, I'd strongly recommend using SAS tokens in conjunction with a container access policy to ensure that your packages can't be accessed by anyone who shouldn't access them. Typically application packages that are destined for an App Service aren't files you want to be made available to everyone on the internet.

Second, consider the replication options you have for your storage account. For something that runs an App Service I'd generally recommend using the RA-GRS replication type to ensure that even if Azure Storage has a regional outage App Services can still access the packages. This needs to be considered as part of a wider disaster recovery strategy though, and also remember that in the event that the primary region for your Azure Storage account is unavailable, you need to do some manual work to switch your App Service to read your package from the secondary region.

Third, virtual network integration is not currently possible for storage accounts that are used by App Services, although it should be soon. Today, Azure allows joining an App Service into a virtual network, but service endpoints - the feature that allows blocking access to Azure Storage outside of a virtual network - aren't supported by App Services yet. This feature is in preview, though, so I'm hopeful that we'll be able to lock down the storage accounts used by App Services packages soon.

Fourth, consider who deploys the storage account(s), and how many you need. In most cases, having a single storage account per App Service is going to be too much overhead to maintain. But if you decide to share a single storage account across many App Services throughout your organisation, you'll need to consider how to share the keys to generate SAS tokens.

Fifth, consider the lifetime of the application versus the storage account. In Azure, resource groups provide a natural boundary for resources that share a common lifetime. An application and all of its components would typically be deployed into a single resource group. If we're dealing with a storage account that may be used across multiple applications, it would be best to have the storage account in its own resource group so that decommissioning one application won't result in the storage account being removed, and so that Azure RBAC permissions can be separately assigned.

Using Azure Pipelines

Azure Pipelines is the new term for what used to be called VSTS's build and release management features. Pipelines let us define the steps involved in building, packaging, and deploying our applications.

There are many different ways we can create an App Service package from Azure Pipelines, upload them to Azure Storage, and then deploy them. Each option has its own pros and cons, and the choice will often depending on how pure you want your build and release processes to be. For example, I prefer that my build processes are completely standalone, and don't interact with Azure at all if possible. They emit artifacts and the release process then manages the communication with Azure to deploy them. Others may not be as concerned about this separation, though.

I also insist on 100% automation for all of my build and release tasks, and vastly prefer to script things out in text files and commit them to a version control system like Git, rather than changing options on a web portal. Therefore I typically use a combination of build YAMLs, ARM templates, and PowerShell scripts to build and deploy applications.

It's fairly straightforward to automate the creation, upload, and deployment of App Service packages using these technologies. In this post I'll describe one approach that I use to deploy an App Service package, but towards the end I'll outline some variations you might want to consider.

Example

I'll use the hypothetical example of an ASP.NET web application. I'm not going to deploy any actual real application here - just the placeholder code that you get from Visual Studio when creating a new ASP.NET MVC app - but this could easily be replaced with a real application. Similarly, you could replace this with a Node.js or PHP app, or any other language and framework supported by App Service.

You can access the entire repository on GitHub here, and I've included the relevant pieces below.

Step 1: Deploy Storage Account

The first step is to deploy a storage account to contain the packages. My preference is for a storage account to be created separately to the build and release process. As I noted above, this helps with reusability - that account can be used for any number of applications you want to deploy - and it also ensures that you're not tying the lifetime of your storage account to the lifetime of your application.

I've provided an ARM template below that deploys a storage account with a unique name, and creates a blob container called packages that is used to store all of our packages. The template also outputs the connection details necessary to upload the blobs and generate a SAS token in later steps.

Step 2: Build Definition

Our Azure Pipelines build definition is pretty straightforward. All we do here is build our app and publish a build artifact. In Azure Pipelines, 'publishing' a build artifact means that it's available for releases to use - the build process doesn't actually save the package to the Azure Storage account. I've used a build YAML file to define the process, which is provided here:

Step 3: Release Definition

Our release definition runs through the remaining steps. I've provided a PowerShell script that executes the release process:

First, it uploads the build artifacts to our storage account's packages container. It generates a unique filename for each blob, ensuring that every release is fully independent and won't accidentally overwrite another release's package.

Next, it generates a SAS token for the packages it's uploaded. The token just needs to provide read access to the blob. I've used a 100-year expiry, but you could shorten this if you need to - just make sure you don't make it too short, or App Services won't be able to boot your application once the expiry date passes.

Finally, it deploys the App Service instance using an ARM template, and passes the full URL to the package - including its SAS token - into the ARM deployment parameters. The key part for the purposes of this post is on line 51 of the template, where we create a new app setting called WEBSITE_RUN_FROM_PACKAGE and set it to the full blob URL, including the SAS token part. Here's the ARM template we execute from the script:

Note that if you want to use this PowerShell script in your own release process, you'll want to adjust the variables so that you're using the correct source alias name for the artifact, as well as the correct resource group name.

Pros and Cons

The approach I've outlined here has a few benefits: it allows for a storage account to be shared across multiple applications; it keeps the build process clean and simple, and doesn't require the build to interact with Azure Storage; and it ensures that each release runs independently, uploading its own private copy of a package.

However there are a few problems with it. First, the storage account credentials need to be available to the release definition. This may not be desirable if the account is shared by multiple teams or multiple applications. Second, while having independent copies of each package is useful, it also means there's some wasted space if we deploy a single app multiple times.

If these are concerns to you, there are a number of things you could consider, depending on your concern.

If your concern is that credentials are being shared, then you could consider creating a dedicated storage account as part of the release process. The release process can provision the storage account (if it doesn't already exist), retrieve the keys to it, upload the package, generate a SAS token, and then deploy the App Service with the URL to the package. The storage account's credentials would never leave the release process. Of course, this also makes it harder to share the storage account across multiple applications.

Keeping the storage account with the application also makes the release more complicated, since you can no longer deploy everything in a single ARM template deployment operation. You'd need at least two ARM deployments, with some scripting required in between. The first ARM template deployment would deploy the storage account and container. You'd then execute some PowerShell or another script to upload the package and generate a SAS token. Then you could execute a second ARM template deployment to deploy the App Service and point it to the package URL.

Another alternative is to pre-create SAS tokens for your deployments to use. One SAS token would be used for the upload of the blobs (and would therefore need write permissions assigned), while a second would be used for the App Service to access all blobs within the container (and would only need read permissions assigned).

Yet another alternative is to use the preview Azure RBAC feature of Azure Storage to authenticate the release process to the storage account. This is outside the scope of this post, but this approach could be used to delegate permissions to the storage account without sharing any account keys.

If your concern is that the packages may be duplicated, you have a few options. One is to simply not create unique names during each release, but instead use a naming scheme that results in consistent names for the same build artifacts. For example, you might use the convention .zip. Subsequent releases could check if the package already exists and leave it alone if it does.

If you don't want to use Azure Storage at all, you can also upload a package directly to the App Service's d:\home\data\SitePackages folder. This way you gain some of the benefits of the Run From Package feature - namely the speed and atomicity of deployments - but lose the advantage of having a simpler deployment with immutable blobs. This is documented on the official documentation page. Also, you can of course use any file storage system you like, such as Amazon S3, to host your packages.

Also, bear in mind that currently App Services on Linux don't support the Run from Package feature at all currently.

Automatic Key Rotation for Azure Services

This post was originally published on the Kloud blog.

Securely managing keys for services that we use is an important, and sometimes difficult, part of building and running a cloud-based application. In general I prefer not to handle keys at all, and instead rely on approaches like managed service identities with role-based access control, which allow for applications to authenticate and authorise themselves without any keys being explicitly exchanged. However, there are a number of situations where do we need to use and manage keys, such as when we use services that don't support role-based access control. One best practice that we should adopt when handling keys is to rotate (change) them regularly.

Key rotation is important to cover situations where your keys may have compromised. Common attack vectors include keys having been committed to a public GitHub repository, a log file having a key accidentally written to it, or a disgruntled ex-employee retaining a key that had previously been issued. Changing the keys means that the scope of the damage is limited, and if keys aren't changed regularly then these types of vulnerability can be severe.

In many applications, keys are used in complex ways and require manual intervention to rotate. But in other applications, it's possible to completely automate the rotation of keys. In this post I'll explain one such approach, which rotates keys every time the application and its infrastructure components are redeployed. Assuming the application is deployed regularly, for example using a continuous deployment process, we will end up rotating keys very frequently.

Approach

The key rotation process I describe here relies on the fact that the services we'll be dealing with - Azure Storage, Cosmos DB, and Service Bus - have both a primary and a secondary key. Both keys are valid for any requests, and they can be changed independently of each other. During each release we will pick one of these keys to use, and we'll make sure that we only use that one. We'll deploy our application components, which will include referencing that key and making sure our application uses it. Then we'll rotate the other key.

The flow of the script is as follows:

  1. Decide whether to use the primary key or the secondary key for this deployment. There are several approaches to do this, which I describe below.

  2. Deploy the ARM template. In our example, the ARM template is the main thing that reads the keys. The template copies the keys into an Azure Function application's configuration settings, as well as into a Key Vault. You could, of course, output the keys and have your deployment script put them elsewhere if you want to.

  3. Run the other deployment logic. For our simple application we don't need to do anything more than run the ARM template deployment, but for many deployments you might copy your application files to a server, swap the deployment slots, or perform a variety of other actions that you need to run as part of your release.

  4. Test the application is working. The Azure Function in our example will perform some checks to ensure the keys are working correctly. You might also run other 'smoke tests' after completing your deployment logic.

  5. Record the key we used. We need to keep track of the keys we’ve used in this deployment so that the next deployment can use the other one.

  6. Rotate the other key. Now we can rotate the key that we are not using. The way that we rotate keys is a little different for each service.

  7. Test the application again. Finally, we run one more check to ensure that our application works. This is mostly a last check to ensure that we haven't accidentally referenced any other keys, which would break our application now that they've been rotated.

We don't rotate any keys until after we've already switched the application to using the other set of keys, so we should never end up in a situation where we've referenced the wrong keys from the Azure Functions application. However, if we wanted to have a true zero-downtime deployment then we could use something like deployment slots to allow for warming up our application before we switch it into production.

A Word of Warning

If you're going to apply this principle in this post or the code below to your own applications, it's important to be aware of an important limitation. The particular approach described here only works if your deployments are completely self-contained, with the keys only used inside the deployment process itself. If you provide keys for your components to any other systems or third parties, rotating keys in this manner will likely cause their systems to break.

Importantly, any shared access signatures and tokens you issue will likely be broken by this process too. For example, if you provide third parties with a SAS token to access a storage account or blob, then rotating the account keys will cause the SAS token to be invalidated. There are some ways to avoid this, including generating SAS tokens from your deployment process and sending them out from there, or by using stored access policies; these approaches are beyond the scope of this post.

The next sections provide some detail on the important steps in the list above.

Step 1: Choosing a Key

The first step we need to perform is to decide whether we should use the primary or secondary keys for this deployment. Ideally each deployment would switch between them - so deployment 1 would use the primary keys, deployment 2 the secondary, deployment 3 the primary, deployment 4 the secondary, etc. This requires that we store some state about the deployments somewhere. Don’t forget, though, that the very first time we deploy the application we won’t have this state set. We need to allow for this scenario too.

The option that I’ve chosen to use in the sample is to use a resource group tag. Azure lets us use tags to attach custom metadata to most resource types, as well as to resource groups. I’ve used a custom tag named CurrentKeys to indicate whether the resources in that group currently use the primary or secondary keys.

There are other places you could store this state too - some sort of external configuration system, or within your release management tool. You could even have your deployment scripts look at the keys currently used by the application code, compare them to the keys on the actual target resources, and then infer which key set is being used that way.

A simpler alternative to maintaining state is to randomly choose to use the primary or secondary keys on every deployment. This may sometimes mean that you end up reusing the same keys repeatedly for several deployments in a row, but in many cases this might not be a problem, and may be worth the simplicity of not maintaining state.

Step 2: Deploy the ARM Template

Our ARM template includes the resource definitions for all of the components we want to create - a storage account, a Cosmos DB account, a Service Bus namespace, and an Azure Function app to use for testing. You can see the full ARM template here.

Note that we are deploying the Azure Function application code using the ARM template deployment method.

Additionally, we copy the keys for our services into the Azure Function app's settings, and into a Key Vault, so that we can access them from our application.

Step 4: Testing the Keys

Once we've finished deploying the ARM template and completing any other deployment steps, we should test to make sure that the keys we're trying to use are valid. Many deployments include some sort of smoke test - a quick test of core functionality of the application. In this case, I wrote an Azure Function that will check that it can connect to the Azure resources in question.

Testing Azure Storage Keys

To test connectivity to Azure Storage, we run a query against the storage API to check if a blob container exists. We don't actually care if the container exists or not; we just check to see if we can successfully make the request:

Testing Cosmos DB Keys

To test connectivity to Cosmos DB, we use the Cosmos DB SDK to try to retrieve some metadata about the database account. Once again we're not interested in the results, just in the success of the API call:

Testing Service Bus Keys

And finally, to test connectivity to Service Bus, we try to get a list of queues within the Service Bus namespace. As long as we get something back, we consider the test to have passed:

You can view the full Azure Function here.

Step 6: Rotating the Keys

One of the last steps we perform is to actually rotate the keys for the services. The way in which we request key rotations is different depending on the services we're talking to.

Rotating Azure Storage Keys

Azure Storage provides an API that can be used to regenerate an account key. From PowerShell we can use the New-AzureRmStorageAccountKey cmdlet to access this API:

Rotating Cosmos DB Keys

For Cosmos DB, there is a similar API to regenerate an account key. There are no first-party PowerShell cmdlets for Cosmos DB, so we can instead a generic Azure Resource Manager cmdlet to invoke the API:

Rotating Service Bus Keys

Service Bus provides an API to regenerate the keys for a specified authorization rule. For this example we're using the default RootManageSharedAccessKey authorization rule, which is created automatically when the Service Bus namespace is provisioned. The PowerShell cmdlet New-AzureRmServiceBusKey can be used to access this API:

You can see the full script here.

Conclusion

Key management and rotation is often a painful process, but if your application deployments are completely self-contained then the process described here is one way to ensure that you continuously keep your keys changing and up-to-date.

You can download the full set of scripts and code for this example from GitHub.

Deploying Blob Containers with ARM Templates

This post was originally published on the Kloud blog.

ARM templates are a great way to programmatically deploy your Azure resources. They act as declarative descriptions of the desired state of an Azure resource group, and while they can be frustrating to work with, overall the ability to use templates to deploy your Azure resources provides a lot of value.

One common frustration with ARM templates is that certain resource types simply can't be deployed with them. Until recently, one such resource type was a blob container. ARM templates could deploy Azure Storage accounts, but not blob containers, queues, or tables within them.

That has now changed, and it's possible to deploy a blob container through an ARM template. Here's an example template that deploys a container called logs within a storage account:

Queues and tables still can't be deployed this way, though - hopefully that's coming soon.

Monitoring Azure Storage Queues with Application Insights and Azure Monitor

This post was originally published on the Kloud blog.

Azure Queues provides an easy queuing system for cloud-based applications. Queues allow for loose coupling between application components, and applications that use queues can take advantage of features like peek-locking and multiple retry attempts to enable application resiliency and high availability. Additionally, when Azure Queues are used with Azure Functions or Azure WebJobs, the built-in poison queue support allows for messages that repeatedly fail processing attempts to be moved to a dedicated queue for later inspection.

An important part of operating a queue-based application is monitoring the length of queues. This can tell you whether the back-end parts of the application are responding, whether they are keeping up with the amount of work they are being given, and whether there are messages that are causing problems. Most applications will have messages being added to and removed from queues as part of their regular operation. Over time, an operations team will begin to understand the normal range for each queue’s length. When a queue goes out of this range, it’s important to be alerted so that corrective action can be taken.

Azure Queues don’t have a built-in queue length monitoring system. Azure Application Insights allows for the collection of large volumes of data from an application, but it does not support monitoring queue lengths with its built-in functionality. In this post, we will create a serverless integration between Azure Queues and Application Insights using an Azure Function. This will allow us to use Application Insights to monitor queue lengths and set up Azure Monitor alert emails if the queue length exceeds a given threshold.

Solution Architecture

There are several ways that Application Insights could be integrated with Azure Queues. In this post we will use Azure Functions. Azure Functions is a serverless platform, allowing for blocks of code to be executed on demand or at regular intervals. We will write an Azure Function to poll the length of a set of queues, and publish these values to Application Insights. Then we will use Application Insights’ built-in analytics and alerting tools to monitor the queue lengths.

Base Application Setup

For this sample, we will use the Azure Portal to create the resources we need. You don't even need Visual Studio to follow along. I will assume some basic familiarity with Azure.

First, we'll need an Azure Storage account for our queues. In our sample application, we already have a storage account with two queues to monitor:

  • processorders: this is a queue that an API publishes to, and a back-end WebJob reads from the queue and processes its items. The queue contains orders that need to be processed.

  • processorders-poison: this is a queue that WebJobs has created automatically. Any messages that cannot be processed by the WebJob (by default after five attempts) will be moved into this queue for manual handling.

Next, we will create an Azure Functions app. When we create this through the Azure Portal, the portal helpfully asks if we want to create an Azure Storage account to store diagnostic logs and other metadata. We will choose to use our existing storage account, but if you prefer, you can have a different storage account than the one your queues are in. Additionally, the portal offers to create an Application Insights account. We will accept this, but you can create it separately later if you want.

1-functionsapp.png

Once all of these components have been deployed, we are ready to write our function.

Azure Function

Now we can write an Azure Function to connect to the queues and check their length.

Open the Azure Functions account and click the + button next to the Functions menu. Select a Timer trigger. We will use C# for this example. Click the Create this function button.

2-function.png

By default, the function will run every five minutes. That might be sufficient for many applications. If you need to run the function on a different frequency, you can edit the schedule element in the function.json file and specify a cron expression.

Next, paste the following code over the top of the existing function:

This code connects to an Azure Storage account and retrieves the length of each queue specified. The key parts here are:

var connectionString = System.Configuration.ConfigurationManager.AppSettings["AzureWebJobsStorage"];

Azure Functions has an application setting called AzureWebJobsStorage. By default this refers to the storage account created when we provisioned the functions app. If you wanted to monitor a queue in another account, you could reference the storage account connection string here.

var queue = queueClient.GetQueueReference(queueName);
queue.FetchAttributes();
var length = queue.ApproximateMessageCount;

When you obtain a reference to a queue, you must explicitly fetch the queue attributes in order to read the ApproximateMessageCount. As the name suggests, this count may not be completely accurate, especially in situations where messages are being added and removed at a high rate. For our purposes, an approximate message count is enough for us to monitor.

log.Info($": ");

For now, this line will let us view the length of the queues within the Azure Functions log window. Later, we will switch this out to log to Application Insights instead.

Click Save and run. You should see something like the following appear in the log output window below the code editor:

2017-09-07T00:35:00.028 Function started (Id=57547b15-4c3e-42e7-a1de-1240fdf57b36)
2017-09-07T00:35:00.028 C# Timer trigger function executed at: 9/7/2017 12:35:00 AM
2017-09-07T00:35:00.028 processorders: 1
2017-09-07T00:35:00.028 processorders-poison: 0
2017-09-07T00:35:00.028 Function completed (Success, Id=57547b15-4c3e-42e7-a1de-1240fdf57b36, Duration=9ms)

Integrating into Azure Functions

Azure Functions has integration with Appliation Insights for logging of each function execution. In this case, we want to save our own custom metrics, which is not currently supported by the built-in integration. Thankfully, integrating the full Application Insights SDK into our function is very easy.

First, we need to add a project.json file to our function app. To do this, click the View files tab on the right pane of the function app. Then click the + Add button, and name your new file project.json. Paste in the following:

This adds a NuGet reference to the Microsoft.ApplicationInsights package, which allows us to use the full SDK from our function.

Next, we need to update our function so that it writes the queue length to Application Insights. Click on the run.csx file on the right-hand pane, and replace the current function code with the following:

The key new parts that we have just added are:

private static string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
private static TelemetryClient telemetry = new TelemetryClient() { InstrumentationKey = key };

This sets up an Application Insights TelemetryClient instance that we can use to publish our metrics. Note that in order for Application Insights to route the metrics to the right place, it needs an instrumentation key. Azure Functions' built-in integration with Application Insights means that we can simply reference the instrumentation key it has set up for us. If you did not set up Application Insights at the same time as the function app, you can configure this separately and set your instrumentation key here.

Now we can publish our custom metrics into Application Insights. Note that Application Insights has several different types of custom diagnostics that can be tracked. In this case, we use metrics since they provide the ability to track numerical values over time, and set up alerts as appropriate. We have added the following line to our foreach loop, which publishes the queue length into Application Insights.

telemetry.TrackMetric($"Queue length – ", (double)length);

Click Save and run. Once the function executes successfully, wait a few minutes before continuing with the next step - Application Insights takes a little time (usually less than five minutes) to ingest new data.

Exploring Metrics in Application Insights

In the Azure Portal, open your Application Insights account and click Metrics Explorer in the menu. Then click the + Add chart button, and expand the Custom metric collection. You should see the new queue length metrics listed.

4-appinsights.png

Select the metrics, and as you do, notice that a new chart is added to the main panel showing the queue length count. In our case, we can see the processorders queue count fluctuate between one and five messages, while the processorders-poison queue stays empty. Set the Aggregation property to Max to better see how the queue fluctuates.

5-appinsightschart.png

You may also want to click the Time range button in the main panel, and set it to Last 30 minutes, to fully see how the queue length changes during your testing.

Setting up Alerts

Now that we can see our metrics appearing in Application Insights, we can set up alerts to notify us whenever a queue exceeds some length we specify. For our processorders queue, this might be 10 messages - we know from our operations history that if we have more than ten messages waiting to be processed, our WebJob isn't processing them fast enough. For our processorders-poison queue, though, we never want to have any messages appear in this queue, so we can have an alert if more than zero messages are on the queue. Your thresholds may differ for your application, depending on your requirements.

Click the Alert rules button in the main panel. Azure Monitor opens. Click the + Add metric alert button. (You may need to select the Application Insights account in the Resource drop-down list if this is disabled.)

On the Add rule pane, set the values as follows:

  • Name: Use a descriptive name here, such as `QueueProcessOrdersLength`.

  • Metric: Select the appropriate `Queue length - queuename` metric.

  • Condition: Set this to the value and time period you require. In our case, I have set the rule to `Greater than 10 over the last 5 minutes`.

  • Notify via: Specify how you want to be notified of the alert. Azure Monitor can send emails, call a webhook URL, or even start a Logic App. In our case, I have opted to receive an email.

Click OK to save the rule.

6-alert.png

If the queue count exceeds your specified limit, you will receive an email shortly afterwards with details:

7-alert.png

Summary

Monitoring the length of your queues is a critical part of operating a modern application, and getting alerted when a queue is becoming excessively long can help to identify application failures early, and thereby avoid downtime and SLA violations. Even though Azure Storage doesn't provide a built-in monitoring mechanism, one can easily be created using Azure Functions, Application Insights, and Azure Monitor.